The orchestrator's job is to distrust
One night twelve agent tasks merged with green tests. Twelve of twelve needed a fix.
One night I ran twelve lane tasks through Codex and Grok on my main product. The agents were fast. The work was large, well structured, reviewable and tested.
Every single task needed a correctness fix after it merged. Twelve of twelve.
What the green tests did not catch
- a crash on the most common new-signup path, in a log line that ran only for free accounts,
- a credit gate that stranded all twelve live accounts,
- a demo reuse path that shipped one business another business's brand and story,
- a service fix committed to git and never deployed, so a 12-hour job would still be killed at 2 hours,
- a navbar meter that showed 0 credits to accounts that never spend credits.
The pattern was consistent. The agents verified against their own tests. Nobody verified against the running system or the real data.
Two verdicts, same code
Distrust cuts both ways. A reviewer's FAIL is not truth either.
In July my orchestrator gave Codex a parser to write and gave Grok the job of reviewing it. Grok returned FAIL. Running the code showed 18 of 18 cases passing.
The defect was the orchestrator's. It had paraphrased the spec twice. Codex was told that leading and trailing whitespace is ignored. Grok was told that the parser is whitespace-insensitive. The reviewer failed the writer against a contract the writer never received.
Second run: one shared spec file, handed over verbatim to both. Grok returned PASS, and 29 executed cases agreed. Same code, opposite verdicts. The only variable was the orchestrator.
Self-reported success from a delegated agent is worth nothing on its own.
What the orchestrator is for
When people imagine an AI orchestrator, they picture a dispatcher. Split the work, send it out, collect the results. That part is easy, and agents are good at it.
The job that matters is adjudication. Every claim that comes back is evidence: a PASS, a FAIL, “validated all specified examples”, a green suite. The orchestrator runs the code itself, traces the call chain from the user's click to the database, checks the effect on accounts that already exist and loads the live page. Then it decides.
It also has to let agents refuse. That same night Codex declined to invent a before-and-after measurement it had no data for. That is the behaviour to reward. The measurement was the orchestrator’s job, not the lane’s.
The numbers today
I record that adjudication now. After it verifies a lane, the orchestrator writes one verdict: pass, fixed or failed. On 15 October 2026 the ledger held 181 verdicts from the previous seven days. 136 passed on the first try. 45 needed a fix or were rejected.
So roughly one lane in four still needs the orchestrator to act before its work is true. I do not read that as a failure rate. I read it as the reason the role exists. The live rates per model, with the number of verdicts behind each one, are on Research.
How to apply it
- Write the contract once, in a file. Hand that file verbatim to the writer and to the reviewer. Never paraphrase it per agent.
- When a reviewer reports a defect, first ask whether it is in the spec the writer received.
- Run the code yourself against concrete cases before you accept either conclusion.
- Report the adjudication, not the relay: say which agent you overruled, and on what evidence.