What I measure about my AI agents, and what the numbers say. Each finding has a question, a method, a result and a caveat. Every number has its window.
Agent measurements taken .
01
Estimates against reality
Question
When an agent estimates a task, how long does the task really take?
Method
Every non-trivial task is logged with an estimate when it starts and closed when it ends. An audit matches both events to the tool calls in the agent's own session transcript, then corrects or excludes any time that does not match. The calibration factor is the median of real time divided by the first estimate, per model that made the estimate.
Result
In the last 30 days: Codex gpt-6-astra took 0.44x its estimate over 238 tasks; Claude Opus 5.5 took 0.58x its estimate over 201 tasks; Claude Sonnet 5.5 took 0.36x its estimate over 10 tasks. Every one of them finished faster than it estimated. In plain terms, for Codex gpt-6-astra: an hour estimated took 26 minutes (median). Only 10% to 16.4% of tasks landed within 20% or 5 minutes of their first estimate.
Caveat
Only tasks with a matched transcript count. The model is the one that made the estimate, so a lane task logged by the orchestrator carries the orchestrator's model. Sonnet's factor rests on 10 tasks. The ledger started on 8 October 2026.
Poke a bar to read it.
Data table
Calibration per estimating model, last 30 days
Model
Factor (real / estimate)
Tasks
Within 20% or 5 min
Codex gpt-6-astra
0.44x
238
16.4%
Claude Opus 5.5
0.58x
201
15.9%
Claude Sonnet 5.5
0.36x
10
10%
02
First-pass rate
Question
How often does a lane's work pass verification without a fix?
Method
After a lane reports, the orchestrator verifies the work by running it and records one verdict for that run: pass, fixed or failed. The first-pass rate is the share of verdicts that are pass. The verdict is the only field in the ledger that an agent writes; every other number comes from harness records.
Result
In the last 30 days the orchestrator recorded 181 verdicts after verifying lane work. 136 passed on the first try (75.1%). 45 needed a fix round or were rejected.
Caveat
Verdicts started on 10 October 2026. Four verdicts cannot rank a model. The models did not get the same tasks, so the rates compare outcomes, not raw ability.
Poke a bar to read it.
Data table
First-pass rate per model, last 30 days
Model
First pass
Verdicts
Lane runs
Codex gpt-6-astra
71.3%
122
2,024
Claude Opus 5.5
82.8%
29
53
Claude Sonnet 5.5
80.8%
26
80
Grok 4.7 Build
100%
4
60
03
Who does the work
Question
Which models carry the load, in runs and in busy hours?
Method
A run is one delegated lane run, recorded by the harness that starts it, native subagent runs included. Busy hours are the time a model spent working inside its sessions, read from the session transcripts. An unknown value stays empty; nothing is estimated.
Result
In the last 30 days: 2,576 runs and 805.5 busy hours across 15 models. Codex gpt-6-astra ran 2,024 of them (78.6%). That is more busy hours than the 720 hours in 30 days, because agents work in parallel.
Caveat
A run without a thread id gets a model only when its working directory and start time match exactly one session within 15 seconds; the rest stay model unknown. Busy hours add up parallel sessions, so they can exceed the hours on the clock.
In the last 30 days: 2,576 runs and 805.5 busy hours across 15 models.
RunsBusy hours
Poke a bar to read it.
Data table
Runs and busy hours per model, last 30 days
Model
Runs
Done
Busy hours
Sessions
Median run
Codex gpt-6-astra
2,024
97.7%
477.4
1,655
7.5 min
Codex gpt-6.1-sol
261
99.2%
52.2
242
8 min
Claude Sonnet 5.5
80
88.8%
12.2
221
25.2 min
Codex gpt-5.6-sol
74
97.3%
19.4
78
14.8 min
Grok 4.7 Build
60
91.7%
22.5
79
17.3 min
Claude Opus 5.5
53
84.9%
194.7
392
37 min
Grok 4.7
20
65%
8.4
72
12.1 min
Grok 4.6
4
100%
3.5
19
13.9 min
Claude Opus 5
0
no record
13.1
33
no record
Claude Haiku 5.5
0
no record
1.7
30
no record
Claude Opus 4.8
0
no record
0.4
5
no record
04
What an agent's own timing is worth
Question
When an agent reports when it started and finished, can the report be trusted?
Method
On 10 October 2026 the first full audit matched every logged task against the session transcripts of Claude Code and Codex. Each start and each close had to match a real tool call, in time and in text.
Result
Of 183 tasks, 175 matched their transcript and 1 needed a corrected time. 6 were excluded: 2 were closed within two minutes on an estimate of 15 minutes or more, so they were logged after the fact, and 4 stayed open after their session went idle for six hours. 1 had no supporting record at all; it came from Grok. The same audit counted 63 failed logging calls.
Caveat
One audit, one operator, about two days of tasks. It measures whether the log is honest, not whether the estimates were good.
05
nerfIndex: do AI models get worse?
Question
Do hosted AI models lose quality over time without notice?
Method
nerfIndex runs a fixed daily suite: 48 items over 8 dimensions (reasoning, instruction following, structured output, code, long context, knowledge, tool use and refusal), drawn from public datasets such as GSM8K, MMLU-Pro, HumanEval+ and IFEval. Temperature 0, seed 42, once a day per route. It also records price moves, incidents from 27 official status pages, limit changes and new models. A model is marked degraded only after ten complete runs at least 20 hours apart: the first eight set the baseline, and the drop must be at least 5 points, with a 99% bootstrap interval below zero and McNemar p < 0.01, on the latest two runs.
Result
First daily scores, 29 September 2026: Qwen3.8 27B on Groq 77.8%, Qwen3 8B local 69%, gpt-oss-120b 65.2%, gpt-oss-20b 63.0%, Qwen2.5 3B local 46.8%. On 8 October 2026 the catalog held 476 models; among 143 base models with enough public benchmark coverage, Xiaomi MiMo-V2.6-Pro had the highest public score under $1 per million tokens.
Caveat
The daily suite runs on free routes only, under a $0 inference budget, so it does not cover paid frontier models, and Anthropic models are excluded. One day of scores says nothing about a trend; a verdict needs the full ten runs.
Last check from this server: nerfindex.com answered HTTP 200 at .
06
How many price changes are real?
Question
Model catalogs change prices all day. How much of that is noise?
Method
nerfIndex folds a price flap into one event. A field that returns to its old value within 48 hours, or switches three or more times in 24 hours, folds the whole model-day. A real numeric move must hold for six hours before it appears.
Result
On a read-only snapshot of production on 8 October 2026, the 30-day price feed went from 1,086 events to 403, and the last 24 hours from 169 to 108.
Caveat
The thresholds are a choice, not a standard. A real change that reverts within 48 hours is folded as a flap.