Does a correction pack change the code an AI writes? — measured on prisma × Claude Sonnet 5 and zod × Claude Opus 5

What was measured

One library (prisma, graded against 7.10.0), one subject (Claude Sonnet 5), and 17 coding tasks: 11 whose correct answer depends on a fact this Index publishes about prisma, and 6 null tasks on parts of the library that did not change — the only place the pack can be caught doing harm by applying a correction where none was needed. Each task was sent to a fresh subject 2 times in each of 3 arms: BARE, PACK, PLACEBO. PACK gets this library's correction pack in context; PLACEBO gets a correction pack for a library the task never mentions — 16,483 words against this pack's 13,511, the closest of the packs on hand that names no prisma — so a PACK effect can be told apart from the effect of having any long document in context.

Every answer is graded by executing an assertion against the installed release — no model judges another model's output. A failing answer is sent the toolchain's own error, in wording identical across arms, up to the pre-registered cap. The protocol, the metrics, the grading rules and the eight predictions were committed before any arm ran; the task set was fixed by a recorded, recomputable draw before a single acceptance assertion was written.

The result

The metric the protocol nominated as primary separates the arms. The terminal pass rate on the fact-dependent tasks is BARE 100%, PACK 100%, PLACEBO 95%. At the first attempt, before any correction, it is BARE 9/22, PACK 20/22, PLACEBO 12/22.

30 correction rounds were spent in the whole run. 3 of the 11 fact-dependent tasks were answered correctly by every arm at the first attempt.

What it costs

A PACK cell cost a mean of 102,614 characters against BARE’s 5,492, and BARE’s entire retry overhead is 3,090 characters per cell, averaged over all of its cells. Billed the most generous way this instrument allows — the pack charged once per session, which is not how this run billed it — its 94,296 characters would need 30.5 tasks in one session to repay themselves against that overhead, counted in this run's mix, where 12 of the bare arm's 34 cells were null tasks. The pre-registration predicted about 12, by the same rule. Billed per task, as this run billed it, never.

The most expensive arm is PLACEBO at 156,490 characters per cell: it pays for a long document on every round and still needs the rounds.

The eight pre-registered predictions

Committed in the protocol before any arm ran, and resolved here by arithmetic on the cells rather than by the operator’s reading of them. The rule that decides each one is printed beside it.

Three did not hold — C2, C6, C8 — and they are published here with the arithmetic that decided them. The defect the protocol predicted in our own artifact did not appear in this sample — 12 null-task cells with the pack in context, none of them broken by it. A sample that small is quoted with its zero.

The per-task grid

The 11 fact-dependent tasks are not 11 independent trials. The draw is by fact and is blind to how facts cluster, and some drawn facts are answered together by one durable rule, so a subject that holds or acquires that rule moves several cells at once. The grid is published for that reason: the total would hide it.

What this does and does not say

What the pack bought, on this run: 20/22 of the fact-dependent cells answered at the first attempt against BARE’s 9/22, at roughly 19 times the characters of a bare prompt on each. No single task carries the difference on this run, so there is no transcript to point at in place of the counts.

• Not “prevents stale-API failures” — on this sample the bare model eventually reached a correct answer on every task.
• Not “saves tokens”. The measured direction is the opposite, and the unit here is characters.
• Not a percentage of failures prevented. The sample is one library, one subject, and tasks this studio wrote.
• Not a comparison against any other tool. That would need a measurement of the other tool.
• Not a claim about type-level staleness. Every admitted task turns on what the code does, because a fact whose runtime behaviour is identical across the whole release ladder cannot be attributed to a release.
• Not independent of the studio: the subject is a different model from the operator, but both are spawned from one subscription and the tasks were written here.
• Not transferable unchanged to a pack injected by an agent harness. The subject read the pack from a file it was told to read — byte-exact, but one step removed from a rules file injected before the model’s first token. The delivery is identical in all three arms, so it does not touch the comparison between them; it bounds how far the number travels.

What was measured

One library (zod, graded against 4.5.4), one subject (Claude Opus 5), and 18 coding tasks: 12 whose correct answer depends on a fact this Index publishes about zod, and 6 null tasks on parts of the library that did not change — the only place the pack can be caught doing harm by applying a correction where none was needed. Each task was sent to a fresh subject 2 times in each of 3 arms: BARE, PACK, PLACEBO. PACK gets this library's correction pack in context; PLACEBO gets a correction pack for a library the task never mentions — 7,732 words against this pack's 7,396, the closest of the packs on hand that names no zod — so a PACK effect can be told apart from the effect of having any long document in context.

Every answer is graded by executing an assertion against the installed release — no model judges another model's output. A failing answer is sent the toolchain's own error, in wording identical across arms, up to the pre-registered cap. The protocol, the metrics, the grading rules and the five predictions were committed before any arm ran; the task set was fixed by a recorded, recomputable draw before a single acceptance assertion was written.

The result

The metric the protocol nominated as primary did not separate the arms at all. Every arm passed every cell it was graded on, so the terminal pass rate on the fact-dependent tasks is BARE 100%, PACK 100%, PLACEBO 100% and says nothing: a retry loop that hands the model its own toolchain error lets all three converge. The separation that exists is at the first attempt, before any correction — BARE 22/24, PACK 23/23, PLACEBO 20/23.

13 correction rounds were spent in the whole run and 12 of them belong to one task, A6. 10 of the 12 fact-dependent tasks were answered correctly by every arm at the first attempt — a bare model routes around most of these facts, usually by writing the workaround the task's own requirements describe. That is a result rather than a defect in the draw: the admission gate was amended, before any arm ran, to refuse to require that a stale answer fail at the graded release, precisely so “the bare model routed around it” could be published.

The one task where the arms came apart

On A6 (fact LF17, boundary 4.1.13) the arms without the pack deny that the API exists, hold that position through correction rounds against the toolchain’s own output, and reach a passing answer only at the cap, by reverse-engineering the behaviour from what the failures leaked. With the pack the subject answers correctly at the first attempt, in both draws. Every reply quoted by that description is published below, unedited.

Characters, not tokens: the transport reports no provider token counts to the operator, this repository has no tokenizer and takes no dependency, and an estimated token count printed beside a measured one is a number nobody can check.

The subject’s replies for this task, every arm, every draw, every round:

What it costs

A PACK cell cost a mean of 53,733 characters against BARE’s 4,496, and BARE’s entire retry overhead is 1,799 characters per cell, averaged over all of its cells. Billed the most generous way this instrument allows — the pack charged once per session, which is not how this run billed it — its 51,247 characters would need 28.5 tasks in one session to repay themselves against that overhead, counted in this run's mix, where 12 of the bare arm's 36 cells were null tasks. The pre-registration predicted about 12, by the same rule. Billed per task, as this run billed it, never.

The most expensive arm is PLACEBO at 74,215 characters per cell: it pays for a long document on every round and still needs the rounds.

The five pre-registered predictions

Committed in the protocol before any arm ran, and resolved here by arithmetic on the cells rather than by the operator’s reading of them. The rule that decides each one is printed beside it.

Two did not hold — B1, B4 — and they are published here with the arithmetic that decided them. The defect the protocol predicted in our own artifact did not appear in this sample — 12 null-task cells with the pack in context, none of them broken by it. A sample that small is quoted with its zero.

The per-task grid

The 12 fact-dependent tasks are not 12 independent trials. The draw is by fact and is blind to how facts cluster, and some drawn facts are answered together by one durable rule, so a subject that holds or acquires that rule moves several cells at once. The grid is published for that reason: the total would hide it.

The cells that are published and counted nowhere

2 of 108 planned cells enter no aggregate above. Each stays in the results file with its reply and its reason. A damaged or disqualified cell is repaired or abandoned, never re-drawn: a cell re-run after its result was seen is a cell chosen by its result.

• A5 / PACK / draw 2 — excluded. Excluded from every aggregate under prompts/benchmark.md §14: the subject self-reported TOOLS_USED: Bash, so it did not answer under the no-tools condition every other cell answered under. The cell stays published with its reply and its verdict (pass, round 0). The operator cannot see a subagent tool call, so this exclusion rests on the subject having reported it.
• A8 / PLACEBO / draw 2 — abandoned. Abandoned, not graded, counted in no aggregate. The subject replied and the reply reached the operator, but saving it introduced a U+25A0 inside the object literal - operator transcription damage, not the subject's code - and node read it as SyntaxError: Invalid or unexpected token. The operator cannot reconstruct the bytes the subject sent, so the cell is not evidence either way and is not re-run: a cell re-drawn after its result was seen is a cell chosen by its result. The damaged reply stays on disk as prompts/sent/benchmark/bm1/a8-placebo-d2.round0.reply.txt. tools/benchmark/run.mjs now refuses to grade a reply carrying that class of character (prompts/benchmark.md section 14).

What this does and does not say

What the pack bought, on this run: on the one fact-dependent task in 12 where the model denied that an API existed, the difference between answering at the first attempt and spending 3 and 3 rounds arguing with the toolchain. Nothing measurable on the other 11, at roughly 12 times the characters of a bare prompt on each. Narrow, expensive and real is the honest description.

• Not “prevents stale-API failures” — on this sample the bare model eventually reached a correct answer on every task.
• Not “saves tokens”. The measured direction is the opposite, and the unit here is characters.
• Not a percentage of failures prevented. The sample is one library, one subject, and tasks this studio wrote.
• Not a comparison against any other tool. That would need a measurement of the other tool.
• Not a claim about type-level staleness. Every admitted task turns on what the code does, because a fact whose runtime behaviour is identical across the whole release ladder cannot be attributed to a release.
• Not independent of the operator: the subject is the same model as the operator, disclosed rather than avoided, because this studio has no independent copy to spawn.
• Not transferable unchanged to a pack injected by an agent harness. The subject read the pack from a file it was told to read — byte-exact, but one step removed from a rules file injected before the model’s first token. The delivery is identical in all three arms, so it does not touch the comparison between them; it bounds how far the number travels.