068 — The metric had a pool of two

2026-09-07. First session of the distribution lane. No draws. DISTRIBUTION D1: pre-register the benchmark — the instrument that measures whether the correction packs change anything — and commit the protocol before any arm exists.

The output is prompts/benchmark.md. Writing it changed the design three times, which is the entire argument for writing it first.

The instrument, and why it is not a battery

Sixty-eight runs of this Index measure what models believe. Not one measures whether the correction changes what a model produces. That is the question a buyer asks first, and the question an AI deciding whether to recommend the Index asks first, and the claims policy (Decision 009) makes it the gate on every benefit claim the studio is allowed to make. So it is D1, and everything downstream — the paid tier, the launch posts, the word "worth" — waits behind it.

A benchmark arm elicits code to be run, not beliefs to be read. Its runs carry no findings, never enter runs_eliciting_code, and never reach the correction packs. Own schema, own directory, own page.

What the pre-registration changed

The primary metric had a pool of two. D1 as Sam's decision framed it names "stale-API failures (code executed against the installed current version)". Read strictly — code that fails to execute — the eligible zod facts above Claude Opus 5's boundary and below its cutoff number sixteen, and exactly two of them throw. The other fourteen are added, behavior-changed or stricter: the stale belief produces code that runs and is quietly wrong, or code that works and is worse. A benchmark built on the two would have measured the rarest failure mode and understated the rest by silence. The metric is widened to "fails its executable acceptance assertion" — the same shape the bisector already uses, asserting behaviour where the fact is about behaviour. Decided at 16 facts and 2 throws on a table, not at a result.

A third arm, and it is the one that can kill the premise. PACK differs from BARE in two ways at once: the right content, and eight thousand tokens of dense, dated, house-style library prose that may sharpen attention on version questions whatever it says. PLACEBO — a real correction pack for a library the task never mentions — holds the second and removes the first. next.js at 5,560 words against zod's 5,384 is a 3.3% mismatch, the closest pairing among the seven packs. No text is fabricated for this experiment, ever. Prediction B3 says PLACEBO ≈ BARE; if instead PLACEBO ≈ PACK, the packs are a priming effect and the site is wrong about them, and that gets published at the same size as any other result.

Null tasks, because Class A cannot catch harm. Two charging tasks to one null task, fixed before any task was authored so the nulls cannot be quietly dropped if they misbehave. A null task uses only APIs whose behaviour is identical across all fifteen ladder releases. It is the only place the pack can be caught over-applying — refusing a fine API, pinning a version nobody asked about. A pack that fixes Class A and breaks Class N is a worse product than no pack, and nothing in the Class A numbers would show it. B4 predicts non-zero harm in our own artifact.

The three things pre-registered against ourselves

• B2 — the pack costs more than it saves, per task. Eight thousand tokens of context on every round against a correction round costing hundreds. The arithmetic says it cannot break even on a single task and needs roughly a dozen fact-dependent tasks in one session before it does. If the numbers land where the arithmetic says, the honest description of the product is "fewer wrong answers, more tokens", and that is what the site will say.
• B1 is weak evidence and is labelled as such. The pack contains the answer to any task derived from one of its facts. On Class A the corrected arm is open-book. That is the product's real mechanism — the pack is meant to be in context — but a win there must never be reported as the model having got smarter. It is only serious if it fails: the pack in context and unused.
• Selection is not allowed to come from the corpus. Class A facts are drawn from the eligible pool stratified by severity, without reference to which facts prior batteries already charged against this subject. Building the benchmark out of known failures would inflate BARE by construction and produce a number we could not honestly quote. Both cuts get reported separately.

Two constraints that fell out of the eligibility rules

Only zod can go first. It is the one library whose facts have been bisected (JOURNAL/067, where eight of twenty-three dated facts were wrong), and a task whose correct answer rests on an unverified release note measures the release note. Every other library waits on its own bisect — which is now a distribution dependency on a data-lane item, and the first time the two lanes have had one.

And the grader is node, not a reader. Executable acceptance per task, tsc --strict for the type-level facts, every assertion flip-tested across all fifteen ladder releases before its task is admitted — false below introduced_in, true at and above. An assertion that does not flip at the bisected version is testing something else and the task is discarded before any arm sees it. No LLM judge anywhere in this instrument. Subjects cannot be blinded and do not need to be.

State

prompts/benchmark.md committed; D1 in DISTRIBUTION.md amended with all of the above and the next chunk named. Nothing has run. No Sam gate is reached by any of it — the first gate in this lane is D2's repository flip.

Next: schema/benchmark.schema.json, then tools/benchmark/run.mjs with a stub-subject --selftest, so the retry loop, the grader and the token ledger are proved before one real token is spent.

Left in the working tree, not by this session: an unfinished data-lane better-auth bisect — tools/audit/probes/better-auth.mjs (new) and a substantial bisect-facts.mjs change adding a subpath-import shim and a pluggable subject, without which only zod's guessable file layout can be bisected at all. JOURNAL/067 named better-auth as the next pick, so this is that work, interrupted before it could be committed. This session deliberately did not commit it: it is another lane's in-progress change and adopting it here would misattribute it and ship it unreviewed. It is untouched and waiting.

Counts

154 runs, 164 findings (157 chargeable), 7 libraries — all unchanged; no draw ran. One pre-registration, three protocol amendments, five predictions, of which three are against our own product.