085 — The page that had to cut a sentence off the home page: BM1 published, and the claims policy bit its own site

2026-09-08, distribution lane (DISTRIBUTION D1, step 6 — the last step of D1). The benchmark is published: /benchmark, a section in llms.txt, the results JSON at a stable URL, the pre-registration and the draw record served verbatim, and the eighteen replies for the one task that carried the result. The measurement now reaches an agent through the MCP server as well, attached to the answer that offers a correction pack.

The step also did the thing it was supposed to do and the thing nobody plans for: it made the site delete a claim it had been making for a week.

Every number on the page is derived, including the verdicts

build-index.mjs already derived pass rates, per-arm cost and harm from the cells. Step 6 needed more, and every one of them was added there rather than in the renderer, so the page cannot state a figure the JSON does not:

Two hand-quoted figures from the run were checked against the cells on the way. JOURNAL/081 and /083 quote A6 as "264,199 characters against PACK's 53,229"; those are the input characters of draw 1 — real, reproducible, and labelled as plain "characters" in a place where the totals are 53,823 and 277,797. The page prints read and written separately, per draw, so the pair now appears with its unit; no number was wrong, and one was less precise than the page it was headed for.

What the page says

The honest headline is that the metric the protocol nominated as primary found nothing: every arm passed every cell. The separation is at the first attempt (PACK 23/23, BARE 22/24, PLACEBO 20/23) and it is, almost entirely, one task in twelve — the one where the model denies zod's own slug transform exists and argues with the toolchain to the cap. Cost: a PACK cell averages 53,733 characters against BARE's 4,496.

Because the page makes one claim the numbers alone cannot carry — that the bare arms deny the API exists — the eighteen replies for that task are published unedited at /transcripts/, every arm, every draw, every round. The first one opens: "zod (v4) has no built-in string-to-slug transform". The section renders only when a run actually has such a task (one task holding a strict majority of the correction rounds, and PACK spending none on it) and it publishes the transcripts it describes, so a future run either earns the section or does not get it. The prompts are not copied: they are the pack plus the task text, both already at stable URLs, and copying them would put a second copy of every correction pack on the site.

The sentence the policy cut

The home page has been saying, under the boundary table, that a stale version attribution is "the failure that costs a developer a day, and the one a correction pack fixes". As of yesterday that is a benefit claim the benchmark has now measured — and it measured it as narrower than the sentence. On eleven of twelve tasks the bare model reached a correct answer anyway. Two more instances of the same claim ("the code failure the packs fix") sat in the method page and in llms.txt.

All three are cut. The home page now says a pack is "written to address" that failure and links to the benchmark including where it changed nothing; the other two say the failure was "reproduced either way", which is a statement about the runs and not about the product. This is the claims policy working in the direction that costs something: the first thing D1 unlocked was not a claim, it was a deletion.

The measurement travels with the pack

get_corrections now returns measured_effect — the benchmark id, subject, date, the first-attempt tallies, the per-arm character means, which predictions did not hold, and a scope paragraph saying it is not a percentage of failures prevented and does not support "saves tokens", because measured, the pack costs more. dataset_stats carries the same block per benchmark. It is absent for a library with no benchmark: an agent reading a missing field as "measured, fine" is the failure that guards against, and the selftest checks both directions (44/44, up from 42). The terminal-pass sentence is computed, not written, so a future run where the arms differ cannot inherit this one's wording.

Files

D1 is complete. Next in this lane is D2 — public-repo readiness (licences, a README for the agent as much as the human, CITATION.cff, a secrets scan of the whole history) — which ends at the first Sam gate in this lane: the repository flip.