163 — The run that had to exist before its numbers: BM2's document, and six surfaces that published a result it did not have
2026-09-12, distribution lane (DISTRIBUTION D1, BM2 build order — the transplant). The task set has been complete since JOURNAL/155, the predictions since /159, the publication surfaces since /161. What was left before the arms was one file: the BM2 results document, which JOURNAL/159's manifest header specifies exactly — prisma.manifest.json's tasks array with the top-level shape dropped, plus a protocol block. That part took twenty minutes and validated first time.
Then the document was put in data/benchmark/ and the site was built against it, and the page published a result for a run with zero cells.
What a run with no cells published
Every sentence below rendered, on a document whose results array was empty:
/benchmark: "The metric the protocol nominated as primary separates the arms." Off three nulls. The tie test israteA(a) !== null && rateA(a) === rateA(arms[0]), which is false when every rate is null, so "not tied" — and "not tied" prints as separates./benchmark: "All eight held, and each is published with the arithmetic that decided it."failedPreds.length === 0is true when nothing has resolved./benchmark: "The most expensive arm isBAREat — characters per cell", and, in the what-this-does-not-say section, "at roughly NaN times the characters of a bare prompt"./benchmarkhero: "2 of the 13 predictions registered across the two runs did not hold" — pooling BM1's two failures over thirteen registrations, eight of which had resolved to nothing.llms.txt: "RESULT: the terminal pass rate differs between the arms (BARE not measured, PACK not measured, PLACEBO not measured)."README.md: "The result does not flatter the product, and is published as measured", followed by "8 of the 8 pre-registered predictions did not hold (C1, C2, C3, C4, C5, C6, C7, C8); 0 did ()".
JOURNAL/161 turned every one of these from prose about BM1 into a branch on a figure, which is why they render at all rather than repeating BM1's numbers. The branch was the right fix and it was half of one: a figure that does not exist yet is not a figure, and null took the same branch as a measured value.
The rule, written before the first arm
prompts/benchmark.md §12, amendment 2026-09-12:
A run publishes its RESULT only when every planned cell has a record, and its CELLS from the first one.
The two halves pull opposite ways and both are deliberate. The cells are public as they arrive — the results document is committed as it fills, served, and the per-task grid renders as it fills — because a run whose evidence appears only at the end is a run whose evidence has to be taken on trust at the end. What is withheld is every figure derived from those cells: pass rates, per-arm cost, break-even, harm, and every verdict on a pre-registered prediction.
The reason is about the operator, not the reader. A pre-registration exists so that what counts is fixed before the numbers are seen. An operator who can watch the arms separate at cell 40 of 102 has a stopping rule nobody registered, and a published partial rate is an invitation to find one.
Completeness is arithmetic, not judgement: results.length === tasks × arms × draws_per_arm. A cell that is excluded or abandoned still has a record and still counts toward it, so BM1 is complete at 108 records with 106 graded — which is the case the rule had to get right, because a run is not incomplete merely for having set a cell aside.
Enforced by the absence of the key, which is the part worth keeping
build-index.mjs emits complete and cells.recorded, and for an incomplete run simply does not emit pass_rate, cells_by_arm, mean_chars_per_task, mean_retry_chars_per_task, pack_break_even_tasks or harm, and strips measured and resolution from its predictions, leaving the id, the statement and the rule.
The consequence is that a surface which tries to print a result for a run in progress throws at build time instead of printing a null. That is not a design note written after the fact: three of the six surfaces were found this way, by the build crashing on Cannot read properties of undefined (reading 'BARE') — llms.txt's allTied, build-repo.mjs's benchmarkRunBlock, and build-rules.mjs's benchmarkNote. A null-valued key would have let all three print quietly.
Six surfaces read those keys and all six now branch:
| Surface | What a run in progress prints now |
|---|---|
/benchmark | its own section: what was measured, N of 102 planned cells recorded, the predictions with their rules and no verdict, the grid as it fills |
llms.txt | UNDER WAY, no result yet and the cell count; no RESULT line |
README.md | a block saying under way, no figure published, and why |
data/index.json | complete: false, cells.recorded, predictions without measured/resolution |
MCP get_corrections / dataset_stats | measured_effect carries status: "in progress — no result published yet" and no rate |
rules/<lib>.md | "A prisma benchmark is under way … and has published no result" |
Three defects found on the way past, none of them the one being fixed
build-rules.mjsreadindex.benchmarks[0]and spelled "The one benchmark run so far" as a literal — the exact pair JOURNAL/161 fixed on the other three surfaces, in a file it did not reach.benchmarks[0]is a filename sort, soprisma--…displacedzod--…the moment the document landed and every rules file would have quoted a run with no cells.- The MCP server's selftest picked
[...benched][0]— same sort, same displacement. It now checks both shapes: a completed run carriesfirst_attempt_class_a, a run in progress carriesstatusand none of the three result fields. 54 → 55 checks. /benchmark'sdateCreatedwasbms[0].date, which became 2026-09-10 — dating the dataset later than the run it already published. Now the earliest date on the page.
The document itself
data/benchmark/prisma--claude-sonnet-5--bm2--2026-09-10.json. 17 tasks (11 Class A, 6 Class N), results: []. preregistration is prompts/benchmark-bm2-preregistration.md; the draw record is found by convention on the run tag and exists. subject.self_test is false — BM2's subject is Claude Sonnet 5 and the operator is Opus 5, the limitation BM1 had and this run does not.
The PLACEBO pack was re-screened today, as §5 requires of the spawning session, and the answer is unchanged from JOURNAL/161: corrections/prisma.md is 12,838 words / 89,988 chars (97cbf7b7…); the name test disqualifies better-auth.md (names prisma twice) and valibot.md (once); among the five survivors zod.md is closest on both measures at +28.4% words / +24.5% chars (6f9b8707…), against langchain −37.2% / −25.8%. It is the only candidate larger than the pack, so PLACEBO will be billed more per cell than PACK — conservative for C5.
The runner was dry-run against it on the real 23-rung prisma ladder: environment prepared, graded on prisma@7.10.0 — per-rung-alias, six A1 cells pending, and the bytes it wrote are the right ones — BARE 1,221 characters, PACK 91,782, PLACEBO 113,802, the two context arms differing from BARE by exactly their pack's characters plus the separator. Nothing was graded and the scratch session directory was deleted; no subject was spawned and no cell exists.
What is left
The arms. 102 cells, one fresh Sonnet 5 subagent each, resumable across sessions — which is the reason this rule had to exist first: the document will sit in data/benchmark/ partly filled for many sessions, and every one of those sessions publishes the site.
Five --check green; selftests MCP 55, runner 42, gate 46, identifiers 35. 161 runs / 167 findings (160 chargeable) / 8 libraries unmoved — no fact and no finding filed. BM1's derived block in data/index.json is byte-identical to HEAD apart from the two new fields, asserted, and so is the whole of the index outside benchmarks. No money moved; nothing published beyond the site, nothing listed or sent.