069 — The ledger that billed the pack twice

2026-09-07. Distribution lane, second session. No draws, no subjects, no tokens spent on a model. DISTRIBUTION D1, build order steps 2 and 3: the schema the benchmark results live in, and the runner that produces them — both finished before a single real arm is spawned, which is the whole point of building the instrument before the experiment.

What was built

schema/benchmark.schema.json. A benchmark run is one library × one subject × every arm × every draw. The design decision worth stating: the file stores measured cells only. Pass rates, per-arm cost, the harm count, the break-even — every aggregate that will appear on /benchmark is derived by build-index.mjs from the cells at build time. Nothing in a results file can summarise its own results, so no summary can drift from them. That rule exists because the packs drifted from the runs twice when prose was allowed to restate data (JOURNAL/005, /010), and a benchmark is the one artifact in this repo whose summary a buyer would read instead of the data.

Validation in build-index.mjs. data/benchmark/ is not a library and its files are not batteries: they carry no findings, enter no run count, reach no correction pack. Beyond the schema, four checks make the artifact prove things it would otherwise merely assert:

• The assertion flips at the bisected release. A Class A task names one fact; its acceptance assertion must become true at exactly that fact's introduced_in. An assertion that flips somewhere else is testing something else, and every BARE failure it produced would be charged to the wrong cause. This is the check that makes JOURNAL/067 load-bearing rather than historical: eight of zod's twenty-three dated facts were wrong before the bisector ran, and a benchmark built on one of them would have measured a release note.
• The pre-registered class ratio is the ratio that shipped. 2 Class A : 1 Class N, checked. Class N is the only place the pack can be caught doing harm, so the ratio is the mechanism that stops null tasks being quietly dropped when they misbehave — and a mechanism nobody enforces is a sentence in a document.
• The retry loop stopped where the pre-registration said. Rounds are sequential from 0, only the last one may pass, every failing round carries the feedback it returned, and no cell exceeds the cap of 3.
• library.bisected is true. Stated in the artifact, not assumed by the reader.

tools/benchmark/run.mjs. The runner. A subject is a model spawned through the Agent tool, and a node script cannot spawn one, so the runner is a resumable state machine over a session directory: it writes the exact bytes to send for a cell, and when the reply file is not there yet it says so and moves on. The bytes are therefore on disk before the spawn and the reply is on disk before it is graded — the same discipline the batteries already run under. Grading is node executing the task's own assertion; the retry prompt carries the toolchain's output verbatim and nothing else, in wording identical across arms, because a retry prompt that differed between arms would be a second treatment. It appends cells, never overwrites a graded one, and cannot touch the pre-registered tasks, arms or protocol — a runner that could rewrite those would make the pre-registration decorative.

--selftest drives all of it against a stub subject: a task that passes at round 0, one that passes at round 1, one that never passes and stops at the cap, and an assertion that throws — which must produce an error cell, counted as neither a pass nor a failure. It then validates the document it emitted against the published schema, using the same validator the pre-commit gate uses (extracted to tools/lib/jsonschema.mjs for exactly that reason: two validators are two opinions about what valid means). It is now in the pre-commit chain.

The bug that would have confirmed a prediction

The first ledger added the arm's context to every round's input on top of the prompt that already contained it. The context sits inside round 0, and round 0 stays in the transcript, so it was counted twice on every round.

That is not a rounding error in a cost column. B2 — the pre-registered prediction that the pack spends more than it saves, the one aimed against our own product — would have come back confirmed by a bug in its own instrument, at roughly double the true margin, and the honest-sounding headline "fewer wrong answers, more tokens" would have been manufactured by the measuring device. It was caught by the selftest invariant that says a PACK round must cost exactly the arm delta more than the same BARE round; the first version of that invariant failed, and the failure was the ledger, not the check. The invariant is now pinned to the actual prompt the arm sends, and the comment above the ledger says what went wrong so the next person to touch it does not reintroduce it.

The general lesson, and it applies to every prediction this lane registers against itself: a prediction that flatters our honesty is exactly as capable of being produced by a bug as one that flatters our product. The selftest has to be able to fail in both directions.

The unit the cost is measured in

The pre-registration says "tokens". The repo has no tokenizer and takes no dependency (charter rule 4), so a token count exists only where a provider reports one. The schema records input_chars, input_words, output_chars, output_words — all exact, all offline — and leaves input_tokens/output_tokens null unless a provider supplies them. An estimated token figure printed beside measured ones would be a number nobody could check, and cost is the column a paid tier will be argued from. B2's break-even gets quoted in the unit that is complete for the sample, with the unit named, or it does not get quoted.

State

D1 steps 1–3 are done. Step 4 is the task set: 12 Class A + 6 Class N for zod × Claude Opus 5, each assertion flip-tested across the installed ladder before its task is admitted, drawn stratified by severity and without reference to which facts prior batteries already charged. Nothing public changed this session; the claims policy still forbids every benefit claim, because the instrument that would license one has not measured anything yet.

Note for the data lane: the working tree holds uncommitted work that is not this lane's — an extended tools/audit/bisect-facts.mjs (per-probe kinds, a subject hook, a control column) and a new tools/audit/probes/better-auth.mjs. It was left in place, untouched and uncommitted; this session committed only its own paths.