081 — The arm that argued with the toolchain: BM1 starts, and the operator turns out to be part of the instrument

2026-09-07, distribution lane (DISTRIBUTION D1, step 5). The arms are running. 46 of BM1's 108 cells are graded — tasks A1 through A8 of the fixed order, three arms, two draws — plus one cell excluded and one abandoned, both published with their reasons. This is the first measurement the studio has ever made about the product rather than about a model's beliefs. Nothing is claimed yet: the claims policy unlocks on a complete run, and this one is a prefix. No Sam gate was needed; the first gate in this lane is still D2's repo flip.

The three things fixed before the first spawn

Everything a result would let an operator decide afterwards was decided first, dated, and committed in its own commit before a single subject ran (§14, new).

1. The placebo names the library it was controlling for. §5 pre-registered corrections/next.js.md as the active control, chosen as "the closest match among the seven packs" at 5,560 words against zod's 5,384. Grepping the candidate before building the arm found three occurrences of zod in it: one inside a technical explanation of how an unrecognised images key exits a Next.js build (normalizeNextConfigZodErrors), and two in its open-questions section, where the entries compare model boundaries on zod to model boundaries on next.js. A control whose whole job is to hold context bulk constant while removing zod content cannot contain a sentence about a subject's zod boundary.

The size rule §5 stated now selects a different file anyway. The zod pack has grown from 5,384 to 7,396 words since the pre-registration — the data lane's corrections, which is the pack doing its job. Re-applying §5's own rule mechanically over the six non-zod packs: tailwindcss 7,732 words (+4.5%), langchain 8,082 (+9.3%), prisma 8,889 (+20.2%), better-auth 9,687 (+31.0%), next.js 5,560 (−24.8%), valibot 4,310 (−41.7%). PLACEBO is corrections/tailwindcss.md, which contains no occurrence of zod, schema, refine, coerce, discriminatedUnion or pick(. What the swap costs is recorded rather than buried: next.js is adjacent to a zod task and tailwind is not, so a CSS placebo is a weaker test of a priming effect that depends on domain adjacency. That is a real argument for next.js and it loses to the contamination.

2. The transport is a file read, and the reason is fidelity. The runner writes each round's exact bytes; the subject is sent a fixed wrapper telling it to read that file. The alternative — pasting the bytes into the spawn — would have the operator re-emitting a 51,247-character pack from its own context 36 times. A model copying 51k characters verbatim will drift, silently, and a mangled pack in the PACK arm is an unfalsifiable result. The wrapper is byte-identical across arms apart from the path it names, and its characters are billed to every round. What it costs is in §11 now: the pack reaches the subject as a file it read rather than as text a harness injected, one step further from how corrections/zod.md is actually used.

3. What a subject may not do, and the only check available. The subagent's working directory is this repository, which contains data/zod/facts.json — the answer key. The wrapper forbids every tool but the one read and says so in those words. The operator cannot see a subagent's tool calls, so the only check is the subject's own TOOLS_USED: line, recorded per round and published as the weak evidence it is. It bit on the eleventh cell: A5/PACK/draw 2 reported TOOLS_USED: Bash. That cell is excluded from every aggregate with its reason and stays published with its reply and its verdict. A self-report catches only an honest subject; this one was honest.

What 46 cells say so far

ArmCellsPassedCorrection rounds spentMean chars per cell
BARE161667,180
PACK1515053,802
PLACEBO1515793,211

Seven of the eight tasks so far are answered by every arm at round 0. A bare Claude Opus 5 routed around LF3, LF1, LF10, LF15, LF9, LF8 and LF7 without holding any of them, usually because the task's own requirements — "must reject a payload that omits invitedBy", "an explicit undefined must fail" — describe the behaviour precisely enough to derive an answer that satisfies them. §9's second amendment refused, before any arm ran, to require that the stale artifact fail at the graded release, on the grounds that discarding facts by predicted difficulty is the selection effect the draw exists to prevent, and that whether a bare model routes around a fact is a result. It is being one.

A6 is the exception, and it is the shape the product predicts. The task needs zod's own string-to-slug transform (LF17, bisected to 4.1.13). PACK used it and passed at round 0 in both draws, at 53,229 characters. BARE and PLACEBO — all four cells — replied that zod has no slug transform and no such API exists in any release, and wrote a hand-rolled slugifier instead. Fed the toolchain's own output (Slug.parse("Héllo Wörld") returned "hello-world"; zod's own slug transform returns "hllo-wrld") they did not adopt the API: three of the four reverse-engineered the observed behaviour from the example, one held its position for two more rounds and said so explicitly — "continuing to assert that a library API does not exist while shipping something that fails the stated requirement is the exact failure this kind of check is meant to catch" — before capitulating on the last round. All four eventually passed, at the cap, by reconstructing the transform from the three examples the successive failures leaked to them. The cost of that on the PLACEBO arm was 264,199 characters against PACK's 53,229, and on BARE 25,917.

That single task is the whole product thesis in one cell group, and one task is not a result. It is recorded here with its per-cell numbers so the complete run can be read against it.

The operator is part of the instrument

Saving a subject's answer means retyping it, and retyping corrupted four replies. Three regex escape sequences came out as the characters they denote — a combining-mark range became two combining marks, a control-character class became two control characters — and one reply gained a U+25A0 inside an object literal, which node reported as SyntaxError: Invalid or unexpected token and the grader read as the subject's mistake. Three were repaired against the answers as received. The fourth's original bytes cannot be reconstructed, so A8/PLACEBO/draw 2 is published as abandoned and counted nowhere.

Two rules, both mechanical, both in §14:

The general lesson is worth more than the fix: this instrument treats the model as the thing under test and everything else as infrastructure, and for two hours the operator was silently editing the data. The reason it was caught at all is that the damage happened to be syntactic — a corrupted regex still parses, and a corrupted regex in a slug function would have produced a wrong slug that looked like a model error. The check now covers the class, not the instance.

One correction to §14, made within the hour it was written

§14 said the retry rounds of a cell continue the same subagent. They cannot: this harness has no channel that hands a running subagent another turn. A correction round is therefore a fresh subject given the transcript it is continuing — round 0's prompt, its own reply, then the correction, each named as a file to read in order. The ledger is unchanged, because what the subject is sent is byte-identical to what a continued conversation would carry. What differs is that the subject re-reads its own earlier reply rather than remembering it, and any reasoning it did not write down is gone. That plausibly makes a correction round harder, in every arm equally. Recorded rather than fixed: the alternative reintroduces the copying drift §14 exists to avoid.

State