071 — The gate that could not see a trap

2026-09-07. Distribution lane, third session. No draws, no subjects, no tokens spent on a model. DISTRIBUTION D1, build order steps 4a and 4b: the tasks are drawn, and the gate that admits them was found to be blind to two thirds of them by the first real task written against it.

4a — the draw, run before any assertion existed

tools/benchmark/draw.mjs. Offline, reads data/index.json and data/zod/facts.json, writes nothing. It computes the eligible pool by §7's four rules — bisected, above the subject's measured boundary, below its stated cutoff month, executably decidable — and lands on 16 of zod's 31 facts for Claude Opus 5 (boundary 4.1.0, stated cutoff 2026-05), which is the number the pre-registration wrote down on 2026-09-07 before the tool existed. Quotas by largest remainder: S1 2, S2 7, S3 3. Order inside each stratum: sha256("<benchmark_id>:<fact_id>").

The hash is not decoration. §7 forbids building the benchmark out of the corpus's known failures, because a task set drawn from facts this subject has already been caught on would inflate BARE's failure rate by construction. An operator picking twelve facts by hand cannot demonstrate it did not do that. A hash of the fact's name can: it is blind to charge history, to file position, and — this is the part that matters more — to how easy the assertion looked once someone started writing one. Assertions are real work and the easy ones cluster on simple API surfaces, which is exactly where a model tends to be right, so a set assembled while writing them silently becomes "the twelve facts whose tests were pleasant". Drawing first makes that difficulty fall where it falls.

It fell somewhere useful. 4 of the 12 drawn facts had already been charged against this subject; the most-charged zod fact of all, LF1, came out third in S1 and went to the reserve. Both cuts — the whole sample and the previously-charged subset — are reported separately when the results land. The draw, the excluded facts with their reasons, the quotas and the reserve are in prompts/benchmark-bm1-draw.md, with the command to recompute it.

4b — and then the first real task did not fit the gate

prompts/benchmark.md §9, pre-registered this morning, says every acceptance assertion is run against all fifteen ladder releases and "must be false below introduced_in and true at and above it". Writing A1 — LF3, .merge() throws when the receiver carries refinements, 4.4.0 — showed that this describes only one kind of fact.

An assertion grades produced code, so it cannot be run against a release alone; something has to be produced. So each task now ships a committed reference solution, the operator's own answer, never shown to a subject. For an additive fact the reference flips exactly as §9 says: A6's answer uses z.string().slugify(), which does not exist below 4.1.13, so the module will not even load there and passes from it. Clean.

For LF3 it does not. The correct answer — keep an unrefined base and derive the variant from it, the shape the fact's own note prescribes — works on every rung of the ladder. Its assertion never flips. Nor does the stale answer's, in the direction §9 expects: .merge() on a refined receiver silently drops the refinement below 4.4.0 and throws from it, so an assertion that checks the refinement survived is false in both cases, for two different reasons. Neither artifact flips, and 11 of the 16 eligible zod facts are traps of this kind — now-throws, stricter, behavior-changed. The gate as written could admit the additive third and nothing else.

The two honest options at that point were both bad: admit trap tasks with no admission test, or drop every trap fact and rebuild the benchmark out of the easiest surfaces in the pool. Instead, the gate was made stricter, and dated before any task was admitted and before any arm ran:

A class A task is admitted when exactly one of two artifacts flips contiguously at the fact's bisected introduced_in. Additive: the correct solution fails below and passes from. Trap: the stale artifact — the code the fact's own stale_code shows a stale model writing — passes below and fails from. The correct solution is run too and must pass at and above the fact's release, which is what proves the task is answerable where the arms are graded.

The amendment is in prompts/benchmark.md §9 with its reasoning. A gate relaxed after a task fails it is not a gate; this one was tightened, and the record says when.

The second thing the flip test forced, which is the more valuable one. Under the new rule, A1 only admits if its acceptance check accepts what pre-4.4 .merge() produced — a schema with the refinement dropped. Which meant the task had to be rewritten to ask for that: derive a variant for an admin route where the cross-field rule deliberately does not apply. The first phrasing — "combine these and keep the rule active" — would have been admitted by nobody's gate and, worse, would have looked like a perfectly good LF3 task while actually testing "do not drop a refinement", a proposition that was already true two minors before 4.4.0. Every BARE failure it produced would have been charged to a fact that had nothing to do with it. The gate caught a mis-attributed task before it was written, which is precisely what it is for.

What is on disk

ADMIT  A1   A  LF3   trap      @4.4.0
       correct  +++++++++++++++
       stale    +++++++++++....
ADMIT  A6   A  LF17  additive  @4.1.13
       correct  ....+++++++++++

What is not claimed

Nothing. No arm has run, no subject has been spawned, and the claims policy is untouched: no benefit claim appears anywhere public and none can until BM1 has a number. Two of eighteen tasks exist.

Next

Step 4c: the remaining 16 tasks — 10 Class A and all 6 Class N. A2 (LF4) needs the tsc --strict grading path, since that fact is a type error with no runtime shadow (JOURNAL/067). Then step 5, BM1 itself.