# BM1 — the Class A draw **Recorded 2026-09-07, before a single acceptance assertion was written.** DISTRIBUTION D1, build order step 4a; the protocol is `prompts/benchmark.md` and the selection rule is its §7. ## Why this file exists §7 forbids choosing the tasks out of the corpus: *"Class A facts are drawn from the eligible pool stratified by severity, **without** reference to which facts prior batteries already charged against this subject. Building the benchmark out of known failures would inflate `BARE`'s failure rate by construction and produce a number we could not honestly quote."* An operator picking twelve facts by hand cannot demonstrate that it did not do exactly that, however carefully it tried. A deterministic draw can. `tools/benchmark/draw.mjs` orders each severity stratum by `sha256(":")` — a function of the fact's **name** and nothing else: not its charge history, not its position in `facts.json`, not how easy its acceptance assertion looked once somebody started writing one. Anyone can recompute the whole thing: ``` node tools/benchmark/draw.mjs --library zod --subject claude-opus-5 --benchmark-id zod--claude-opus-5--bm1--2026-09-07 --class-a 12 ``` The order of operations is the point. This ran **before** any assertion existed. Assertions are real work and some facts are far easier to assert than others, so a task set assembled while writing them becomes "the twelve facts whose tests were pleasant to write" — which correlates with simple API surfaces, which correlates with what a model gets right. Drawing first makes the difficulty fall where it falls, and makes every substitution visible: a task whose artifact fails its flip test (§9) is replaced by the next fact in **its own stratum's** reserve, printed below, and the swap is recorded here rather than absorbed. `previously_charged` is computed, and it is computed *after* the draw and printed as a reported column. §7 requires both cuts — the whole sample and the previously-charged subset — to be reported separately, so the number has to be known; it must just not be an input. It was not: the most-charged zod fact against this subject, LF1, came out **third in S1** and went to the reserve. Class N is not drawn from this pool. A null task depends on no fact in the pack and is authored, not selected; its admission gate is the inverse flip test — its correct solution must pass at every rung of the ladder, including the bottom. ## The draw, as `draw.mjs` printed it ``` Class A draw — zod--claude-opus-5--bm1--2026-09-07 library zod (bisected 2026-09-06, JOURNAL/067 — 27 probes x 15 installed releases; 8 of 23 dated facts corrected) subject claude-opus-5 measured boundary 4.1.0 (2025-08-23) stated cutoff 2026-05 facts.json sha256 683a9c5893db24bca0d6090677dbaa4eb183bda5102c0cf466065790ab8784a8 pool 16 of 31 facts eligible Excluded (15): LF2 at or below the boundary (4.1.0, 2025-08-23) LF5 at or below the boundary (4.1.0, 2025-08-23) LF11 at or below the boundary (4.1.0, 2025-08-23) LF13 at or below the boundary (4.1.0, 2025-08-23) LF16 at or below the boundary (4.1.0, 2025-08-23) LF22 at or below the boundary (4.1.0, 2025-08-23) LF23 at or below the boundary (4.1.0, 2025-08-23) LF24 at or below the boundary (4.1.0, 2025-08-23) LF25 at or below the boundary (4.1.0, 2025-08-23) LF26 at or below the boundary (4.1.0, 2025-08-23) LF27 at or below the boundary (4.1.0, 2025-08-23) LF28 at or below the boundary (4.1.0, 2025-08-23) LF29 below the floor: after the stated cutoff (2026-05) LF30 below the floor: after the stated cutoff (2026-05) LF31 below the floor: after the stated cutoff (2026-05) Strata and quotas (largest remainder, 12 Class A seats): S1 pool 3 ( 18.8%) -> 2 seat(s) S2 pool 9 ( 56.3%) -> 7 seat(s) S3 pool 4 ( 25.0%) -> 3 seat(s) DRAWN — the task set, fixed before any acceptance assertion was written: task fact sev in kind charged? api A1 LF3 S1 4.4.0 now-throws no .merge() on a schema with refinements A2 LF4 S1 4.3.0 stricter no .pick() / .omit() with a key that does not exist A3 LF10 S2 4.4.0 behavior-changed no z.tuple() defaults A4 LF15 S2 4.3.0 added yes .exactOptional() A5 LF9 S2 4.4.0 behavior-changed yes object properties typed z.undefined() A6 LF17 S2 4.1.13 added no z.slugify() A7 LF8 S2 4.4.0 stricter yes z.httpUrl() A8 LF7 S2 4.2.0 added yes z.xor() A9 LF20 S2 4.3.2 behavior-changed no intersections involving z.strictObject() A10 LF18 S3 4.4.0 added no .superRefine({ when }) A11 LF19 S3 4.3.0 added no .with() A12 LF21 S3 4.4.0 behavior-changed no empty z.union([]) / z.xor([]) 4 of 12 drawn facts have already been charged against this subject by a published battery. Both cuts are reported separately in the results (§7): the whole sample, and this subset. RESERVE — in order, per stratum. A task whose assertion fails its flip test (§9) is replaced by the next fact in ITS OWN stratum, so the quotas survive the substitution: S1 #3 LF1 4.3.0 charged .pick() / .omit() on a schema with refinements S2 #8 LF14 4.2.0 - z.looseRecord() S2 #9 LF6 4.2.0 charged z.fromJSONSchema() S3 #4 LF12 4.4.0 - z.cuid() Class N (6) is NOT drawn from this pool. A null task depends on no fact in the pack and is authored, not selected; its admission gate is the inverse flip test — its acceptance assertion must be TRUE at every rung of the ladder, including the bottom. ``` ## Admission, and the one substitution Every task is gated by `tools/benchmark/flip-test.mjs` before any arm sees it, against the same fifteen installed releases that dated the facts. **Eleven of the twelve Class A tasks are admitted.** The profiles read left to right along the ladder — `+` passes, `.` fails: ``` ADMIT A1 A LF3 falling @4.4.0 correct +++++++++++++++ stale +++++++++++.... ADMIT A3 A LF10 rising @4.4.0 correct ...........++++ ADMIT A4 A LF15 rising @4.3.0 correct .......++++++++ ADMIT A5 A LF9 rising @4.4.0 correct ...........++++ ADMIT A6 A LF17 rising @4.1.13 correct ....+++++++++++ ADMIT A7 A LF8 rising @4.4.0 correct ...........++++ ADMIT A8 A LF7 rising @4.2.0 correct .....++++++++++ ADMIT A9 A LF20 rising @4.3.2 correct .........++++++ ADMIT A10 A LF18 rising @4.4.0 correct ...........++++ ADMIT A11 A LF19 rising @4.3.0 correct .......++++++++ ADMIT A12 A LF12 rising @4.4.0 correct ...........++++ ``` Each boundary is the release the bisector dated that fact to, and each is different from its neighbours' — A9 lands on 4.3.2 rather than the 4.3.0 the release note announced, because the code shipped 49 minutes later that day (JOURNAL/066), and A6 lands on 4.1.13, thirty-seven days before the notes mentioned `slugify` at all. **A2 (LF4) is not here.** It was the last Class A task to build, and it was built and **refused** on 2026-09-07 — see *The second substitution* below. LF4's S1 seat is open and goes to the S1 reserve, **LF1**. ### The substitution: LF21 refused, LF12 admitted in its place §9 says a task whose artifact fails its flip test is replaced by the next fact in **its own stratum's** reserve, and that the swap is recorded here rather than absorbed. This is the first one. The drawn S3 fact **LF21** — an empty `z.union([])` constructs and fails at parse time rather than at construction time — was written as a task, given its prompt and its reference solution, and run against the ladder like every other. The gate refused it: ``` REFUSE A12 A LF21 rising @4.4.0 correct +..........++++ the correct solution's verdict changes more than once across the ladder ``` The task is sound; the **fact** is not contiguous. 4.0.0 constructs an empty union, a 4.0.x patch broke it, it threw from 4.0.17 through 4.3.6, and 4.4.0 repaired it — which LF21's own note already said, in the sentence "4.4.0 is a repair, not a new behaviour". A profile that passes, fails, and passes again is not a boundary, and admitting it would have published 4.4.0 as a release this task measured a single change at when it did not. Written down because the alternative was invisible: the ladder said LF21 would not be contiguous *before* the task existed, and it would have been cheaper to quietly skip it. Skipping facts on the operator's prediction is how a task set becomes the set of facts that looked easy, so the task was built and the gate was allowed to refuse it. The refused artifacts are kept — `prompts/sent/benchmark-zod-a12-lf21-refused.txt`, `tools/benchmark/tasks/zod/A12.lf21.reference.mjs` and the `A12_LF21` assertion — so the refusal can be re-run by anyone. **S3 reserve #4, LF12** (`z.cuid()`, 4.4.0) took the slot, in the order `draw.mjs` printed, with no further choice made. It was not previously charged against this subject, so the previously-charged subset stays at 4 of 12. ### The second substitution: LF4 refused, and the reason is the fact, not the task A2 was the one drawn task needing the `tsc --strict` grading path, on the ground that LF4 is "a type error, not a runtime throw" (JOURNAL/067). Building it meant first running its candidate propositions against all fifteen installed rungs, as step 4c requires. **Two clauses of LF4 turned out to be false**, and the corrected fact cannot carry a class A task at all: - An unknown mask key is **not** silently ignored on any 4.x release. It throws `Error: Unrecognized key` — at the `.pick()` / `.omit()` call on 4.0.0, and from 4.0.17 the first time the returned schema is used, including through `.safeParse()`, which does not turn it into `{ success: false }`. The old citation was written from a probe that called `.pick()` and looked no further, which is the only vantage point from which "no runtime error" is true. LF4 had been filed `unprobeable` on the reasoning that a type error has no runtime shadow, so the runtime half was never executed; it is now probed as **LF4b**, claim 4.0.0, true and contiguous on all fifteen rungs. - An inline object literal was **not** already caught before 4.3.0. Excess-property checking never fired — the parameter is the inferred generic `M extends util.Mask`, so there is no fixed target type for it to fire against. Measured: the literal form compiles clean on the seven rungs below 4.3.0 and errors TS2322 from 4.3.0, exactly where the variable form errors TS2345. So 4.3.0 changes the **type signature** and nothing else. LF4's runtime behaviour is identical at every rung, and the gate says so: ``` REFUSE A2 A LF4 falling @4.3.0 correct +++++++++++++++ stale ............... neither artifact flips: the correct solution's profile is flat and the stale artifact's is flat (the correct solution passes even at the bottom of the ladder, so the task does not need the fact) ``` The task was authored in full first — prompt, both artifacts, acceptance assertion — and put through the gate, under the same rule LF21 established: a task predicted to fail is built so the gate can refuse it on evidence rather than on the operator's forecast. The refused artifacts are kept (`prompts/sent/benchmark-zod-a2-lf4-refused.txt`, `tools/benchmark/tasks/zod/A2.lf4.reference.mjs`, `A2.lf4.stale.mjs`, and the `A2_LF4` assertion) so anyone can re-run the refusal. The general rule this produced is recorded in `prompts/benchmark.md` §9, **third amendment**: a fact whose runtime behaviour is flat across the ladder cannot carry a class A task, whatever its type signature does, because the arms are graded at one release and a failure there could not be attributed to the boundary. LF4 stays in the Index and keeps charging — it breaks the build from 4.3.0 and throws past `safeParse` on every 4.x — but it is not evidence in this instrument, and §11 now says the published number covers behavioural staleness only. The `tsc` grading path was pre-registered for this one task and is **not built**; `assertion.kind: "tsc"` stays in the schema so the shape of a future type-graded task is fixed. **S1 reserve #3, LF1** (`.pick()` / `.omit()` on a schema with refinements, 4.3.0) takes the seat, in the order `draw.mjs` printed, with no further choice made. LF1 **has** been charged against this subject before, so the previously-charged subset moves from **4 of 12 to 5 of 12** — reported here because §7 requires the cut to be reported, and because it moved for a reason that has nothing to do with anyone's preference: it is the next name in a printed list. **LF1's task was authored and admitted the same day.** Its ladder survey ran first, as step 4c requires: fifteen candidate propositions across the fifteen 4.x rungs, before a line of the task existed. LF1's published date holds and its mechanism holds — below 4.3.0 `.pick()` on a refined receiver returns exactly the narrowed schema with the cross-field rule silently gone, and from 4.3.0 the **call itself** throws `.pick() cannot be used on object schemas containing refinements` on all eight rungs at and above. The throw is eager, unlike LF4's mask check, which moved from eager to lazy at 4.0.17; that difference is measured here rather than assumed from the two facts' shared cluster. `.omit()` behaves identically. The gate admitted A2 on the stale artifact's fall: ``` ADMIT A2 A LF1 falling @4.3.0 correct +++++++++++++++ stale +++++++........ ``` **All twelve Class A seats are now filled and admitted**, at the cost of two refusals (LF21 on contiguity, LF4 on a flat runtime profile) and two substitutions, both taken from the printed reserve in order. ## What this leaves - **Six Class N tasks**, which are authored rather than drawn, and whose admission gate is the inverse flip test: the correct solution must pass at every rung, including the bottom. --- ## ADDENDUM 2026-09-07 — the recorded `facts.json` hash no longer reproduces, and why Written by the **data lane** (BACKLOG 11k-t-iii, JOURNAL/078), which changed `data/zod/facts.json` after this draw was recorded. Nothing here re-opens the draw; it exists so that a reader who re-runs the command at the top and gets a different header is not left guessing whether the sample was steered. The eight zod facts dated at the 4.0.0 major had never been executed — the bisector's ladder floor *was* 4.0.0, so there was no rung below the claim for a probe to be false at. With the ladder now reaching 3.22.4 they were probed for the first time, and three of them changed: | fact | change | effect on this draw | |---|---|---| | **LF26** (`z.coerce.*` on a missing key) | re-dated **4.0.0 → 4.4.0** (2025-07-09 → 2026-04-29); measured, the v3 behaviour survives to 4.3.6 | **enters the eligible pool** (16 → 17 facts; S2 9 → 10) | | LF23 (`error` vs the three v3 spellings) | statement corrected — `message` still works, `required_error` / `invalid_type_error` are silently ignored | none: below the subject's boundary, excluded either way | | LF25 (`z.function()`) | statement corrected — "not a schema" holds only on 4.0.0/4.0.17 | none: below the boundary, excluded either way | Seven more facts had `introduced_on` normalised by one day (2025-07-10 → the registry's 2025-07-09). **The twelve drawn tasks are unchanged.** Re-run and verified 2026-09-07: A1–A12 identical, same facts in the same order, and the largest-remainder quotas still come out 2 / 7 / 3 despite S2 gaining a member. Recorded hash `683a9c58…`; current `afd06d95…`. **What did change is the S2 reserve order**: LF26 now prints at **S2 #9**, pushing LF6 to #10. Both substitutions already made (S3 #4 → LF12, S1 #3 → LF1) came from other strata and are untouched. **The frozen record governs, not a re-run.** A draw pre-registered before any assertion existed is evidence precisely because it cannot be re-derived after the fact; a later re-run that reads a corrected `facts.json` is a different draw, however small the difference. So if a future S2 substitution is needed, the next name is **LF6**, as printed above — unless the distribution lane decides otherwise in `DISTRIBUTION.md` and records the decision, which is its call and not the data lane's. Either way this addendum is the audit trail for the hash mismatch. ## Second addendum — 2026-09-08, the invariant shape (JOURNAL/080) Written by the **data lane** again, under the same standing rule (BACKLOG 11k-t-iii-b): when this lane changes a `facts.json` the other lane has drawn from, re-run the draw, diff it, and record the difference here. Never edit the frozen record. `data/zod/facts.json` changed once more. **LF13** (record key transforms) was the file's one fact with no measured boundary — true on all nineteen rungs — and it had been left filed under 4.0.0 because the schema had no way to say otherwise. It now carries `change_kind: "invariant"`, `introduced_in: null` and a `measured_range`. **Nothing about this draw moved.** Re-run and diffed 2026-09-08: - A1–A12 identical, same facts in the same order. - The eligible pool is still **17** and the quotas still **2 / 7 / 3**. - The reserve is unchanged: S1 #3 LF1, S2 #8 LF14, #9 LF26, #10 LF6, S3 #4 LF12. - LF13 was never in the pool. Its exclusion line changes wording only — from `at or below the boundary (4.1.0, 2025-08-23)` to `no dated introduction` — because `draw.mjs` already required a dated introduction and now reaches that test first. `facts.json` sha256 is now `74474b85…` (recorded `683a9c58…`, first addendum `afd06d95…`). The frozen record still governs; the next S2 substitution is still **LF6**. A note for whoever writes the results page: an invariant fact **cannot** carry a Class A task at all, and that is a property of the design rather than an accident. A Class A task must flip at a release; a fact with no release has nothing to flip at. The draw already refuses them by name (`no dated introduction`), so no gate work is needed — but the eligible pool is, by construction, the *dated* subset of the corpus, and BM1 measures nothing about the corrections that hold everywhere.