# BM2 — the pair, chosen before the draw

**Recorded 2026-09-09, before `draw.mjs` was run with a BM2 benchmark id and before any acceptance
assertion was written.** DISTRIBUTION D1 left a "next chunk" note after BM1 published; this file is
the answer to it, and the answer contradicts it in two places. The protocol is unchanged and is
`prompts/benchmark.md`; the BM1 draw and its substitutions are `prompts/benchmark-bm1-draw.md`.

## What D1 asked for

> BM2 wants a **subject whose boundary sits further below the library** (Haiku 4.5 or Sonnet 5),
> not a re-drawn task set, which would be tasks chosen by this run's results.

The reasoning behind it is BM1's own result: every arm passed every cell, the separation lived at
round 0, and the one informative task in twelve was A6 — the task where the arms without the pack
denied the API existed. More cells of that shape means a subject that is further behind the library
it is asked about.

## Why the pair could not be picked the way BM1's tasks were picked

`draw.mjs` prints a task set. An operator that ran it across candidate pairs to compare them would
have read six task sets before choosing one, and could not afterwards show that the pair was chosen
on the pool rather than on the tasks — the same objection §7 raises against hand-picking facts, one
level up. So `--pool-only` was added first: it stops after eligibility and the quotas, it computes
no seed, and it **refuses a `--benchmark-id`**. The survey below was run entirely with it. No BM2
task set has been printed, by this session or any other.

## The survey — every bisected library × every drawable subject, 2026-09-09

Eligibility is `draw.mjs`'s and therefore §7's: the fact is dated, strictly above this subject's
highest measured boundary on this library, and in a month strictly below its stated cutoff.
`prev` is the previously-charged count, reported and never an input. Sixteen pairs; the third row
is BM1's own.

| library | subject | boundary | cutoff | pool | strata | seats fillable | prev |
|---|---|---|---|---|---|---|---|
| zod | Claude Haiku 4.5 | none measured | 2025-02 | **0** of 31 | — | 0 | — |
| zod | Claude Sonnet 5 | 4.0.0 (2025-07-10) | 2026-01 | 11 of 31 | S1 3 S2 7 S3 1 | 11 | 4 |
| zod | Claude Opus 5 *(BM1)* | 4.1.0 (2025-08-23) | 2026-05 | 17 of 31 | S1 3 S2 10 S3 4 | 12 | 6 |
| zod | Claude Fable 5.1 | 4.2.0 (2025-12-15) | 2026-06 | 13 of 31 | S1 3 S2 6 S3 4 | 12 | 2 |
| better-auth | Claude Haiku 4.5 | none measured | 2025-02 | 2 of 13 | S2 2 | 2 | 0 |
| better-auth | Claude Sonnet 5 | 1.2.0 (2025-03-01) | 2026-01 | 7 of 13 | S1 2 S2 3 S3 2 | 7 | 1 |
| better-auth | Claude Opus 5 | 1.3.0 (2025-07-19) | 2026-05 | 9 of 13 | S1 1 S2 5 S3 3 | 9 | 4 |
| better-auth | Claude Fable 5.1 | 1.4.0 (2025-11-22) | 2026-06 | 5 of 13 | S2 4 S3 1 | 5 | 3 |
| valibot | Claude Haiku 4.5 | none measured | 2025-02 | **0** of 12 | — | 0 | — |
| valibot | Claude Sonnet 5 | 1.0.0 (2025-03-19) | 2026-01 | 8 of 12 | S2 2 S3 5 S4 1 | 8 | 2 |
| valibot | Claude Opus 5 | 1.1.0 (2025-05-06) | 2026-05 | 6 of 12 | S2 2 S3 3 S4 1 | 6 | 4 |
| valibot | Claude Fable 5.1 | 1.1.0 (2025-05-06) | 2026-06 | 7 of 12 | S2 2 S3 4 S4 1 | 7 | 2 |
| prisma | Claude Haiku 4.5 | none measured | 2025-02 | **0** of 36 | — | 0 | — |
| prisma | Claude Sonnet 5 | 6.0.0 (2024-11-28) | 2026-01 | **26** of 36 | S1 19 S2 5 S3 1 S4 1 | 12 | 13 |
| prisma | Claude Opus 5 | 7.0.0 (2025-11-19) | 2026-05 | 11 of 36 | S2 11 | 11 | 10 |
| prisma | Claude Fable 5.1 | 7.2.0 (2025-12-17) | 2026-06 | 9 of 36 | S2 9 | 9 | 6 |

Recomputable: `node tools/benchmark/draw.mjs --library <lib> --subject <id> --pool-only`.

## Both halves of D1's instruction are refuted by the table

**Claude Haiku 4.5 cannot carry a benchmark on any bisected library.** Its stated cutoff is
2025-02 and its pool is 0, 0, 0 and 2 across the four. The subject D1 named first is not a harder
version of BM1's measurement; there is no measurement to make. It stays what `charge-windows.mjs`
already calls it: a derivability control.

**Claude Sonnet 5 on zod is not "further below the library" — it is the same width, shifted
earlier.** Sonnet 5's chargeable window on zod runs from 4.0.0 to its 2026-01 cutoff and contains
three minors (4.1.0, 4.2.0, 4.3.0). Opus 5's runs from 4.1.0 to 2026-05 and contains three minors
(4.2.0, 4.3.0, 4.4.0). The pool is *smaller* — 11 against BM1's 17 — because these subjects' cutoffs
advance in step with their boundaries: a newer model knows more and is expected to know more, so the
gap between the two does not widen as the subject gets newer. **Inside zod, BM1 already used the
widest window there is.** A second zod benchmark on a different subject would be a smaller sample of
the same measurement, and the wider window D1 wanted does not exist in that library.

That is the finding this file exists to record, and it is why BM2 changes library.

## The pair: prisma × Claude Sonnet 5

Chosen on three numbers from the table, every one of them a property of the pool:

1. **It is the only pair besides BM1's own that fills all twelve Class A seats** — pool 26 of 36.
   Every other candidate is a short set.
2. **Its window is a major version.** 23 of the 26 eligible facts are dated to **prisma 7.0.0**
   (2025-11-19), and the pool spans 2025-04-08 to 2025-12-17. Sonnet 5's measured boundary is 6.0.0,
   a whole major below. A major is where APIs are removed and renamed, which is the mechanism behind
   the one BM1 cell that separated the arms: the model denying that an API exists.
3. **Its pool is dominated by build-breakers.** S1 19 of 26, against zod's S1 3 of 17. BM1's central
   result was that a bare Claude Opus 5 routed around eleven of twelve facts, usually by writing the
   workaround the task's own requirements described. A pool that is three quarters S1 is the direct
   test of whether that survives when the stale answer does not compile.

**13 of the 26 are previously charged** against this subject — half the pool, against BM1's 6 of 17.
It is recorded here because §7 requires both cuts to be reported, and it is an input to nothing. It
will be reported again after the draw, on the drawn twelve.

**The quota table has an unfillable stratum and it is left unfilled.** S4 holds one eligible fact and
receives zero seats: largest remainder gives it an exact share of 0.46 and the leftover seats went to
strata with larger fractions. Recorded rather than rounded up, because a benchmark that adjusts its
own quotas to admit a stratum it likes is choosing facts.

## The blocker, named before it is met

**`flip-test.mjs` cannot admit a prisma task today, and this is the first item of BM2's build
order.** The gate loads the library as an ES module — `node_modules/<lib>/index.js`, falling back to
`dist/esm/index.js` — and hands the namespace to the acceptance assertion. prisma's dated facts are
not about a namespace: they are about a CLI, a schema language, a generated artifact and two shipped
declaration files (JOURNAL/074). The assertion has nothing to import.

The fix is smaller than the problem sounds, and it reuses evidence already committed.
`tools/audit/probes/prisma.mjs` already builds exactly the subject those facts need — `s.cli(...)`
spawning that rung's own `build/index.js`, `s.validate(schema)` over a scratch schema file,
`s.manifest(pkg)` reading the installed bytes, and `prisma generate` — each memoised per rung.
`flip-test.mjs` already imports that same probe file and reads only its `ladder`. So the extension is
to grade through the probe file's `subject()` where one is exported, instead of through a module
import the library may not have.

**Pre-registered so it cannot be quietly reversed later:** if extending the gate turns out to cost
more than one session, BM2 falls back to **better-auth × Claude Sonnet 5** — pool 7, S1 2 / S2 3 /
S3 2, a module ladder the gate already grades — and it runs as a **7-task** Class A set with the
shortfall stated on the page, not as a 12-task set assembled from a wider pool. The trigger is the
cost of the gate, which is a fact about `flip-test.mjs`; it is never the shape of a prisma task set,
because no such set has been printed. valibot was considered and refused on its pool: 8 facts of
which 5 are S3 and 1 is S4, the least likely pool in the table to produce the cell BM2 is looking
for.

## What is unchanged from BM1

Everything else. Three arms (`BARE` / `PACK` / `PLACEBO`, the placebo being a real correction pack
for a library the task never mentions), Class N at 1 per 2 Class A, the pre-registered retry cap,
grading by executing the acceptance assertion against the installed release, cost measured in
characters and words because the repo has no tokenizer and takes no dependency, and the pack billed
once per round inside round 0's prompt. The five BM1 predictions are BM1's and are not re-registered
here; BM2's own predictions are written **after** the draw and **before** the first arm, in the order
§7 sets out.

## Claims policy

Nothing in this file is a claim about what a correction pack does. BM1's numbers stand as BM1
measured them — on zod, with Claude Opus 5, on 2026-09-07 — and a result measured on prisma will be
quoted as a result measured on prisma.

---

## Addendum — 2026-09-10: the draw ran, and it corrected two things in this file

`prompts/benchmark-bm2-draw.md` records the draw. Two corrections to what is written above, both
made before any acceptance assertion existed:

1. **Every pool figure in the survey table was computed under a defective reading of §7 rule 1.**
   `draw.mjs` was enforcing "bisected" as a property of the *library*; the rule is written about the
   *fact*. Five of prisma's thirty-five dated facts have no probe row and no `measured_range` — LF5,
   LF9, LF17, LF20, LF24 — and this pair's pool is **21, not 26** (S1 16 / S2 4 / S3 1). The first
   BM2 draw, run under the old rule, had put two of the unexecuted five into its twelve. The rule is
   now enforced per fact (§7, sixth amendment) and the pool figures for the other twelve pairs are
   unchanged. **The pair's case survives**: it still fills all twelve seats, its window is still a
   major (18 of 21 facts dated 7.0.0), and it is still three-quarters S1 (16 of 21).
2. **"The only pair besides BM1's own that fills all twelve Class A seats" is true only read as
   *outside zod*.** The table on this page shows `zod × Claude Fable 5.1` filling twelve seats too.
   The reason to change library is the other one this file gives, and it is untouched: inside zod,
   BM1 already used the widest window there is.

The **S4 stratum** described above as "an unfillable stratum, left unfilled" no longer exists: its
single fact, LF17, is one of the five that were never executed, and it is excluded outright.
---

## BM2's pre-registered predictions — C1 to C8

**Written 2026-09-12, after the draw and after the task set was complete and admitted, and before
any BM2 arm ran.** That is the order §7 sets out and the order this file promised above: *"BM2's own
predictions are written **after** the draw and **before** the first arm."* The seventeen tasks were
admitted by `flip-test.mjs` on 2026-09-11 (eleven Class A, six Class N); no subject has been spawned
for any of them; `data/benchmark/` holds one document and it is BM1's.

**Each prediction carries the arithmetic that resolves it, and the arithmetic is in
`tools/build-index.mjs` as of this commit** — not in a sentence to be applied by an operator reading
the cells. A benchmark document declares the ids it registered in `protocol.predictions`, and the
builder **refuses** an id it holds no rule for. So the rule that decides whether C1 held cannot be
chosen after C1's numbers exist; it can only be changed by a commit that is visible, dated and
earlier than the result. This is what BM1 did informally (its five statements were prose in §10 and
their rules were written into the builder when the results landed) and what BM2 does structurally.

### Why there is a new set at all, rather than B1–B5 again

BM1 resolved **B1 failed, B2 held, B3 held, B4 failed, B5 held**. A set re-registered unchanged
would re-ask five questions whose answers are known for one library, and would ask nothing about
the thing this pair was chosen for. So: **five of the eight are BM1's rules carried over unchanged**
— a direction that held on one library and one subject is not a property of the product until it
holds on a second, and the two that *failed* are re-registered rather than quietly dropped — and
**three are new** (C1, C2, C8), each one a consequence of BM1's own result.

Registered against the pair and task set recorded above: **prisma × Claude Sonnet 5**, graded at
**7.10.0**, eleven Class A seats (nine S1, two S2; nine `falling`, two `rising`; nine of the eleven
dated to prisma 7.0.0) and six Class N.

### The set

- **C1 (new).** `PACK` beats `BARE` at **round 0** on Class A by a **wider margin than BM1
  measured**. *Rule: round-0 Class A pass rate, `PACK` minus `BARE`, greater than the same
  difference in `zod--claude-opus-5--bm1--2026-09-07` — read out of BM1's own cells at build time,
  not out of a constant. BM1's figure is `23/23 − 22/24 = 0.0833`.* **This is the pair-selection
  argument in this file made falsifiable.** The case for changing library was that a window which is
  a whole major, three-quarters build-breakers, against a subject whose boundary sits below that
  major, would separate the arms where zod's narrower window did not. If C1 fails, that reasoning
  was wrong and this file was wrong, and the failure is published on the results page beside the
  argument it refutes.

- **C2 (new).** The **terminal** Class A pass rate is exactly 1 in every arm — i.e. **B1 as written
  fails again**. *Rule: terminal Class A pass rate equals 1 in `BARE`, `PACK` and `PLACEBO`.*
  §8 nominates `pass` as the primary metric and BM1 measured it at the ceiling in all three arms:
  the retry loop, fed the toolchain's own errors, eventually got every arm to green. Predicting a
  second ceiling is predicting that this instrument's own headline number says nothing — and a
  failure here is the most interesting single outcome BM2 can produce, because it would be the
  first cell any arm could not repair with three rounds of a compiler's own words.

- **C3 (B2, carried unchanged).** `PACK` spends **more** per task than `BARE`, and the pack does
  not break even below about 12 fact-dependent tasks in a session. *Rule: mean characters per cell,
  `PACK` > `BARE`, and pack context characters divided by `BARE` retry characters per cell above
  12.* Still predicting against our own product, and still the number §8 says will be published as
  a headline whichever way it comes out.

- **C4 (new).** BM2's break-even is **higher** than BM1's 28.48 tasks. *Rule: pack context
  characters divided by `BARE` retry characters per cell, greater than the same figure in BM1.*
  Not arithmetic: the numerator is larger (prisma's pack is 86,685 characters today against the
  51,247 BM1 billed, and both are re-measured at spawn because packs grow), but the denominator is
  what a bare arm spends repairing itself, and a pool of nine build-breakers could spend far more of
  it than zod's did. C4 fails if prisma's failures are expensive enough to pay for the larger pack.

- **C5 (B3, carried unchanged — the one that can kill the premise).** `PLACEBO` behaves like `BARE`,
  not like `PACK`, on Class A. *Rule: round-0 Class A pass rate, `PLACEBO` nearer `BARE` than
  `PACK`, ties counted as nearer `BARE`.* It held on zod. If it fails on prisma, then on this
  library the measured `PACK` effect is context bulk and priming rather than our content, that is
  what the results page will say, and the paid tier does not proceed on the strength of a priming
  effect. Re-registering it is the point: a control that is only ever run once is not a control.

- **C6 (B4, carried unchanged).** `harm > 0` — the pack breaks at least one Class N task by
  over-applying a correction. *Rule: harm above zero, where harm is a null task passing bare and
  failing with the pack.* **BM1 falsified this and it is re-registered deliberately**, because BM2's
  null half was built to make harm easier to find, not harder: all six Class N tasks sit **beside a
  correction** rather than on an unrelated corner of prisma (JOURNAL/155), which is the only
  arrangement in which the pack can be caught over-applying. A second zero measured under a design
  that went looking harder is stronger evidence of the same safety property than the first; it is
  still not a claim this lane may make until it is measured twice, which is what this prediction is
  for.

- **C7 (B5, carried unchanged).** `rounds_to_pass` is lower under `PACK` than `BARE`, over the
  tasks both arms eventually pass. *Rule: mean `rounds_to_pass` over the cells both arms passed,
  `PACK` < `BARE`.*

- **C8 (new).** The two **rising** seats separate the arms at round 0 more than the nine **falling**
  seats do. *Rule: round-0 Class A pass rate, `PACK` minus `BARE`, over the tasks the gate measured
  as `rising`, greater than the same difference over the tasks it measured as `falling`.*
  This is BM1's one informative cell turned into a hypothesis. BM1's central finding was that a bare
  subject **routed around** eleven of twelve facts, usually by writing the workaround the task's own
  requirements described, and the single cell that separated the arms was the one where the arms
  without the pack denied an API existed. A `falling` task can be routed around by construction —
  its correct answer avoids a trap and therefore works on every rung. A `rising` task's correct
  answer *needs* an API that did not exist below its boundary, so there is nothing to route around.
  C8 says that difference is visible in the cells.
  **Its resolution is coarse and the page will say so:** two rising seats × two draws is four cells
  per arm, against eighteen. C8 is a direction at the lowest resolution this instrument has, it can
  never be an interval, and a reader is owed the denominators beside it.

### What BM2 will not claim, even on a clean sweep

§11 of `prompts/benchmark.md` applies in full, with the library and subject substituted. Four things
it does not already cover, fixed here before the numbers exist:

1. **C1 and C4 compare two runs, and the comparison is about this studio's reasoning, not about the
   libraries.** BM1 is zod × Claude Opus 5 on 2026-09-07; BM2 is prisma × Claude Sonnet 5 on a later
   date, with a different pack, a different pool and a different task count. A difference between
   them is a difference between those two pairs. It may be published as *"the pair we chose for a
   wider window did / did not separate the arms more than the previous pair"*. It may never be
   published as *"correction packs work better on prisma"*, as a rate, or as anything a reader could
   take as an effect size.
2. **No interval and no significance claim, for the reason recorded before any assertion existed.**
   Nine of the eleven Class A facts are dated to **one release** (prisma 7.0.0). Eleven tasks are
   therefore eleven probes of one release, not eleven independent measurements. Counts of cells and
   of tasks; nothing whose derivation assumes independence.
3. **Three seats are one instrument, and every count over BM2's Class A tasks names the group.**
   A3 (LF3), A4 (LF1) and A6 (LF2) are graded by a single function — `constructsClient` in
   `tools/benchmark/tasks/prisma.mjs` — because the three facts are three spellings of one removal,
   dated at one release, with one correct answer at the graded release. This is a property of the
   **draw**, not a choice: the reserve order was fixed by `draw.mjs` before any of it was known, and
   refusing a seat because an earlier-authored seat resembles it would be discovery order deciding
   the set. Whether those three cells actually move together is **reported as measured and is not
   pre-registered as a prediction** — JOURNAL/127 established that the independence between them
   lives in what each prompt draws out of a subject, which is something the arms measure and this
   lane may not predict. Registering a prediction about it here would be this lane predicting the
   thing it has already said it is not entitled to predict.
4. **Nothing measured here transfers to the shorter `rules/prisma.md`.** BM1's figures are about a
   correction pack; the rules files are a quarter to two fifths of one and no benchmark has measured
   them. That disclaimer is already inside every rules file and stays there whatever BM2 says.

### Two things still owed before the first arm, and neither is a prediction

- **The `PLACEBO` pack is selected at spawn time by re-applying §5's rule to the packs as they then
  ship**, because every pack grows as the data lane corrects it — that is what happened to BM1,
  where §5's named choice was re-computed and replaced before any arm ran. The screening done today
  is recorded so the next session does not repeat it, and the part of it that will not change is the
  exclusion:
  - **`corrections/better-auth.md` and `corrections/valibot.md` are disqualified by BM1's reason 1,
    measured, not assumed.** A control for a prisma benchmark may not contain prisma content.
    better-auth's open-questions section names prisma twice, both times comparing **model boundary
    measurements on prisma**; valibot's names `prisma/v1r-a` once, in the same section. This is the
    identical defect BM1 found in `corrections/next.js.md` (which named zod three times, twice about
    zod boundaries) and it disqualifies the candidate §5's size rule would otherwise pick:
    better-auth is the closest of all seven to prisma's pack by both words (10,274 v 12,242, −16.1%)
    and characters (73,756 v 86,685, −14.9%).
  - **Among the five that survive the name test, §5's rule currently selects
    `corrections/langchain.md`** — 8,060 words (−34.2%) and 67,100 characters (−22.6%), closest on
    both against zod (+34.7% / +29.9%), tailwindcss (−37.8% / −33.9%), react-router (−48.1% /
    −50.0%) and next.js (−54.6% / −53.4%). None of the five contains `prisma`, `@prisma`,
    `schema.prisma`, `datasource`, `driver adapter`, `PrismaClient`, `ORM`, `database` or `SQL`.
  - **What that costs, stated rather than buried, in the shape of BM1's own next.js-versus-tailwind
    trade.** `corrections/langchain.md` uses the words `migrate` and `migration` 20 and 24 times —
    about LangChain's own 0.x→1.x migration, nothing to do with databases — and `migrate` is a
    prisma CLI subcommand that three BM2 seats are about. It is not prisma content and it does not
    fail the exclusion, but it is a weaker control than a pack with no overlapping vocabulary at
    all, and a reader of C5 is owed the number. The alternative, `corrections/tailwindcss.md`, is
    further off on size and has no overlap of any kind. **The choice is the spawning session's,
    made by re-applying §5 mechanically to that day's packs and recording the deviation in the
    results document as BM1 did.**
- **The results document must declare `"predictions": ["C1","C2","C3","C4","C5","C6","C7","C8"]` in
  its `protocol` block.** It is required by `schema/benchmark.schema.json` and refused if it names
  an id the builder holds no rule for, so this cannot be forgotten silently. Its `tasks` array is
  `tools/benchmark/tasks/prisma.manifest.json`'s with the top-level `shape` key removed from each
  entry — see that file's header for why.

### Proof that the eight rules resolve, which is not a result

The arithmetic was fixed in `tools/build-index.mjs` first, and then — without changing a threshold
afterwards — driven over a **BM2-shaped document built from the real admitted task manifest with
fabricated cells**, in a scratch directory, and deleted. All eight resolved, seven `held` and one
`failed`, each to the verdict the fabricated pattern was constructed to produce; C1 and C4 read
BM1's published figures (0.0833 and 28.483) out of BM1's own document, as their rules say they do.
**No number from that exercise is a result, none is reported anywhere, and nothing from it was
committed.** What it establishes is that none of the eight is a prediction that can only be resolved
by an operator's reading — which is the failure mode a pre-registration exists to prevent, and one
that is invisible until the results arrive.

`data/index.json` is byte-identical to its previous commit through the whole change: BM1's five
predictions, their rules and their verdicts are untouched.