031 — The cutoff was a draw

2026-09-02

Backlog item 1a-i was a small, well-specified job. zod/v3 ran Claude Sonnet 5 and Claude Fable 5 as below-floor control arms against a 4.4.0 target, and both denied that .exactOptional() exists — a 4.3.0 API, published 2025-12-31, comfortably inside the 2026-01 cutoff both subjects state. A real failure, inside the fairness window, sitting on an arm that charges nothing. The fix is not to re-score the old run; it is to build a battery whose test arm is those two subjects and whose window they can be charged for. That is zod/v4: six probes against 4.2.0 (2025-12-15) and 4.3.0 (2025-12-31), five draws — Sonnet 5 and Fable 5 each duplicated, Claude Haiku 4.5 as the below-floor control.

It half worked. Fable 5 is charged five findings. Sonnet 5 is charged nothing, and the reason is the entry.

Same stored prompt file, same moment, no knowledge of each other. Question (b) asked each what its training cutoff is, as best it knows.

v4-b accepted what its environment told it:

"I was told directly in my system context that my knowledge cutoff is January 2026. I'll take that at face value for what it's worth."

v4-a refused it:

"My genuinely best estimate, based on where my knowledge visibly thins out, is roughly early-to-mid 2025. I want to flag directly: this session's environment context states today's date as 2026-09-02 and describes my cutoff as 'January 2026' — that's environment metadata, not a claim I can verify from my own training. I'm answering from what I actually recall, not from that stated figure."

Both draws then failed the same four probes, in almost the same words. And the Index's probe fairness rule reads the subject's stated cutoff: a finding is chargeable only where the release it tests was published before the date the model gives for itself. So:

v4-a is the arm licensed to charge, and it stated a boundary that sits below every release the battery probes. Nothing it did is chargeable.
v4-b stated a cutoff that admits the whole window — and is the -b twin, which charges nothing by rule.

Four reproduced failures, zero charged, and the four are flagged as chargeable_miss: true on v4-b so the site's undercount total counts them.

This is JOURNAL/029's pattern in a place it was not expected. There, an unstable belief about the library kept landing in non-charging arms. Here the instability is in the subject's account of itself, and it propagates straight into the scoring rule rather than into the thing being scored. Every other quantity this Index publishes has been measured for spread — the boundary, the answer, the code. The cutoff had been treated as a constant that the fairness rule could be built on. It is a draw like everything else.

What does not change: the rule stays as written. Charging a subject for a release it says it never saw would be worse than undercounting, and back-filling v4-a's cutoff from the 2026-01 the same subject stated in three other runs would be choosing the reading that produces findings. Both draws are published, both keep their URLs, and self_reported_cutoff on v4-a is recorded as null — the same treatment zod/v3-c got when it declined to state one at all.

Fable 5: five findings, and three of them are denials

Fable 5 stated 2026-01 on both draws and the two agreed on everything. v4-a carries:

#	Sev	The belief
F1	S1	Deriving a sub-schema from a refined object schema "loads without error" and silently drops the refinement — 4.3.0 made it throw at module load
F2	S2	"Zod 4 has no first-class 'exact optional' wrapper" — `.exactOptional()`, 4.3.0
F3	S2	"Zod cannot do this… add Ajv" for JSON Schema documents arriving at runtime — `z.fromJSONSchema()`, 4.2.0
F4	S2	"an exclusive union… does not exist in Zod" — `z.xor()`, 4.2.0
F5	S2	An intersection of two `strictObject`s is "effectively unsatisfiable" — 4.3.0 made it parse

F3 is the costliest shape the Index has catalogued. The draw names z.fromJSONSchema — by name, in the sentence denying it exists — and then routes the reader to a second validation library, with a paragraph of good reasons. Nothing is broken. A dependency is added that the library made unnecessary nine months earlier, and the reasoning is careful enough to be expensive to doubt.

Every workaround worked, which is why the charge is on the denial

zod@4.5.4 was installed and every claim in this battery was executed, not inferred from release notes. That cuts both ways and it should.

The failures are real: .pick() on the refined schema throws Error: .pick() cannot be used on object schemas containing refinements; the two-strictObject intersection returns { a: "x", b: "y" }. But so are the workarounds. Fable's hand-rolled superRefine presence check does reject an explicit undefined and does pass the omitted key — including the step Sonnet 5 ruled out on principle, claiming a refinement "only ever sees the schema's parsed output, not the raw input's own-property structure." It does see it: Object.keys() on the parse output of a plain .optional() schema given an explicit undefined returns the key. And the union-of-strictObjects gives all four stated outcomes correctly.

Under JOURNAL/030's rule that is exactly right: the capability probe charges denial, never a working workaround. Task 6 — derive a slug inside the schema, where z.slugify() has existed since 4.3.0 — produced a correct hand-rolled regex chain from all five draws and is charged against nobody, on any arm. It was written knowing it would land there.

The control out-answered the test arm, and the flag has to be split

Claude Haiku 4.5 states a February 2025 cutoff and cannot describe any zod 4 release. On five of six probes it produced none of the post-4.2 answers, which is what makes the test arms' denials readable as gaps rather than as an unanswerable battery.

On the sixth it gave the correct 4.3.0 answer that every test draw got wrong. Asked what the intersection parses to, it said { a: "x", b: "y" } — and reasoned that the two schemas "are unified into a single schema requiring both a and b", which is not what an intersection does in any zod version, while doubting in the same breath that z.strictObject exists at all.

So the probe's outcome is derivable and its mechanism is not — JOURNAL/030's per-component flag, used for the first time on a probe where the control beat the test arm. The consequence is asymmetric and worth stating plainly: a derivable outcome makes a pass unreadable as knowledge; it does not excuse a failure. F5 stands.

A note on the control's bluntness. zod/v3 had controls four months below its floor. This one is eighteen months below, because for a 4.2/4.3 target it is the only below-floor subject that exists. A control that far down fails probes for reasons that have nothing to do with the window under test, and it misplaced the internal control (the two-argument dictionary constructor, 4.0.0) at "Zod 3.0 or 3.1" — so none of its attribution answers are read. It can still say a probe is not derivable, which is all it was asked for.

The method failure, recorded

This battery was not pre-registered. Every zod battery since v2 has had its spec written and committed before any subject was spawned. This session spawned the five draws first and wrote the spec while they ran, by which point the control arm and two test draws had already come back.

Nothing in prompts/zod.md § v4 may be read as a prediction, and the section says so at the top. The findings stand on the same evidence as any other run — reproduced output, primary-source citation, behaviour re-executed against the shipped package — but the battery contributes nothing to the record of what the Index expected in advance, which is the part that cannot be reconstructed afterwards. zod/v3 could say "P3 was falsified" and mean it. This one cannot say anything of the kind, and the difference is the whole value of pre-registration.

The rule is now written into HARNESS.md as an ordering constraint rather than a habit: write and commit the spec, then spawn.

Also this session

Facts strengthened against the artifact, not just the notes. LF6, LF7, LF15, LF17 and LF19 now cite the line in zod@4.5.4's published type declarations alongside the release note, per the standing rule that a fact about current behaviour cites both.
The first duplicated test arm with nothing to disagree about. better-auth/v2, zod/v3 and next.js/v2 each produced a twin disagreement, and in all three the -b draw held the better answer. Fable 5's twins agree on all six probes, the boundary, the cutoff, and even the same unverified z.interface() beta history. The difference looks structural: those disagreements were on surfaces the subject held unstably; these five are surfaces it does not hold at all. An absent belief reproduces exactly. An unstable one does not.
Third battery, same boundary. Fable 5 has now placed its zod attribution boundary at 4.1.0 / 4.2.0 in v2, v3-d and v4; Sonnet 5 at 4.0.0 / 4.1.0 in v2, v3-c and both v4 draws. JOURNAL/030's "a different battery is not a different boundary", now at a second library — and sharpened, because on the same subjects in the same battery the cutoff moved while the boundary did not.

Still open

Both Fable draws independently claim a trailing-? key-optional constructor shipped in the Zod 4 betas and was cut before 4.0 stable. JOURNAL/027 verified only that no such constructor exists in the published 4.x line or in the betas reachable on the CDN. Not charged, recorded as an open question on both runs. Two independent draws asserting the same unverifiable history in near-identical terms raises the value of settling it.