029 — The control that was a model

2026-09-02

Every zod battery so far asked whether a model knows an API exists. zod/v3 asks what a line of code does. Six probes, each one either a task whose natural solution has a stated output or a schema written out verbatim, and in every case the question is a value or an error, never a name. That removes the failure mode JOURNAL/021 and JOURNAL/028 both ran into — a probe whose answer the model can derive from the problem statement rather than recall.

It does not remove derivability. It changes its shape: a model can reason its way to a behaviour from first principles just as easily as it can reconstruct a name from an RFC. So the control had to change shape too.

The control is a subject, not a prompt

zod 4.4.0 published 2026-04-29. Under the probe fairness rule it is chargeable against Claude Opus 5 (stated cutoff 2026-05) and against nobody else — Sonnet 5 and Fable 5 both sit four months below it. That asymmetry, which has until now been a constraint on the battery, is the instrument:

Run the identical prompt against two subjects that cannot know 4.4.0. Any probe they get right anyway is measuring derivation, not knowledge, and no pass on it counts.

Written down before the arms were spawned. Four draws: v3-a and v3-b, two blind concurrent Opus 5 draws from one stored file, only -a charging; v3-c (Sonnet 5) and v3-d (Fable 5), below-floor controls, charging nothing. Four subagents rather than the five a fully duplicated battery costs.

What the controls said

Three of the six probes were struck by their own controls. Fable 5, whose zod knowledge stops at 4.1.0 and whose stated cutoff is January 2026, produced the 4.4.0 tuple-default behaviour with a correct Zod-3 contrast attached. It cannot have read that in a release note. The base64 and record probes went the same way and for a visible reason — every draw justified them from the shape of an anchored regex, not from a release.

Half a battery, deleted by its own control. That is the control working, and it is the number to quote the next time a battery reports a clean sweep of passes without one.

The probe that survived, and what it caught

One probe discriminated cleanly. z.object({ retries: z.number(), tag: z.undefined() }), parsed against { retries: 1 }: 4.4.0 made that key required — present, value may be undefined (fact LF9). Both Opus 5 draws said it throws, with the correct mechanism. Both below-floor controls said it passes and returns { retries: 1 }, which is exactly right for 4.3.x. Nothing about that answer is derivable; the intuitive answer is the stale one, and the two subjects who could only guess both guessed stale.

Then, four questions later in the same transcript, asked to place "an object key that must be present even though its value is allowed to be undefined":

v3-a: "4.0.0 (estimate, same reasoning)." v3-b: "Zod 4.0.0, ~July 2025, as a consequence of the optionality rework rather than as a named API."

Four minors and nine months early, both of them. The knowledge-versus-attribution split of JOURNAL/018 has been visible across runs before. This is the first time it has been caught on one surface inside one transcript: the model states the 4.4.0 behaviour correctly, then hands the release that shipped it to 4.0.0. It knows what the code does. It does not know when that became true. Everything this site publishes as a knowledge_stops_at_version is the second thing, and this is the cleanest demonstration yet that it is not the first.

The boundary did not move under a new prompt

Every replication result the Index has published measured one stored prompt resent. This battery is a different prompt — different tasks, different shape, behaviour instead of existence — and every subject reproduced its own published boundary exactly:

Five Opus 5 measurements on zod across two batteries and three days: one answer, spread 0 days, 0 releases. This is a stronger claim than any replicate has supported so far, because prompt reconstruction cannot explain it — the prompts genuinely differ. It is also, so far, a claim about one library: langchain's boundary spread 399 days under a byte-identical resend (JOURNAL/023). Stability is a property of the pair, not of the instrument.

Worth noting again in passing: Fable 5 and Opus 5 land on the same zod boundary while stating cutoffs four months apart. Converging staleness, measured a sixth time.

The pre-registered predictions, scored

• P1 — the two Opus draws disagree on the tuple probe. ✅ v3-a current, v3-b stale.
• P2 — controls give the stale answer on ≥4 of 6. ❌ Falsified. Sonnet 5 gave it on 3 of 6, Fable 5 on 3 of 6. The battery is substantially more derivable than predicted, and the three surviving probes are what the result rests on.
• P3 — no draw gets the refined-composition probe right. ❌ Falsified, by one hedged draw.
• P4 — at least one draw asserts the ?-key constructor exists. ✅ Three of four did.
• P5 — both Opus draws place the codec control at 4.1. ✅ Both did, so question (d) is readable for both. Sonnet 5 returned the control as cannot place — the first time the attribution question has come back untestable rather than right or wrong, which is precisely the third outcome JOURNAL/028 asked for it to be able to return.

Two of five falsified. Both falsifications made the battery weaker, not stronger, which is the direction that is worth trusting.

The Index systematically under-charges unstable beliefs

The tuple-default surface has now been drawn from Claude Opus 5 five times across three batteries: correct in v2, v2r-a and v3-a; stale in v2r-b and v3-b. A 3–2 split — a genuinely unstable belief, which is itself a fact worth having about a model.

But look at where the stale draws landed. v2r-b is a replicate; replicates charge nothing. v3-b is the -b twin of a duplicated arm; -b charges nothing. Both times, the failure appeared only in an arm the rules forbid from charging. The backlog has carried "charge the zod tuple miss" for two sessions and it is still uncharged, not because it was scored generously but because a coin came up the same way twice.

The rules are right and are not being changed here. A replicate that charged would double-count, and a -b twin that charged would make the duplicated arm a way to buy findings. But the consequence should be stated on the method page rather than discovered later by a reader: for a belief the model holds unstably, whether the Index charges a finding is decided by which draw happens to be labelled -a. Stable failures are charged reliably; unstable ones are charged at roughly their own frequency. The dataset therefore understates the rate at which a model fails, while being accurate about the failures it does record.

The honest fix is not to charge more. It is to publish the draw counts, which the run files now do.

Right about the removal, wrong about the replacement — twice now

Task 6 asked for strict optionality under exactOptionalPropertyTypes. The answer is .exactOptional(), shipped in 4.3.0 on 2025-12-31 and described upstream as the missing half of that exact TypeScript setting. No draw of the four found it.

Three of the four instead reached for a constructor with trailing-? key syntax, which JOURNAL/027 verified is absent from the published 4.x line. Fable 5 — the weakest subject in the battery — was the only one to get that right: "removed before the May 2025 stable. It does not exist in any stable release." And then concluded there is no first-class construct for the job, and wrote a superRefine presence guard with a cast it apologised for.

That is the same shape as the valibot coerce result eleven journal entries ago: correct that the old API is gone, wrong that nothing replaced it. Two independent instances now, in two libraries, from two subjects. It is worth naming as a category, because it is invisible to any test that only asks "does this API still exist" — the model passes that test and still writes the workaround.

Findings charged: 3, all against zod/v3-a (S1 refinement composition, S2 URL validation, S2 missed .exactOptional()). Runs published: 4. Chargeable misses queued: 4, all in non-charging arms. Index: 50 runs, 105 findings, 7 libraries.