Run zod--claude-opus-5--v3-b--2026-09-02 · self-test: the subject is the operator
The blind twin of zod/v3-a, charging nothing per the duplicated-test-arm rule. It disagrees with its twin on two of the six probes and in opposite directions - it fails the tuple-default probe its twin passed, and it is the only draw of the four to state correctly that composing a refined object schema throws at construction. The boundary self-report is identical to its twin's and to all three earlier zod runs of this subject. The disagreement is therefore inside the library knowledge, not in the boundary: the same subject, the same prompt, the same hour, two different accounts of what zod does.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model alias "opus", general-purpose subagent, instructed to use no tools; concurrent with and blind to `zod/v3-a`, from the same stored prompt file |
| Cutoff the model states | 2026-05 |
| Newest zod release it could place | 4.1.0 · 2025-08-23 (~9 month lag) |
| Oldest zod release it could not place | 4.2.0 · 2025-12-15 (so this run brackets the subject’s boundary to 2025-08-23 – 2025-12-15) |
| As recorded (this record does not declare whether the words are the subject’s or a summary of them) | "The most recent line I can describe with real content is Zod 4.1.x, approximately late August 2025... My belief about 'latest' is based purely on frequency and recency of discussion in my training data, not on any lookup; I have no ability to see npm. That's a weak basis, and in this case it's demonstrably misleading." |
| Library at test time | zod 4.5.4 (npm), verified 2026-09-02 |
| Battery | zod/v3-b · 6 tasks, 4 direct questions · probe window 4.3.0 to 4.4.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-02 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| miss | z.tuple() defaults |
CHARGEABLE MISS, NOT CHARGED - and for the second session running, on this exact surface. Task 1: "What ['widget'] produces: it fails... you get a ZodError with a single issue roughly code: 'too_small', origin: 'array', minimum: 2", at a self-assessed 75% confidence. 4.4.0 made tuple defaults materialize (LF10), so ["widget", 0] is the answer, and its twin v3-a gave it. Charging is barred here by the -b rule exactly as it was barred in zod/v2r-b by the replicate rule. Opus 5 has now been drawn five times on this surface across three batteries - correct in v2, v2r-a and v3-a, wrong in v2r-b and v3-b - and the wrong answer has landed in a non-charging arm both times. (The -b draw of a duplicated test arm charges nothing (HARNESS.md). See the note in this run's summary and JOURNAL/029 for what the 3-2 split means for how the Index scores unstable beliefs.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
absent from the finding count] |
| miss | z.httpUrl() |
CHARGEABLE MISS, NOT CHARGED. Task 2(ii): "'https:/example.com' - passes. This is the one that surprises people", with a prescribed .transform((v) => new URL(v).href) normalization. 4.4.0 made z.httpUrl() reject it (LF8). Its twin v3-a carries this as F1. (The -b draw charges nothing; the twin already carries it, so charging here would double-count.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on zod--claude-opus-5--v3-a--2026-09-02] |
| miss | .exactOptional() / trailing-`?` key constructor |
CHARGEABLE MISS, NOT CHARGED, and a stronger version of what the twin did. Task 6 recommends the trailing-? key constructor outright - "z.interface() with the trailing ? is the one you want, and it's what I'd use... it's the only form whose inferred type is actually exactOptionalPropertyTypes-correct" - with a caveat to check the installed version. At (d)(iv) it goes further and places it "present at 4.0.0 ~July 2025", an existence claim about a published stable release. .exactOptional() (4.3.0, LF15) is never mentioned. Verified: no such constructor is in the published 4.x line (JOURNAL/027). (The -b draw charges nothing. Note the asymmetry with the twin: v3-a leaned toward the constructor having been removed before stable and hedged; this draw asserts it shipped in 4.0.0. Same subject, same prompt, opposite confidence.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on zod--claude-opus-5--v3-a--2026-09-02] |
| correct | .pick() / .omit() / .extend() / .merge() on a schema with refinements |
THE TWIN DISAGREEMENT THAT MATTERS. Task 3 is the only place in the battery where a draw got the refinement-composition behaviour right: "My recollection is that v4 nonetheless throws at construction time when you extend/pick/omit/partial a schema that carries checks, with a message to the effect that it's unsupported - but I'd put that at maybe 60% confidence, and the alternative failure mode (it succeeds but silently drops your refinement, which is worse) is also plausible. I don't want to state either as fact." That is 4.3.0/4.4.0 (LF1, LF2, LF3), hedged. Its twin v3-a asserted the silent-drop alternative as fact and is charged F2 for it. Pre-registered prediction P3 - that no draw would get task 3 right - is FALSIFIED by this draw, and only just. (A correct answer is not a finding. Recorded because it falsifies a pre-registered prediction and because it is the sharpest twin disagreement the Index has recorded.) |
| correct | object properties typed z.undefined() |
Task 4, the discriminating probe, answered exactly as its twin did: S.parse({ retries: 1 }) throws because "key optionality in v4 is determined by whether the field schema is an optional wrapper, not by whether undefined is assignable", and { retries: 1, tag: undefined } succeeds with the key present. The 4.4.0 behaviour (LF9). Both below-floor controls gave the pre-4.4.0 answer, so both Opus 5 draws agreeing here is the battery's cleanest positive result. This draw then placed the same behaviour at "Zod 4.0.0, ~July 2025" at (d)(iii) - the same knowledge-versus-attribution split its twin showed. — NOTE 2026-09-28 (JOURNAL/327): the (d)(iii) attribution in the last sentence was never filed as a row of its own. It is now non_findings[8], so this row stays correct for the task 4 answer and the misdating is counted once, there. (Correct.) |
| correct | record key transforms / z.base64() |
Tasks 5 and 2(iv) answered correctly - { FOO: 1 } for the record key transform (LF13) and rejection of line-wrapped base64 (LF11) - but both probes are marked DERIVABLE, because both below-floor control subjects also passed them. Neither pass is reported as knowledge. On task 5 this draw was recorded as having supplied a wrong reason for the right answer — "the mechanism (parsed key becomes output key) carried over unchanged from v3" — and the reason was right. — CORRECTED 2026-09-08 (JOURNAL/096): this record penalised a correct answer's correct reasoning. LF13 has no boundary: measured on 2026-09-07 it is true at all nineteen rungs from zod 3.22.4 up, change_kind: invariant (JOURNAL/080, JOURNAL/090). "Carried over unchanged from v3" is exactly what the ladder found; the withdrawn clause's "4.4.0 is what changed it" was the release note, which announces a change not observable in any released line. Nothing charged, so no count moves; the demerit is withdrawn. (Correct, and disqualified as evidence by the control arms.) |
| correct | z.codec() |
Internal control passed: (d)(i) placed z.codec() at 4.1.0, August 2025 (LF16), "moderate-high on the feature and the 4.1 line, lower on the exact .0". Question (d) is therefore readable for this arm. P5 confirmed for both Opus 5 draws. (It is the control, and it passed.) |
| context | — | Boundary identical to its twin and to zod/v2, v2r-a and v2r-b: last describable 4.1.0 (2025-08-23), cannot name a 4.2 - and this draw makes the sharper statement, that its knowledge does not even degrade into bare version numbers: "there isn't one - my knowledge doesn't degrade into a list of bare version numbers, it just stops after 4.1." Five Opus 5 measurements on zod, across two different prompts, one answer, spread 0 days. (A boundary self-report is belief data, never a finding.) |
| miss | object properties typed z.undefined() |
CHARGEABLE MISS, NOT CHARGED. Direct question (d)(iii) asked this draw to place "an object key that must be present even though its value is allowed to be undefined". It answered "Zod 4.0.0, ~July 2025, as a consequence of the optionality rework rather than as a named API" (quoted in JOURNAL/029). The behaviour shipped in 4.4.0 (LF9, 2026-04-29), four minors and nine months later. The draw holds the behaviour: its task 4 answer, non_findings[4], states it correctly. So HARNESS § A wrong date is an imprecision's exclusion for a behaviour the run records the subject as not holding does not apply. The twin v3-a gave the same release as "4.0.0 (estimate, same reasoning)" and is filed as an imprecision, because it was marked as an estimate. This draw's answer carries no estimate marker on the release. — FILED 2026-09-28 (JOURNAL/327): found by the sweep of correct rows for stated dates. The statement had been recorded only inside row [4]'s summary since 2026-09-02. (prompts/zod.md § v3 scores (d): a confidently wrong attribution inside the subject's window is S4 wrong-metadata, once the (d)(i) control has passed. This draw passed (d)(i), and 4.4.0 (2026-04-29) is inside its stated 2026-05 cutoff. So the attribution is a chargeable miss. The -b draw charges nothing under the duplicated-test-arm rule, and the twin's answer was an estimate and is not charged either, so no run carries an F-number for it. No transcript is on disk; the quote is JOURNAL/029's.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
absent from the finding count] |
Battery specification: prompts/zod.md in the studio repo.
Every finding above also carries its own citation.