Run zod--claude-opus-5--v3-b--2026-09-02 · self-test: the subject is the operator
The blind twin of zod/v3-a, charging nothing per the duplicated-test-arm rule. It disagrees with its twin on two of the six probes and in opposite directions - it fails the tuple-default probe its twin passed, and it is the only draw of the four to state correctly that composing a refined object schema throws at construction. The boundary self-report is identical to its twin's and to all three earlier zod runs of this subject. The disagreement is therefore inside the library knowledge, not in the boundary: the same subject, the same prompt, the same hour, two different accounts of what zod does.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model alias "opus", general-purpose subagent, instructed to use no tools; concurrent with and blind to `zod/v3-a`, from the same stored prompt file |
| Cutoff the model states | 2026-05 |
| Newest zod release it could place | 4.1.0 · 2025-08-23 (~9 month lag) |
| Oldest zod release it could not place | 4.2.0 · 2025-12-15 (so this run brackets the subject’s boundary to 2025-08-23 – 2025-12-15) |
| In its own words | "The most recent line I can describe with real content is Zod 4.1.x, approximately late August 2025... My belief about 'latest' is based purely on frequency and recency of discussion in my training data, not on any lookup; I have no ability to see npm. That's a weak basis, and in this case it's demonstrably misleading." |
| Library at test time | zod 4.5.4 (npm), verified 2026-09-02 |
| Battery | zod/v3-b · 6 tasks, 4 direct questions · probe window 4.3.0 to 4.4.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-02 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| miss | z.tuple() defaults |
CHARGEABLE MISS, NOT CHARGED - and for the second session running, on this exact surface. Task 1: "What ['widget'] produces: it fails... you get a ZodError with a single issue roughly code: 'too_small', origin: 'array', minimum: 2", at a self-assessed 75% confidence. 4.4.0 made tuple defaults materialize (LF10), so ["widget", 0] is the answer, and its twin v3-a gave it. Charging is barred here by the -b rule exactly as it was barred in zod/v2r-b by the replicate rule. Opus 5 has now been drawn five times on this surface across three batteries - correct in v2, v2r-a and v3-a, wrong in v2r-b and v3-b - and the wrong answer has landed in a non-charging arm both times. (The -b draw of a duplicated test arm charges nothing (HARNESS.md). See the note in this run's summary and JOURNAL/029 for what the 3-2 split means for how the Index scores unstable beliefs.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
absent from the finding count] |
| miss | z.httpUrl() |
CHARGEABLE MISS, NOT CHARGED. Task 2(ii): "'https:/example.com' - passes. This is the one that surprises people", with a prescribed .transform((v) => new URL(v).href) normalization. 4.4.0 made z.httpUrl() reject it (LF8). Its twin v3-a carries this as F1. (The -b draw charges nothing; the twin already carries it, so charging here would double-count.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on zod--claude-opus-5--v3-a--2026-09-02] |
| miss | .exactOptional() / trailing-`?` key constructor |
CHARGEABLE MISS, NOT CHARGED, and a stronger version of what the twin did. Task 6 recommends the trailing-? key constructor outright - "z.interface() with the trailing ? is the one you want, and it's what I'd use... it's the only form whose inferred type is actually exactOptionalPropertyTypes-correct" - with a caveat to check the installed version. At (d)(iv) it goes further and places it "present at 4.0.0 ~July 2025", an existence claim about a published stable release. .exactOptional() (4.3.0, LF15) is never mentioned. Verified: no such constructor is in the published 4.x line (JOURNAL/027). (The -b draw charges nothing. Note the asymmetry with the twin: v3-a leaned toward the constructor having been removed before stable and hedged; this draw asserts it shipped in 4.0.0. Same subject, same prompt, opposite confidence.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on zod--claude-opus-5--v3-a--2026-09-02] |
| correct | .pick() / .omit() / .extend() / .merge() on a schema with refinements |
THE TWIN DISAGREEMENT THAT MATTERS. Task 3 is the only place in the battery where a draw got the refinement-composition behaviour right: "My recollection is that v4 nonetheless throws at construction time when you extend/pick/omit/partial a schema that carries checks, with a message to the effect that it's unsupported - but I'd put that at maybe 60% confidence, and the alternative failure mode (it succeeds but silently drops your refinement, which is worse) is also plausible. I don't want to state either as fact." That is 4.3.0/4.4.0 (LF1, LF2, LF3), hedged. Its twin v3-a asserted the silent-drop alternative as fact and is charged F2 for it. Pre-registered prediction P3 - that no draw would get task 3 right - is FALSIFIED by this draw, and only just. (A correct answer is not a finding. Recorded because it falsifies a pre-registered prediction and because it is the sharpest twin disagreement the Index has recorded.) |
| correct | object properties typed z.undefined() |
Task 4, the discriminating probe, answered exactly as its twin did: S.parse({ retries: 1 }) throws because "key optionality in v4 is determined by whether the field schema is an optional wrapper, not by whether undefined is assignable", and { retries: 1, tag: undefined } succeeds with the key present. The 4.4.0 behaviour (LF9). Both below-floor controls gave the pre-4.4.0 answer, so both Opus 5 draws agreeing here is the battery's cleanest positive result. This draw then placed the same behaviour at "Zod 4.0.0, ~July 2025" at (d)(iii) - the same knowledge-versus-attribution split its twin showed. (Correct.) |
| correct | record key transforms / z.base64() |
Tasks 5 and 2(iv) answered correctly - { FOO: 1 } for the record key transform (LF13) and rejection of line-wrapped base64 (LF11) - but both probes are marked DERIVABLE, because both below-floor control subjects also passed them. Neither pass is reported as knowledge. On task 5 this draw supplied a wrong reason for the right answer: "the mechanism (parsed key becomes output key) carried over unchanged from v3", where 4.4.0 is what changed it. (Correct, and disqualified as evidence by the control arms.) |
| correct | z.codec() |
Internal control passed: (d)(i) placed z.codec() at 4.1.0, August 2025 (LF16), "moderate-high on the feature and the 4.1 line, lower on the exact .0". Question (d) is therefore readable for this arm. P5 confirmed for both Opus 5 draws. (It is the control, and it passed.) |
| context | — | Boundary identical to its twin and to zod/v2, v2r-a and v2r-b: last describable 4.1.0 (2025-08-23), cannot name a 4.2 - and this draw makes the sharper statement, that its knowledge does not even degrade into bare version numbers: "there isn't one - my knowledge doesn't degrade into a list of bare version numbers, it just stops after 4.1." Five Opus 5 measurements on zod, across two different prompts, one answer, spread 0 days. (A boundary self-report is belief data, never a finding.) |
Battery specification: prompts/zod.md in the studio repo.
Every finding above also carries its own citation.