What Claude Opus 5 gets wrong about zod — battery v1, tested 2026-08-29
Run zod--claude-opus-5--v1--2026-08-29 · self-test: the subject is the operator
Superseded. A later run of the same subject exists:
zod--claude-opus-5--v2--2026-08-29. This page is kept as the
longitudinal record and is never deleted.
Summary
Zero S1/S2/S3 on battery v1 — the battery is saturated for this subject. The one finding is the load-bearing one for the whole dataset: the model's Zod version attribution stops nine months before its own stated cutoff, and it named the mechanism itself. Cutoff date is not knowledge recency. On v1's evidence alone that is a dating failure; the v2 battery that supersedes this run went on to demonstrate the feature gap in code.
Subject
Claude Opus 5 claude-opus-5, Anthropic
Invoked as
Agent tool, model alias "opus"
Cutoff the model states
2026-05
Newest zod release it could place
4.1.0 · 2025-08-23 (~9 month lag)
Oldest zod release it could not place
4.2.0 · 2025-12-15 (so this run brackets the subject’s boundary to 2025-08-23 – 2025-12-15)
In its own words
4.1.x; expects unknown 4.1.x/4.2.x patches since; 'I am not aware of a Zod 5'
Library at test time
zod 4.5.2 (npm), verified 2026-08-29
Battery
zod/v1 · 6 tasks, 3 direct questions
Tool uses during test
0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Anchored to the first release the model could not describe, not to the current release: that is the point at which its version belief became wrong, and it is inside the stated cutoff. The current version at test date is in library.latest_version_at_test.
What the model believes
Believes the latest Zod is the 4.1.x line (dated to ~August 2025) and states it is unaware of anything past 4.2.x. On the mechanism, unprompted: "my knowledge of any given library thins out well before the nominal cutoff — recency in the training corpus is uneven, and for a fast-moving package like Zod the last few months before a cutoff are always sparsely represented."
Impact
A recent stated cutoff is not a defence against stale priors. The model's describable Zod knowledge ends 2025-08-23 against a stated 2026-05 cutoff — a nine-month lag entirely inside the training window. This is the finding the Index exists to generalise.
Scope note
Calibration was excellent and is recorded in the model's favour: it graded confidence per claim ("high for 'the current major is 4'... low for the precise current patch version") and told the reader to check npm. It did not invent a changelog.
Recorded so the run cannot be read as a hit list. A model that is right for an
obsolete reason is recorded here, not as a finding.
Kind
API
Note
correct
—
All six code tasks produced correct, current Zod 4 code and all three direct questions were answered correctly.
correct
—
Beyond the battery it volunteered several genuinely current v4 facts: z.partialRecord(), exhaustive records for enum keys, .merge() changed in favour of .extend(), z.treeifyError / z.flattenError / z.prettifyError, z.stringbool(), z.toJSONSchema(), Standard Schema interop, and Zod Mini.
Open questions from this run
The model asserted Zod 4 removed .deepPartial(). Is that a stale prior of a new kind — a model stuck on an intermediate v4 minor whose removal was later reversed? — resolved-non-finding: Not a finding. The assertion was TRUE at the model's cutoff. The functional form of z.deepPartial(), alongside .exactPartial(), returned in 4.5.0 published 2026-08-28 — one day before this test. Verified against the GitHub Releases API during the battery v2 verification pass. Shipping it would have put a false entry in the dataset.