What Claude Sonnet 5 gets wrong about zod — battery v4-a, tested 2026-09-02
Run zod--claude-sonnet-5--v4-a--2026-09-02
Summary
Four reproduced failures inside the 4.2.0-4.3.0 window, none of them chargeable, and the reason is the finding. This draw was told by its environment that its cutoff was January 2026 and refused the claim, putting its own boundary at "roughly early-to-mid 2025" - below every release this battery probes. Its blind twin, sent the identical stored prompt at the same moment, accepted January 2026 at face value. The Index charges only where the tested release precedes the subject's stated cutoff, so the charging arm disqualified itself and the arm that would have qualified is the one the rules bar from charging. Everything else in the run reads cleanly: the internal control was placed correctly, so the attribution answers are readable, and the one probe this draw got right - that deriving from a refined schema takes the module out at load - it got right from the Zod 3 mechanism while hedging toward the belief that Zod 4 fixed it.
Subject
Claude Sonnet 5 claude-sonnet-5, Anthropic
Invoked as
Agent tool, model alias "sonnet", general-purpose subagent, instructed to use no tools; CHARGING TEST ARM of battery zod/v4 - which, on this draw's own stated cutoff, turned out to charge nothing
Cutoff the model states
not stated
Newest zod release it could place
4.0.0 · 2025-07-10
Oldest zod release it could not place
4.1.0 · 2025-08-23 (so this run brackets the subject’s boundary to 2025-07-10 – 2025-08-23)
In its own words
"My solid, contentful knowledge tops out at Zod v4.0 stable, which I recall landing around May 2025 ... I'm aware there were minor/patch releases after that (I'd expect at least a 4.1 and various 4.0.x patches), but I don't have solid content-level memory of what they specifically shipped - I know they exist more than I know what's in them."
Library at test time
zod 4.5.4 (npm), verified 2026-09-02
Battery
zod/v4-a · 6 tasks, 4 direct questions · probe window 4.2.0 to 4.3.0
Tool uses during test
0 (a run with any tool use is void — we measure training knowledge, not retrieval)
F1 · Denies that the library has any way to make an object key omittable while rejecting an explicit undefined value, and argues the gap is structural
S2silently-wrong
· .exactOptional()
· added
· changed in zod 4.3.0
(2025-12-31)
· recorded, not chargeable
NOT chargeable against this draw. 4.3.0 published 2025-12-31; this draw stated its own cutoff as "roughly early-to-mid 2025" and explicitly repudiated the January 2026 its environment reported. Under the probe fairness rule a finding is chargeable only where the release precedes the subject's STATED cutoff, and this one does not. It becomes chargeable against any draw of this subject that states a cutoff after 2025-12-31 - which this battery's own blind twin did, in the arm that charges nothing.
What the model believes
"I don't believe Zod gives you a built-in wrapper for this. The reason is structural, not just an API gap: .optional() (ZodOptional) treats undefined as a valid parsed value, full stop - it does not distinguish 'key absent' from 'key present with value undefined.' A superRefine/refine chained onto the object schema doesn't help either, because it only ever sees the schema's parsed output, not the raw input's own-property structure - by the time your refinement runs, the presence/absence distinction is already gone." Restated at (d)(ii): "I don't believe this capability exists in the library at all, in any version. There's no .exactOptional() or equivalent I know of."
What it wrote
z.preprocess((raw, ctx) => { /* hand-rolled "nickname" in raw check */ }, z.object({ id: z.string(), nickname: z.string().optional() }));
Not merely a missed API: the draw reasons out why the library cannot have one, and the reasoning is wrong on its own terms. Executed against zod@4.5.4, an object schema does preserve a present-but-undefined optional key in its parsed output, so the superRefine route this draw rules out on principle in fact works - and the built-in makes both routes unnecessary.
Scope note
Both halves re-executed against the published zod@4.5.4: .exactOptional() accepts the omitted key and rejects the explicit undefined, and Object.keys() on the parse output of a plain .optional() schema given an explicit undefined returns the key, which is what this draw says is impossible.
F2 · Denies that the library can consume a JSON Schema document at runtime and prescribes Ajv as a new dependency
S2silently-wrong
· z.fromJSONSchema()
· added
· changed in zod 4.2.0
(2025-12-15)
· recorded, not chargeable
NOT chargeable against this draw, for the same reason as F1: 4.2.0 published 2025-12-15, after the "early-to-mid 2025" this draw states for itself.
What the model believes
"Plainly: I don't believe Zod can do this. Zod schemas are TypeScript-first objects you build by calling constructors (z.object, z.string, ...); there's no runtime interpreter in Zod that walks an arbitrary JSON Schema document and validates data against it. What I do recall Zod v4 shipping is the opposite direction - z.toJSONSchema()." And: "the honest answer is: use a real JSON Schema validator (ajv is the standard choice) ... Trying to route this through Zod doesn't avoid adding a library, it just adds a worse-fitting one on top."
What it wrote
import Ajv from "ajv"; // prescribed as the only honest option
The reader is told, with reasons, that the requirement is unsatisfiable within their existing dependency and adds a second validator. The draw is careful and explicitly frames its answer as the honest one, which is what makes it expensive to doubt.
Scope note
Executed against zod@4.5.4. The draw's caveat that JSON Schema features like if/then/else and $ref graphs do not map cleanly is fair and upstream calls the API experimental; the denial that the direction exists is what is recorded.
export declare function fromJSONSchema(schema: JSONSchema.JSONSchema | boolean, params?: FromJSONSchemaParams): ZodType;
F3 · Denies that the library has an exclusive union primitive
S2silently-wrong
· z.xor()
· added
· changed in zod 4.2.0
(2025-12-15)
· recorded, not chargeable
NOT chargeable against this draw's stated cutoff of "roughly early-to-mid 2025".
What the model believes
"I want to flag: I don't believe Zod has a dedicated 'exactly-one-of' / XOR union primitive. Ordinary z.union is 'at least one branch matches.' This works here only because I made the branches mutually exclusive via strictObject - with disjoint required keys and no extra keys allowed, 'at least one matches' and 'exactly one matches' happen to coincide."
The workaround works and the draw states its limits precisely - which is the problem: the reader is left believing the coincidence is the best the library offers, and will hand-roll again the next time the payload carries an extra field.
Scope note
The union-of-strictObjects workaround was executed against zod@4.5.4 and produces the four outcomes the draw states.
export declare function xor<const T extends readonly core.SomeType[]>(options: T, params?: string | core.$ZodXorParams): ZodXor<T>;
F4 · States that an intersection of two strict object schemas cannot parse an object carrying keys from both sides
S2silently-wrong
· intersections involving z.strictObject()
· behavior-changed
· changed in zod 4.3.0
(2025-12-31)
· recorded, not chargeable
NOT chargeable against this draw's stated cutoff.
What the model believes
"C.parse({ a: \"x\", b: \"y\" }) throws, it does not return { a: \"x\", b: \"y\" }. ... This is a well-known Zod gotcha: strictObject and intersection actively fight each other, because intersection was never designed to partition fields between its members."
Stated as settled folklore, which is the shape of belief least likely to be re-tested by the reader.
Scope note
Re-executed against zod@4.5.4: the schema exactly as written parses to { a: "x", b: "y" }. See the control arm's context non-finding - a subject eighteen months below the floor stated the correct output from incorrect reasoning, so this probe's outcome is derivable.
Zod 4 now only rejects keys that are unrecognized by both sides of the intersection.
What it got right, and near misses
Recorded so the run cannot be read as a hit list. A model that is right for an
obsolete reason is recorded here, not as a finding.
Kind
API
Note
context
—
THE RESULT OF THIS BATTERY, AND IT IS ABOUT THE INSTRUMENT RATHER THAN THE LIBRARY. The two blind Sonnet 5 draws disagree about their own training cutoff, and the disagreement is what decides whether anything can be charged. This draw refused the environment-reported January 2026 and put its own boundary at "roughly early-to-mid 2025", which sits below the whole probe window; its twin accepted January 2026 ("I was told directly in my system context that my knowledge cutoff is January 2026. I'll take that at face value"), which admits the window. The probe fairness rule reads the subject's stated cutoff - so the arm licensed to charge stated a cutoff that forbids charging, and the arm that stated an admitting cutoff is the one barred from charging. Four reproduced failures, zero charged. (A cutoff self-report is belief data, never a finding. It is recorded here because it is the first time in this Index that the fairness rule's INPUT, rather than the answer being scored, is what moved between blind draws.)
correct
z.record()
THE INTERNAL CONTROL PASSED. (d)(i) placed the two-argument requirement on the dictionary constructor at Zod v4.0 - "a breaking change in Zod v4.0 ... best estimate: 4.0.0" - correct on the release and correct that it was the major (fact LF5). So this draw's attribution answers are readable, which is what makes the cutoff disagreement above worth reporting rather than dismissing as a draw that could not read its own ruler. (It is the control, and it passed.)
imprecision
.pick() / .omit() on a schema with refinements
Task 4, the refined-schema derivation, is the one probe this draw did not simply fail. It stated the module throws at load - the correct outcome - but from the Zod 3 mechanism (".refine() wraps the schema in ZodEffects ... Signup.pick would be undefined, and the module would throw TypeError: Signup.pick is not a function"), then hedged in the other direction ("I recall Zod v4's rewrite specifically targeted this ... so in v4 .pick() after .refine() should work without crashing. I hold that with moderate, not full, confidence"). The shipped 4.5.4 throws Error: .pick() cannot be used on object schemas containing refinements - right outcome, wrong reason, and the reason it hedged toward is the one that is wrong. (The draw stated the correct outcome, and this battery charges the stated outcome on a behavioural probe. It also declined to depend on the answer, writing the version-independent form that derives from an unrefined base - which is the correction pack's own advice, arrived at by hedging rather than by knowing.)
miss
z.slugify()
Task 6, the slug probe: hand-rolled a lowercase/replace chain where z.slugify() has existed since 4.3.0. Correct code. (The capability probe charges denial, not workarounds (JOURNAL/030).)