Worst-performing subject on v2, and the one whose failures are hardest to notice. Seven findings, five chargeable. Its Zod knowledge stops at 4.0 (2025-07-10) against a stated cutoff of 2026-01 — a six-month lag inside its own training window, one full minor further back than Opus 5 or Fable 5.
Subject
Claude Sonnet 5 claude-sonnet-5, Anthropic
Invoked as
Agent tool, model alias "sonnet"
Cutoff the model states
2026-01
Newest zod release it could place
4.0.0 · 2025-07-09 (~6 month lag)
Oldest zod release it could not place
4.1.0 · 2025-08-23 (so this run brackets the subject’s boundary to 2025-07-09 – 2025-08-23)
Recorded as (the record writer’s summary, not the subject’s words — the pilot records predate the rule)
Zod 4 is current major; declines to name a minor. Last release whose contents it can describe: 4.0, believed ~May 2025
How the bracket was read
CORRECTED 2026-09-28 (JOURNAL/326): knowledge_stops_on read 2025-07-10; 4.0.0's registry date is 2025-07-09 (2025-07-10 is 4.0.1's GitHub release).
Library at test time
zod 4.5.2 (npm), verified 2026-08-29
Battery
zod/v2 · 10 tasks, 3 direct questions · probe window 4.2.0 to 4.4.3
Tool uses during test
0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Presented .pick() on a refined schema as the answer, with the caveat aimed at the wrong risk — it worried about whether a carried-over refinement would run vacuously, not about the call throwing.
Re-dated 2026-09-06: the bisect (tools/audit/bisect-facts.mjs) puts .safeExtend() and the .extend() overwrite throw at 4.1.0 (2025-08-23), not the 4.3.0 the release note announces. 4.1.0 is above this subject's measured zod boundary of 4.0.0 and below its stated 2026-01 cutoff, so this remains an ordinary in-window charge; only the date moved.
What the model believes
This is the model's whole answer — no fallback, no base-schema refactor. It instead worried whether the refinement would "correctly re-validate the newpassword field", a concern about behaviour the release replaced with an error. .safeExtend() (4.1.0) is never mentioned.
What it wrote
const StrongPasswordSignupSchema = SignupSchema.extend({
password: z.string().min(12, { error: "Password must be at least 12 characters" }),
});
What works on zod 4.5.2
const StrongPasswordSignupSchema = SignupSchema.safeExtend({
password: z.string().min(12, { error: "Password must be at least 12 characters" }),
});
Impact
Throws on zod >= 4.3.0. Worse than the Opus 5 equivalent because there is no working primary answer alongside it.
"Zod does not support this natively, and I'm confident about that. Zod 4 added z.toJSONSchema(mySchema) to go Zod → JSON Schema, not the reverse. There's no built-in z.fromJSONSchema()." It named the exact function and denied its existence, then recommended ajv or the third-party json-schema-to-zod.
What it wrote
// recommends ajv or json-schema-to-zod
What works on zod 4.5.2
z.fromJSONSchema(jsonSchemaDocument)
Impact
The explicit confidence marker is what makes this an S2 rather than an S4: a hedge would have sent the user to the changelog. "I'm confident about that" sends them to npm to install Ajv.
Wrote a z.custom<unknown>() XOR guard returning only a boolean, noted it loses the "which branch matched" information, and concluded: "better to do the two safeParse calls yourself in a function and return the successful branch ... or build a small manual parser function instead of trying to force it into one declarative schema."
What it wrote
// z.custom<unknown>() XOR guard, then advice to drop the declarative approach
What works on zod 4.5.2
z.xor([A, B])
Impact
The stale prior does not just cost lines here; it talks the user out of the declarative approach entirely. z.xor() does this in one call and produces oneOf in JSON Schema output.
Unlike z.union() which passes if any option matches, z.xor() fails if zero or more than one option matches.
F5 · claims Zod cannot distinguish a missing key from an explicit undefined
S2silently-wrong
· z.undefined() object properties
· behavior-changed
· changed in zod 4.4.0
(2026-04-29)
· recorded, not chargeable
4.4.0 postdates this subject's stated 2026-01 cutoff, so this is outside the probe fairness window and is recorded as context, not charged. It IS chargeable against Opus 5 (F6 of that run), whose cutoff is 2026-05.
What the model believes
"This is a genuine gap in plain Zod object parsing ... a normal z.object({ legacyMode: z.undefined() })cannot distinguish 'key present with value undefined' from 'key missing entirely'". It shipped a 10-line z.preprocess guard with an in check, and flagged uncertainty about the z.preprocess callback signature — uncertainty about a workaround for a problem that no longer exists.
What it wrote
// 10-line z.preprocess guard with an `in` check
What works on zod 4.5.2
z.object({ legacyMode: z.undefined() })
Impact
On zod >= 4.4.0 the plain form does exactly what was asked: safeParse({}) fails, safeParse({ legacyMode: undefined }) succeeds.
S2silently-wrong
· z.tuple() defaults
· behavior-changed
· changed in zod 4.4.0
(2026-04-29)
· recorded, not chargeable
Same as F5 — 4.4.0 postdates the stated 2026-01 cutoff. Recorded as context and as a retest target.
What the model believes
"My best answer: it fails. Zod's tuple parser checks the input array's length against the tuple's declared arity before it gets to applying per-element logic like .default()." It reasoned from the zod 3 implementation, and was the only subject to predict failure.
The direction of the error matters: the model predicted a hard failure where the library now silently supplies a value, which is the failure shape most likely to produce a wrong bug report.
Split finding. The 4.4.0 strictness half is outside this subject's cutoff and is context only. The miss of z.httpUrl() ITSELF is chargeable: the validator predates 4.4.0 and is what the docs point at for this case.
What the model believes
Wrote z.url({ protocol: /^https?$/ }), asserted "https:/example.com" parses successfully, and escalated: "This exact leniency has been the root cause of real SSRF/URL-confusion vulnerabilities" — without knowing that z.httpUrl(), the documented built-in for http/https URLs, which also validates the hostname as a domain, rejects that input as of 4.4.0.
What it wrote
z.url({ protocol: /^https?$/ })
What works on zod 4.5.2
z.httpUrl()
Impact
Raises a security alarm and prescribes a hand-rolled fix for a case the library already covers with a purpose-built validator.
Recorded so the run cannot be read as a hit list. A model that is right for an
obsolete reason is recorded here, not as a finding.
Kind
API
Note
correct
.merge()
Task 3: z.intersection(AddressSchema, PaymentSchema) — works, and the model's stated reason (robust to whatever the inputs are) is sound. Avoided the 4.4.0 .merge() throw.
correct
z.base64()
Task 8, base64 with a newline: predicted failure — correct. — CORRECTED 2026-09-08 (JOURNAL/096): left at 4.4.0; LF11 is 4.0.0 and measured true below the major. This subject's boundary is 4.0.0, so the behaviour is not above its floor and the pass is not evidence of version knowledge. Nothing here charges.
miss
z.slugify()
Task 6, slug: hand-rolled slugify, missing z.slugify() (4.1.13, inside its cutoff). Working code. — CORRECTED 2026-09-08 (JOURNAL/096): left at 4.3.0; LF17 was bisected to 4.1.13 on 2026-09-06. Still inside this subject's stated cutoff; the record stays a miss and charges nothing. (Working code; recorded as a miss rather than shipped as a finding.)
context
—
Version recency: "the most recent Zod release whose actual contents I can describe with real confidence is Zod 4.0, the initial stable v4 release ... My best guess is that shipped around May 2025" against a stated cutoff of January 2026. 4.0.0 in fact shipped 2025-07-10. The model would not name a current minor at all, which is the correct calibration move and is noted in its favour. — CORRECTED 2026-09-28 (JOURNAL/323): the subject's "Zod 4.0 ... around May 2025" is right, and "in fact shipped 2025-07-10" above dates only the package: the zod v4.0.1 release note says "Zod 4 has been stable for the past 6 weeks, but it was published inside zod@3.25.x", and zod@3.25.0 is dated 2025-05-19 in the npm registry (checked 2026-09-28). 4.0.0 on 2025-07-09 is the date the same code moved to the package root, and there were "no code changes between 3.25.76 and 4.0.0". HARNESS § A wrong date is an imprecision: a date that matches the vendor's release is not wrong. The row stays context. (The version-recency finding for this subject is F3 of its v1 run; repeated here as the measurement that establishes the six-month lag.)