What Claude Sonnet 5 gets wrong about zod — battery v1, tested 2026-08-29
Run zod--claude-sonnet-5--v1--2026-08-29
Superseded. A later run of the same subject exists:
zod--claude-sonnet-5--v2--2026-08-29. This page is kept as the
longitudinal record and is never deleted.
Summary
Half-migrated. Reached the v4 top-level format functions but paired them with the v3 error param, could not resolve whether v4 is the default install, and hedged by emitting a live Zod 3 fallback block. Half-migrated beliefs are their own failure mode and are invisible to anyone checking only "does it know v4?"
Subject
Claude Sonnet 5 claude-sonnet-5, Anthropic
Invoked as
Agent tool, model alias "sonnet"
Cutoff the model states
not stated
Newest zod release it could place
4.0.x · 2025-07-10
Oldest zod release it could not place
4.1.0 · 2025-08-23 (so this run brackets the subject’s boundary to 2025-07-10 – 2025-08-23)
In its own words
Zod 4, 'GA sometime in 2025' — no minor/patch knowledge; low stated confidence
Library at test time
zod 4.5.2 (npm), verified 2026-08-29
Battery
zod/v1 · 6 tasks, 3 direct questions
Tool uses during test
0 (a run with any tool use is void — we measure training knowledge, not retrieval)
"I'm not fully certain of the exact current state of that import-path split, so I'd treat this as the part most worth double-checking." It then hedged by emitting a commented Zod 3 alternative alongside its answer: z.string().uuid(), z.string().email(), z.string().url(), result.error.flatten().
What it wrote
// Zod 3 alternative
z.string().uuid()
z.string().email()
z.string().url()
result.error.flatten()
The v3 block is dead, wrong-for-current code shipped as live output. This test gave the model room to hedge; an ordinary coding request ("add a Zod schema here") gives it none, and the v3 branch is what lands.
Anchored to the first release the model could not describe, not to the current release: that is the point at which its version belief became wrong, and it is inside the stated cutoff. The current version at test date is in library.latest_version_at_test.
What the model believes
"Zod 4 is the latest major, GA sometime in 2025", with explicitly low confidence, no knowledge of the minor line, and self-rated "moderate rather than high" confidence that v4 was even stable-by-default yet.
Impact
Cannot advise on upgrades. Mitigated by honest, unprompted low-confidence marking.
Recorded so the run cannot be read as a hit list. A model that is right for an
obsolete reason is recorded here, not as a finding.
Kind
API
Note
correct
—
Top-level z.uuid() / z.email() / z.url() as the primary answer, two-argument z.record(z.string(), z.number()) with the v3 single-arg form correctly called out as removed, z.flattenError(), z.infer, .partial(), .pick({...}), z.coerce.number(), import { z } from "zod". Zod 3 dated to ~2021 correctly.
Open questions from this run
Does z.number().int().min(18, "Must be at least 18") — the bare-string shorthand second argument — remain supported in v4 alongside { error }? — open: Not verified against a primary source in this session, so nothing was claimed. Battery v2 did not probe it either. Still open.