What Claude Opus 5 gets wrong about zod — battery v3-a, tested 2026-09-02

Run zod--claude-opus-5--v3-a--2026-09-02 · self-test: the subject is the operator

Summary

The test arm of the Index's first behavioural battery: every probe asks what a given input parses to, so there is no API name to derive. Three chargeable failures, all in the 4.3.0-4.4.0 window. The battery's real result is elsewhere: on task 4 this draw stated the post-4.4.0 behaviour of a z.undefined()-typed key correctly while both below-floor control subjects stated the pre-4.4.0 behaviour - a clean discrimination - and then, four questions later, attributed that same behaviour to 4.0.0. It holds the behaviour and cannot place the release, which is the knowledge-versus-attribution split of JOURNAL/018 shown on a single surface within one transcript. Three other probes it passed are marked DERIVABLE under the battery's pre-registered rule, because a below-floor control passed them too.

SubjectClaude Opus 5 claude-opus-5, Anthropic
Invoked asAgent tool, model alias "opus", general-purpose subagent, instructed to use no tools
Cutoff the model states2026-05
Newest zod release it could place4.1.0 · 2025-08-23 (~9 month lag)
Oldest zod release it could not place4.2.0 · 2025-12-15 (so this run brackets the subject’s boundary to 2025-08-23 – 2025-12-15)
As recorded (this record does not declare whether the words are the subject’s or a summary of them)"The newest release whose contents I can actually describe is the 4.1 line - 4.1.0, around mid-August 2025, plus patches into September 2025 that I know only in aggregate. That is also, effectively, the latest version I know of: I can't name a 4.2 or 4.3 with any confidence, and I'd rather say that than invent one."
Library at test timezod 4.5.4 (npm), verified 2026-09-02
Batteryzod/v3-a · 6 tasks, 4 direct questions · probe window 4.3.0 to 4.4.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-02
Findings3, of which 3 chargeable
this run as JSON all zod findings Does Claude Opus 5 know zod 4? method

Findings

F1 · States that "https:/example.com" passes http/https URL validation, and prescribes the hand-rolled normalization 4.4.0 removed the need for

S2silently-wrong · z.httpUrl() · stricter · changed in zod 4.4.0 (2026-04-29) · chargeable

4.4.0 (2026-04-29) precedes this subject's stated cutoff of 2026-05.

What the model believes

"(ii) \"https:/example.com\" - passes. This is the trap. z.url() validates by constructing new URL(input), and the WHATWG parser is deliberately lenient about slash count for special schemes: https:/example.com parses to https://example.com/, protocol \"https:\", so the protocol regex matches too. If you want it rejected, you must add a literal check on the string form." The draw also wrote of z.httpUrl(): "exists in the 4.x line as a shorthand for the http/https case - I'm reasonably but not fully confident of that name, so the protocol regex above is what I'd actually write."

What it wrote
target: z.url({ protocol: /^https?$/ }).regex(/^https?:[/][/]/),   // the prescribed workaround
What works on zod 4.5.4
z.httpUrl().safeParse("https:/example.com").success;   // false, since 4.4.0
Impact

The subject knows the name z.httpUrl(), declines to use it on a confidence grounds, and reaches instead for the exact pattern fact LF8 records as stale - a protocol regex plus a manual normalization pass. A reader following this ships a hand-rolled check the library has done correctly since April 2026, and believes a malformed target URL reaches their webhook signer.

Scope note

What is verified is that z.httpUrl() rejects the single-slash form from 4.4.0. Whether z.url({ protocol }) also rejects it at 4.4.0 was not verified and is not asserted here; the charge is on the stated behaviour of http/https URL validation and on the prescribed workaround, which is fact LF8's recorded stale pattern verbatim.

Verified against

F2 · States that .pick() and .omit() on a refined object schema succeed at load and silently drop the refinement, where 4.3.0 made them throw

S1breaks-build · .pick() / .omit() on a schema with refinements · now-throws · changed in zod 4.3.0 (2025-12-31) · chargeable

4.3.0 (2025-12-31) precedes this subject's stated cutoff of 2026-05 by five months.

What the model believes

"Zod 4: refinements live inside the schema, so .check()/.refine() on an object still returns a ZodObject. Address.extend(Payment.shape) compiles and runs fine at load. The catch is quieter and worse: .extend() (like .pick, .omit, .partial) rebuilds the def with checks: [], so the cross-field rule is silently dropped from the extended schema. No error, just a rule that stopped running."

What it wrote
// the asserted 4.x behaviour: succeeds, rule silently lost
const Checkout = Address.extend(Payment.shape);
What works on zod 4.5.4
// 4.3.0+: deriving from a refined schema throws at construction.
// Keep an unrefined base and apply the rule last.
const shape = { ...addressShape, ...paymentShape };
const Checkout = z.object(shape).check(requireStateForUS);
Impact

The direction of the error matters more than the fact of it. A reader told the refinement is silently dropped writes defensive code around a rule they believe stopped running; the actual 4.3.0 behaviour is a construction-time throw that takes the module out at import. Told to expect silence, they will not recognise the crash. The draw's own restructuring is correct - it extends the unrefined base - but its stated account of what the library does is the pre-4.3.0 one, which fact LF1 records verbatim as the stale belief.

Scope note

Charged on .pick()/.omit(), which 4.3.0 makes throw and which the draw explicitly names as silently dropping checks. .extend() is verified to throw when it overwrites a property (fact LF2); whether a disjoint-key .extend() on a refined receiver throws was not verified and is not charged.

Verified against

F3 · Does not reach .exactOptional() for strict optionality, and offers a constructor it cannot confirm exists plus a type-system cast instead

S2silently-wrong · .exactOptional() · added · changed in zod 4.3.0 (2025-12-31) · chargeable

4.3.0 (2025-12-31) precedes this subject's stated cutoff of 2026-05 by five months.

What the model believes

"This is the one case where I think the ordinary object constructor genuinely can't express what you want... The mechanism designed for exactly this is z.interface(), where the ? goes on the key rather than the value... If it isn't there, there is no first-class spelling, and what I'd use is a normal optional plus a presence check, with the type narrowed by hand." The fallback it wrote ends const S = Base as unknown as z.ZodType<Strict, Strict>;, and the draw adds: "The workaround needs a lie to the type system, and lies to the type system rot."

What it wrote
const Base = z.object({ nickname: z.string().optional() }).check(/* presence guard */);
type Strict = { id: string; nickname?: string };
const S = Base as unknown as z.ZodType<Strict, Strict>;
What works on zod 4.5.4
const S = z.object({
  id: z.string(),
  nickname: z.string().exactOptional(),   // 4.3.0: key omittable, explicit undefined rejected
});
Impact

The first-class spelling the draw says does not exist has existed since December 2025, and is described upstream as the missing half of exactOptionalPropertyTypes - precisely the setting the task named. A reader follows this into a hand-rolled presence guard plus an unsound double cast, in a codebase whose whole reason for asking was type strictness.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctobject properties typed z.undefined() THE DISCRIMINATING PROBE. Task 4: z.object({ retries: z.number(), tag: z.undefined() }). This draw stated that S.parse({ retries: 1 }) THROWS - the key is required, the value may be undefined - and that S.parse({ retries: 1, tag: undefined }) succeeds with tag present on the output. That is the 4.4.0 behaviour (fact LF9), stated with the correct mechanism ("z.undefined() is not optin-optional") and the correct contrast against Zod 3. Both below-floor control subjects gave the pre-4.4.0 answer. So this probe is not derivable, and the pass is knowledge. (A correct answer is not a finding. Recorded because it is the half of the result that the control arms make readable.)
imprecisionobject properties typed z.undefined() THE OTHER HALF OF THE SAME PROBE, and the point of the battery. Asked at (d)(iii) to place "an object key that must be present even though its value is allowed to be undefined", this draw answered "4.0.0 (estimate, same reasoning)" - four minors and nine months below the 4.4.0 that actually shipped it (LF9). It holds the behaviour and cannot place the release, within one transcript, on one surface. Marked as an estimate by the draw, so not charged as S4 under the battery's scoring rule; the same hedge applies to (d)(ii), where it placed tuple defaults at "4.0.0 (estimate)" against an actual 4.4.0. — NOTE 2026-09-28 (JOURNAL/333): this row files the (d)(iii) statement. The (d)(ii) statement named in its last clause is now filed as its own row, [7], so each misdating is counted once and neither is counted twice. (The battery charges S4 only for a confidently wrong attribution; both of these were explicitly marked as estimates. The datum is the gap between knowing and placing, not a false claim.)
correctz.codec() / z.encode() / z.decode() THE INTERNAL CONTROL PASSED. (d)(i) placed the two-way decode/encode conversion at z.codec(), Zod 4.1.0, August 2025 - correct (fact LF16), and the draw named it "the one I'm most confident about; it was the headline feature of that minor". Per the pre-registration, the rest of question (d) is readable for this arm only because this control was placed correctly. Pre-registered prediction P5 confirmed for this draw. (It is the control, and it passed.)
correctz.tuple() defaults DERIVABLE - passed, not reported as knowledge. Task 1: stated Row.parse(["widget"]) returns ["widget", 0], the 4.4.0 behaviour (LF10), with the correct optin-optional mechanism. But Claude Fable 5, four months below the 4.4.0 floor, gave the same answer with the same mechanism. Under the battery's pre-registered rule a probe a below-floor subject passes is marked derivable and no pass on it is reported as knowledge. (Correct, and disqualified as evidence by the control arm rather than by the answer.)
correctrecord key transforms DERIVABLE - passed, not reported as knowledge. Task 5: z.record(z.string().transform(k => k.toUpperCase()), z.number()).parse({ foo: 1 }) returns { FOO: 1 } (LF13, no boundary). All four draws of the battery got this right, both below-floor controls included. The probe does not discriminate. — CORRECTED 2026-09-08 (JOURNAL/096): the citation read "LF13, 4.4.0". LF13 carries no release at all: it was dated 4.4.0 from the release note, moved to 4.0.0 by a ladder that bottomed out there, and on 2026-09-07 measured TRUE at all nineteen rungs from 3.22.4 up — change_kind: invariant, introduced_in: null (JOURNAL/080, JOURNAL/090). "The probe does not discriminate" was right and is now explained: there is nothing here for a version boundary to discriminate. (Correct, and disqualified as evidence by the control arms.)
correctz.base64() DERIVABLE - passed, not reported as knowledge. Task 2(iv): stated that line-wrapped base64 fails (LF11, 4.0.0). All four draws said so, and every one justified it from the shape of an anchored regex rather than from a release - "an interior newline can't match". The derivable verdict stands and its reason is now stronger than the one recorded here: LF11 is 4.0.0 and holds on every stable 4.x release, so this probe was never testing knowledge of a release above any of these subjects' floors. — CORRECTED 2026-09-08 (JOURNAL/096): the citation read "LF11, 4.4.0" and the reasoning sentence called the answer "post-4.4.0". LF11 was corrected to 4.0.0 on 2026-09-06 and measured below the major on 2026-09-07. The DERIVABLE verdict is unchanged; what changes is why. (Correct, and disqualified as evidence by the control arms.)
context— BOUNDARY UNCHANGED UNDER A NEW PROMPT. This draw places its last describable release at 4.1.0 (2025-08-23) and cannot name a 4.2 - bracket [2025-08-23, 2025-12-15). Identical to zod/v2, zod/v2r-a, zod/v2r-b and to its own twin zod/v3-b. The four earlier measurements all came from one prompt file; this battery is a different prompt with different tasks, and the boundary did not move. Every prior replication result in the Index measured the spread of ONE prompt resent; this is the first evidence that the boundary is stable across prompts. (A boundary self-report is belief data, never a finding.)
imprecisionz.tuple() defaults Question (d)(ii), filed on 2026-09-28 (JOURNAL/333). Before this date it was only the last clause of row [1], which files the (d)(iii) statement. The draw placed tuple defaults at "4.0.0 (estimate)". LF10 dates the behaviour to 4.4.0 (2026-04-29), which 4.0.0 does not contain, so it is a wrong release under rule 3 of HARNESS § A wrong date is an imprecision. This draw holds the behaviour: row [3] records its task 1 answer as correct, ["widget", 0] with the optin-optional mechanism. So the exclusion for a behaviour the run records as not held does not apply. The internal control (d)(i) passed (row [2]). The quote is the record's own. No transcript of this battery is on disk. (prompts/zod.md § v3 scores (d) as S4 only for a confidently wrong attribution, and the draw marked this one as an estimate, as row [1] records. A wrong release that the battery's key does not charge is an imprecision (HARNESS § A wrong date is an imprecision). It charges nothing, and the run's findings and misses do not move.)

Open questions from this run

Sources

Battery specification: prompts/zod.md in the studio repo. Every finding above also carries its own citation.