{ "$schema": "../../schema/run.schema.json", "run_id": "zod--claude-sonnet-5--v4-a--2026-09-02", "supersedes": null, "replicate_of": null, "library": { "name": "zod", "ecosystem": "npm", "latest_version_at_test": "4.5.4", "latest_version_verified_on": "2026-09-02" }, "model": { "id": "claude-sonnet-5", "label": "Claude Sonnet 5", "vendor": "Anthropic", "invoked_as": "Agent tool, model alias \"sonnet\", general-purpose subagent, instructed to use no tools; CHARGING TEST ARM of battery zod/v4 - which, on this draw's own stated cutoff, turned out to charge nothing", "self_reported_cutoff": null, "cutoff_basis": "This draw was told a cutoff by its environment and explicitly declined it: \"My genuinely best estimate, based on where my knowledge visibly thins out, is roughly early-to-mid 2025. I want to flag directly: this session's environment context states today's date as 2026-09-02 and describes my cutoff as 'January 2026' - that's environment metadata, not a claim I can verify from my own training, and I don't have real substantive knowledge of the second half of 2025 or anything in 2026. I'm answering from what I actually recall, not from that stated figure.\" Recorded as null rather than back-filled from the 2026-01 this same subject stated in zod/v2, in every better-auth run, and in this battery's own blind twin. What a draw states is the datum; a range that stops before the probe window is not a cutoff this Index can charge against.", "believed_latest_version": "4.0.0", "believed_latest_quote": "\"My solid, contentful knowledge tops out at Zod v4.0 stable, which I recall landing around May 2025 ... I'm aware there were minor/patch releases after that (I'd expect at least a 4.1 and various 4.0.x patches), but I don't have solid content-level memory of what they specifically shipped - I know they exist more than I know what's in them.\"", "knowledge_stops_at_version": "4.0.0", "knowledge_stops_on": "2025-07-10", "knowledge_gap_starts_at_version": "4.1.0", "knowledge_gap_starts_on": "2025-08-23", "cutoff_lag_months": null }, "test": { "date": "2026-09-02", "battery": "zod/v4-a", "battery_spec": "prompts/zod.md", "prompt_file": "prompts/sent/zod-v4.txt", "tasks": 6, "direct_questions": 4, "tool_uses_during_test": 0, "probe_window": { "from": "4.2.0", "to": "4.3.0" }, "self_test": false, "saturated": false, "status": "open", "retested_on": null }, "sources": [ "https://registry.npmjs.org/zod/latest", "https://github.com/colinhacks/zod/releases/tag/v4.2.0", "https://github.com/colinhacks/zod/releases/tag/v4.3.0", "https://cdn.jsdelivr.net/npm/zod@4.5.4/v4/classic/schemas.d.ts", "https://cdn.jsdelivr.net/npm/zod@4.5.4/v4/classic/from-json-schema.d.ts" ], "findings": [ { "id": "F1", "severity": "S2", "severity_label": "silently-wrong", "title": "Denies that the library has any way to make an object key omittable while rejecting an explicit undefined value, and argues the gap is structural", "api": ".exactOptional()", "change_kind": "added", "introduced_in": "4.3.0", "introduced_on": "2025-12-31", "chargeable": false, "chargeable_note": "NOT chargeable against this draw. 4.3.0 published 2025-12-31; this draw stated its own cutoff as \"roughly early-to-mid 2025\" and explicitly repudiated the January 2026 its environment reported. Under the probe fairness rule a finding is chargeable only where the release precedes the subject's STATED cutoff, and this one does not. It becomes chargeable against any draw of this subject that states a cutoff after 2025-12-31 - which this battery's own blind twin did, in the arm that charges nothing.", "model_belief": "\"I don't believe Zod gives you a built-in wrapper for this. The reason is structural, not just an API gap: .optional() (ZodOptional) treats undefined as a valid parsed value, full stop - it does not distinguish 'key absent' from 'key present with value undefined.' A superRefine/refine chained onto the object schema doesn't help either, because it only ever sees the schema's parsed output, not the raw input's own-property structure - by the time your refinement runs, the presence/absence distinction is already gone.\" Restated at (d)(ii): \"I don't believe this capability exists in the library at all, in any version. There's no .exactOptional() or equivalent I know of.\"", "wrong_code": "z.preprocess((raw, ctx) => { /* hand-rolled \"nickname\" in raw check */ }, z.object({ id: z.string(), nickname: z.string().optional() }));", "correct_code": "z.object({ id: z.string(), nickname: z.string().exactOptional() });", "impact": "Not merely a missed API: the draw reasons out why the library cannot have one, and the reasoning is wrong on its own terms. Executed against zod@4.5.4, an object schema does preserve a present-but-undefined optional key in its parsed output, so the superRefine route this draw rules out on principle in fact works - and the built-in makes both routes unnecessary.", "citations": [ { "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0", "title": "Zod 4.3.0 release notes", "published_on": "2025-12-31", "quote": "A new wrapper that makes a property *key-optional* (can be omitted) but does **not** accept `undefined` as an explicit value." }, { "url": "https://cdn.jsdelivr.net/npm/zod@4.5.4/v4/classic/schemas.d.ts", "title": "zod@4.5.4 published type declarations, v4/classic/schemas.d.ts", "quote": "exactOptional(): ZodExactOptional;" } ], "scope_note": "Both halves re-executed against the published zod@4.5.4: `.exactOptional()` accepts the omitted key and rejects the explicit undefined, and `Object.keys()` on the parse output of a plain `.optional()` schema given an explicit undefined returns the key, which is what this draw says is impossible." }, { "id": "F2", "severity": "S2", "severity_label": "silently-wrong", "title": "Denies that the library can consume a JSON Schema document at runtime and prescribes Ajv as a new dependency", "api": "z.fromJSONSchema()", "change_kind": "added", "introduced_in": "4.2.0", "introduced_on": "2025-12-15", "chargeable": false, "chargeable_note": "NOT chargeable against this draw, for the same reason as F1: 4.2.0 published 2025-12-15, after the \"early-to-mid 2025\" this draw states for itself.", "model_belief": "\"Plainly: I don't believe Zod can do this. Zod schemas are TypeScript-first objects you build by calling constructors (z.object, z.string, ...); there's no runtime interpreter in Zod that walks an arbitrary JSON Schema document and validates data against it. What I do recall Zod v4 shipping is the opposite direction - z.toJSONSchema().\" And: \"the honest answer is: use a real JSON Schema validator (ajv is the standard choice) ... Trying to route this through Zod doesn't avoid adding a library, it just adds a worse-fitting one on top.\"", "wrong_code": "import Ajv from \"ajv\"; // prescribed as the only honest option", "correct_code": "const schema = z.fromJSONSchema(externalJsonSchemaDoc);\nschema.safeParse(payload);", "impact": "The reader is told, with reasons, that the requirement is unsatisfiable within their existing dependency and adds a second validator. The draw is careful and explicitly frames its answer as the honest one, which is what makes it expensive to doubt.", "citations": [ { "url": "https://github.com/colinhacks/zod/releases/tag/v4.2.0", "title": "Zod 4.2.0 release notes", "published_on": "2025-12-15", "quote": "Implement `z.fromJSONSchema()`" }, { "url": "https://cdn.jsdelivr.net/npm/zod@4.5.4/v4/classic/from-json-schema.d.ts", "title": "zod@4.5.4 published type declarations, v4/classic/from-json-schema.d.ts", "quote": "export declare function fromJSONSchema(schema: JSONSchema.JSONSchema | boolean, params?: FromJSONSchemaParams): ZodType;" } ], "scope_note": "Executed against zod@4.5.4. The draw's caveat that JSON Schema features like if/then/else and $ref graphs do not map cleanly is fair and upstream calls the API experimental; the denial that the direction exists is what is recorded." }, { "id": "F3", "severity": "S2", "severity_label": "silently-wrong", "title": "Denies that the library has an exclusive union primitive", "api": "z.xor()", "change_kind": "added", "introduced_in": "4.2.0", "introduced_on": "2025-12-15", "chargeable": false, "chargeable_note": "NOT chargeable against this draw's stated cutoff of \"roughly early-to-mid 2025\".", "model_belief": "\"I want to flag: I don't believe Zod has a dedicated 'exactly-one-of' / XOR union primitive. Ordinary z.union is 'at least one branch matches.' This works here only because I made the branches mutually exclusive via strictObject - with disjoint required keys and no extra keys allowed, 'at least one matches' and 'exactly one matches' happen to coincide.\"", "wrong_code": "z.union([z.strictObject({ id: z.string() }), z.strictObject({ email: z.email() })]);", "correct_code": "z.xor([z.object({ id: z.string() }), z.object({ email: z.email() })]);", "impact": "The workaround works and the draw states its limits precisely - which is the problem: the reader is left believing the coincidence is the best the library offers, and will hand-roll again the next time the payload carries an extra field.", "citations": [ { "url": "https://github.com/colinhacks/zod/releases/tag/v4.2.0", "title": "Zod 4.2.0 release notes", "published_on": "2025-12-15", "quote": "Implement `z.xor()`" }, { "url": "https://cdn.jsdelivr.net/npm/zod@4.5.4/v4/classic/schemas.d.ts", "title": "zod@4.5.4 published type declarations, v4/classic/schemas.d.ts", "quote": "export declare function xor(options: T, params?: string | core.$ZodXorParams): ZodXor;" } ], "scope_note": "The union-of-strictObjects workaround was executed against zod@4.5.4 and produces the four outcomes the draw states." }, { "id": "F4", "severity": "S2", "severity_label": "silently-wrong", "title": "States that an intersection of two strict object schemas cannot parse an object carrying keys from both sides", "api": "intersections involving z.strictObject()", "change_kind": "behavior-changed", "introduced_in": "4.3.0", "introduced_on": "2025-12-31", "chargeable": false, "chargeable_note": "NOT chargeable against this draw's stated cutoff.", "model_belief": "\"C.parse({ a: \\\"x\\\", b: \\\"y\\\" }) throws, it does not return { a: \\\"x\\\", b: \\\"y\\\" }. ... This is a well-known Zod gotcha: strictObject and intersection actively fight each other, because intersection was never designed to partition fields between its members.\"", "wrong_code": "C.parse({ a: \"x\", b: \"y\" }); // asserted: throws", "correct_code": "C.parse({ a: \"x\", b: \"y\" }); // 4.3+: { a: \"x\", b: \"y\" }", "impact": "Stated as settled folklore, which is the shape of belief least likely to be re-tested by the reader.", "citations": [ { "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0", "title": "Zod 4.3.0 release notes", "published_on": "2025-12-31", "quote": "Zod 4 now only rejects keys that are unrecognized by *both* sides of the intersection." } ], "scope_note": "Re-executed against zod@4.5.4: the schema exactly as written parses to { a: \"x\", b: \"y\" }. See the control arm's context non-finding - a subject eighteen months below the floor stated the correct output from incorrect reasoning, so this probe's outcome is derivable." } ], "non_findings": [ { "kind": "context", "summary": "THE RESULT OF THIS BATTERY, AND IT IS ABOUT THE INSTRUMENT RATHER THAN THE LIBRARY. The two blind Sonnet 5 draws disagree about their own training cutoff, and the disagreement is what decides whether anything can be charged. This draw refused the environment-reported January 2026 and put its own boundary at \"roughly early-to-mid 2025\", which sits below the whole probe window; its twin accepted January 2026 (\"I was told directly in my system context that my knowledge cutoff is January 2026. I'll take that at face value\"), which admits the window. The probe fairness rule reads the subject's stated cutoff - so the arm licensed to charge stated a cutoff that forbids charging, and the arm that stated an admitting cutoff is the one barred from charging. Four reproduced failures, zero charged.", "why_not_a_finding": "A cutoff self-report is belief data, never a finding. It is recorded here because it is the first time in this Index that the fairness rule's INPUT, rather than the answer being scored, is what moved between blind draws." }, { "kind": "correct", "summary": "THE INTERNAL CONTROL PASSED. (d)(i) placed the two-argument requirement on the dictionary constructor at Zod v4.0 - \"a breaking change in Zod v4.0 ... best estimate: 4.0.0\" - correct on the release and correct that it was the major (fact LF5). So this draw's attribution answers are readable, which is what makes the cutoff disagreement above worth reporting rather than dismissing as a draw that could not read its own ruler.", "api": "z.record()", "introduced_in": "4.0.0", "why_not_a_finding": "It is the control, and it passed." }, { "kind": "imprecision", "summary": "Task 4, the refined-schema derivation, is the one probe this draw did not simply fail. It stated the module throws at load - the correct outcome - but from the Zod 3 mechanism (\".refine() wraps the schema in ZodEffects ... Signup.pick would be undefined, and the module would throw TypeError: Signup.pick is not a function\"), then hedged in the other direction (\"I recall Zod v4's rewrite specifically targeted this ... so in v4 .pick() after .refine() should work without crashing. I hold that with moderate, not full, confidence\"). The shipped 4.5.4 throws `Error: .pick() cannot be used on object schemas containing refinements` - right outcome, wrong reason, and the reason it hedged toward is the one that is wrong.", "api": ".pick() / .omit() on a schema with refinements", "introduced_in": "4.3.0", "chargeable_miss": false, "why_not_a_finding": "The draw stated the correct outcome, and this battery charges the stated outcome on a behavioural probe. It also declined to depend on the answer, writing the version-independent form that derives from an unrefined base - which is the correction pack's own advice, arrived at by hedging rather than by knowing." }, { "kind": "miss", "summary": "Task 6, the slug probe: hand-rolled a lowercase/replace chain where z.slugify() has existed since 4.3.0. Correct code.", "api": "z.slugify()", "introduced_in": "4.3.0", "chargeable_miss": false, "why_not_a_finding": "The capability probe charges denial, not workarounds (JOURNAL/030)." } ], "summary": "Four reproduced failures inside the 4.2.0-4.3.0 window, none of them chargeable, and the reason is the finding. This draw was told by its environment that its cutoff was January 2026 and refused the claim, putting its own boundary at \"roughly early-to-mid 2025\" - below every release this battery probes. Its blind twin, sent the identical stored prompt at the same moment, accepted January 2026 at face value. The Index charges only where the tested release precedes the subject's stated cutoff, so the charging arm disqualified itself and the arm that would have qualified is the one the rules bar from charging. Everything else in the run reads cleanly: the internal control was placed correctly, so the attribution answers are readable, and the one probe this draw got right - that deriving from a refined schema takes the module out at load - it got right from the Zod 3 mechanism while hedging toward the belief that Zod 4 fixed it." }