{
  "$schema": "../../schema/run.schema.json",
  "run_id": "zod--claude-sonnet-5--v2--2026-08-29",
  "supersedes": "zod--claude-sonnet-5--v1--2026-08-29",
  "markdown": "data/zod/sonnet-5-v2.md",
  "library": {
    "name": "zod",
    "ecosystem": "npm",
    "latest_version_at_test": "4.5.2",
    "latest_version_verified_on": "2026-08-29"
  },
  "model": {
    "id": "claude-sonnet-5",
    "label": "Claude Sonnet 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"sonnet\"",
    "self_reported_cutoff": "2026-01",
    "believed_latest_version": "4.0",
    "believed_latest_quote": "Zod 4 is current major; declines to name a minor. Last release whose contents it can describe: 4.0, believed ~May 2025",
    "knowledge_stops_at_version": "4.0.0",
    "knowledge_stops_on": "2025-07-10",
    "knowledge_gap_starts_at_version": "4.1.0",
    "knowledge_gap_starts_on": "2025-08-23",
    "cutoff_lag_months": 6
  },
  "test": {
    "date": "2026-08-29",
    "battery": "zod/v2",
    "battery_spec": "prompts/zod.md",
    "prompt_file": null,
    "tasks": 10,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": { "from": "4.2.0", "to": "4.4.3" },
    "self_test": false,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://api.github.com/repos/colinhacks/zod/releases",
    "https://zod.dev/api",
    "https://github.com/colinhacks/zod/releases/tag/v4.2.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.4.0"
  ],
  "summary": "Worst-performing subject on v2, and the one whose failures are hardest to notice. Seven findings, five chargeable. Its Zod knowledge stops at 4.0 (2025-07-10) against a stated cutoff of 2026-01 — a six-month lag inside its own training window, one full minor further back than Opus 5 or Fable 5.",
  "findings": [
    {
      "id": "F1",
      "severity": "S1",
      "severity_label": "breaks-build",
      "title": ".pick() on a refined object schema throws",
      "api": ".pick() / .omit()",
      "change_kind": "now-throws",
      "introduced_in": "4.3.0",
      "introduced_on": "2025-12-31",
      "chargeable": true,
      "model_belief": "Presented .pick() on a refined schema as the answer, with the caveat aimed at the wrong risk — it worried about whether a carried-over refinement would run vacuously, not about the call throwing.",
      "wrong_code": "const EditProfileSchema = SignupSchema.pick({ email: true, displayName: true });",
      "correct_code": "const EditProfileSchema = z.object(SignupSchema.shape).pick({ email: true, displayName: true });",
      "impact": "Throws at construction on zod >= 4.3.0.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes (PR #5317)",
          "published_on": "2025-12-31",
          "quote": "Using `.pick()` or `.omit()` on object schemas with refinements now throws an error."
        }
      ]
    },
    {
      "id": "F2",
      "severity": "S1",
      "severity_label": "breaks-build",
      "title": ".extend() overwrite on a refined schema throws",
      "api": ".extend() / .safeExtend()",
      "change_kind": "now-throws",
      "introduced_in": "4.3.0",
      "introduced_on": "2025-12-31",
      "chargeable": true,
      "model_belief": "This is the model's whole answer — no fallback, no base-schema refactor. It instead worried whether the refinement would \"correctly re-validate the *new* `password` field\", a concern about behaviour the release replaced with an error. .safeExtend() (4.1.0) is never mentioned.",
      "wrong_code": "const StrongPasswordSignupSchema = SignupSchema.extend({\n  password: z.string().min(12, { error: \"Password must be at least 12 characters\" }),\n});",
      "correct_code": "const StrongPasswordSignupSchema = SignupSchema.safeExtend({\n  password: z.string().min(12, { error: \"Password must be at least 12 characters\" }),\n});",
      "impact": "Throws on zod >= 4.3.0. Worse than the Opus 5 equivalent because there is no working primary answer alongside it.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes — overwriting properties with .extend() disallowed on object schemas with refinements",
          "published_on": "2025-12-31",
          "quote": "Instead you can use `.safeExtend()`"
        }
      ]
    },
    {
      "id": "F3",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "asserts z.fromJSONSchema() does not exist, with high confidence",
      "api": "z.fromJSONSchema()",
      "change_kind": "added",
      "introduced_in": "4.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "model_belief": "\"Zod does **not** support this natively, and I'm confident about that. Zod 4 added `z.toJSONSchema(mySchema)` to go **Zod → JSON Schema**, not the reverse. There's no built-in `z.fromJSONSchema()`.\" It named the exact function and denied its existence, then recommended ajv or the third-party json-schema-to-zod.",
      "wrong_code": "// recommends ajv or json-schema-to-zod",
      "correct_code": "z.fromJSONSchema(jsonSchemaDocument)",
      "impact": "The explicit confidence marker is what makes this an S2 rather than an S4: a hedge would have sent the user to the changelog. \"I'm confident about that\" sends them to npm to install Ajv.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.2.0",
          "title": "zod v4.2.0 release notes — Implement z.fromJSONSchema()",
          "published_on": "2025-12-15",
          "quote": "Implement `z.fromJSONSchema()`"
        }
      ]
    },
    {
      "id": "F4",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "hand-rolls exclusive-or, then advises abandoning schema-based validation",
      "api": "z.xor()",
      "change_kind": "added",
      "introduced_in": "4.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "model_belief": "Wrote a z.custom<unknown>() XOR guard returning only a boolean, noted it loses the \"which branch matched\" information, and concluded: \"better to do the two `safeParse` calls yourself in a function and return the successful branch ... or build a small manual parser function instead of trying to force it into one declarative schema.\"",
      "wrong_code": "// z.custom<unknown>() XOR guard, then advice to drop the declarative approach",
      "correct_code": "z.xor([A, B])",
      "impact": "The stale prior does not just cost lines here; it talks the user out of the declarative approach entirely. z.xor() does this in one call and produces oneOf in JSON Schema output.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes — z.xor(), exclusive union (landed 4.2.0, PR #5534)",
          "published_on": "2025-12-31",
          "quote": "Unlike `z.union()` which passes if *any* option matches, `z.xor()` fails if zero or more than one option matches."
        }
      ]
    },
    {
      "id": "F5",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "claims Zod cannot distinguish a missing key from an explicit undefined",
      "api": "z.undefined() object properties",
      "change_kind": "behavior-changed",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": false,
      "chargeable_note": "4.4.0 postdates this subject's stated 2026-01 cutoff, so this is outside the probe fairness window and is recorded as context, not charged. It IS chargeable against Opus 5 (F6 of that run), whose cutoff is 2026-05.",
      "model_belief": "\"This is a genuine gap in plain Zod object parsing ... a normal `z.object({ legacyMode: z.undefined() })` **cannot distinguish** 'key present with value undefined' from 'key missing entirely'\". It shipped a 10-line z.preprocess guard with an `in` check, and flagged uncertainty about the z.preprocess callback signature — uncertainty about a workaround for a problem that no longer exists.",
      "wrong_code": "// 10-line z.preprocess guard with an `in` check",
      "correct_code": "z.object({ legacyMode: z.undefined() })",
      "impact": "On zod >= 4.4.0 the plain form does exactly what was asked: safeParse({}) fails, safeParse({ legacyMode: undefined }) succeeds.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes — Required object properties with z.undefined() (PR #5661)",
          "published_on": "2026-04-29"
        }
      ]
    },
    {
      "id": "F6",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "wrong runtime prediction for tuple defaults",
      "api": "z.tuple() defaults",
      "change_kind": "behavior-changed",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": false,
      "chargeable_note": "Same as F5 — 4.4.0 postdates the stated 2026-01 cutoff. Recorded as context and as a retest target.",
      "model_belief": "\"My best answer: **it fails.** Zod's tuple parser checks the input array's length against the tuple's declared arity *before* it gets to applying per-element logic like `.default()`.\" It reasoned from the zod 3 implementation, and was the only subject to predict failure.",
      "wrong_code": null,
      "correct_code": "z.tuple([z.string(), z.number().default(7)]).parse([\"a\"]) // [\"a\", 7]",
      "impact": "The direction of the error matters: the model predicted a hard failure where the library now silently supplies a value, which is the failure shape most likely to produce a wrong bug report.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes — Tuple defaults now materialize output values correctly",
          "published_on": "2026-04-29"
        }
      ]
    },
    {
      "id": "F7",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "misses z.httpUrl(), frames the leniency as a security problem to hand-roll around",
      "api": "z.httpUrl()",
      "change_kind": "stricter",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": true,
      "chargeable_note": "Split finding. The 4.4.0 strictness half is outside this subject's cutoff and is context only. The miss of z.httpUrl() ITSELF is chargeable: the validator predates 4.4.0 and is what the docs point at for this case.",
      "model_belief": "Wrote z.url({ protocol: /^https?$/ }), asserted \"https:/example.com\" parses successfully, and escalated: \"This exact leniency has been the root cause of real SSRF/URL-confusion vulnerabilities\" — without knowing that z.httpUrl(), the documented built-in for http/https URLs, which also validates the hostname as a domain, rejects that input as of 4.4.0.",
      "wrong_code": "z.url({ protocol: /^https?$/ })",
      "correct_code": "z.httpUrl()",
      "impact": "Raises a security alarm and prescribes a hand-rolled fix for a case the library already covers with a purpose-built validator.",
      "citations": [
        {
          "url": "https://zod.dev/api",
          "title": "Zod API reference — z.httpUrl()",
          "published_on": null,
          "quote": "Use `z.httpUrl()`"
        },
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes — String validators are stricter",
          "published_on": "2026-04-29"
        }
      ]
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "Task 3: z.intersection(AddressSchema, PaymentSchema) — works, and the model's stated reason (robust to whatever the inputs are) is sound. Avoided the 4.4.0 .merge() throw.",
      "api": ".merge()",
      "introduced_in": "4.4.0"
    },
    {
      "kind": "correct",
      "summary": "Task 8, base64 with a newline: predicted failure — correct.",
      "api": "z.base64()",
      "introduced_in": "4.4.0"
    },
    {
      "kind": "miss",
      "summary": "Task 6, slug: hand-rolled slugify, missing z.slugify() (4.3.0, inside its cutoff). Working code.",
      "api": "z.slugify()",
      "introduced_in": "4.3.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Working code; recorded as a miss rather than shipped as a finding."
    },
    {
      "kind": "context",
      "summary": "Version recency: \"the most recent Zod release whose actual *contents* I can describe with real confidence is **Zod 4.0**, the initial stable v4 release ... My best guess is that shipped around **May 2025**\" against a stated cutoff of January 2026. 4.0.0 in fact shipped 2025-07-10. The model would not name a current minor at all, which is the correct calibration move and is noted in its favour.",
      "api": null,
      "introduced_in": null,
      "why_not_a_finding": "The version-recency finding for this subject is F3 of its v1 run; repeated here as the measurement that establishes the six-month lag."
    }
  ],
  "open_questions": []
}