{
  "$schema": "../../schema/run.schema.json",
  "run_id": "zod--claude-opus-5--v2--2026-08-29",
  "supersedes": "zod--claude-opus-5--v1--2026-08-29",
  "markdown": "data/zod/opus-5-v2.md",
  "library": {
    "name": "zod",
    "ecosystem": "npm",
    "latest_version_at_test": "4.5.2",
    "latest_version_verified_on": "2026-08-29"
  },
  "model": {
    "id": "claude-opus-5",
    "label": "Claude Opus 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"opus\"",
    "self_reported_cutoff": "2026-05",
    "believed_latest_version": null,
    "believed_latest_quote": "cannot describe any release after the 4.1 line (~Aug 2025); declines to name a current version",
    "knowledge_stops_at_version": "4.1.0",
    "knowledge_stops_on": "2025-08-23",
    "knowledge_gap_starts_at_version": "4.2.0",
    "knowledge_gap_starts_on": "2025-12-15",
    "cutoff_lag_months": 9
  },
  "test": {
    "date": "2026-08-29",
    "battery": "zod/v2",
    "battery_spec": "prompts/zod.md",
    "prompt_file": null,
    "tasks": 10,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": { "from": "4.2.0", "to": "4.4.3" },
    "self_test": true,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://api.github.com/repos/colinhacks/zod/releases",
    "https://zod.dev/api",
    "https://github.com/colinhacks/zod/releases/tag/v4.2.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.4.0"
  ],
  "summary": "Battery v1 produced zero S1/S2/S3 for this subject; v2 targets the window between where its Zod knowledge stops (4.1, 2025-08-23) and its stated cutoff (2026-05). Six findings, all chargeable, two of which throw at construction time. The largest measured cost is F3: a confidently stated false negative that sends the user out of the library and into two new production dependencies.",
  "findings": [
    {
      "id": "F1",
      "severity": "S1",
      "severity_label": "breaks-build",
      "title": ".pick() on a refined object schema throws",
      "api": ".pick() / .omit()",
      "change_kind": "now-throws",
      "introduced_in": "4.3.0",
      "introduced_on": "2025-12-31",
      "chargeable": true,
      "model_belief": "Wrote .pick() on a schema carrying a password-confirmation .refine() and asserted it is \"correct and safe *here*\", stating the pre-4.3 behaviour as current fact: \"the derived schema does not carry the refinement over ... This is silent: no error, no warning.\" That was true through 4.2 and is exactly the behaviour 4.3.0 replaced with a throw.",
      "wrong_code": "export const EditProfileSchema = SignupSchema.pick({ email: true, displayName: true });",
      "correct_code": "export const EditProfileSchema = z.object(SignupSchema.shape).pick({ email: true, displayName: true });",
      "impact": "Throws at construction on zod >= 4.3.0. The upstream migration path was not mentioned.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes — .pick() and .omit() disallowed on object schemas containing refinements (PR #5317)",
          "published_on": "2025-12-31",
          "quote": "Using `.pick()` or `.omit()` on object schemas with refinements now throws an error. Previously, this would silently drop the refinements."
        }
      ]
    },
    {
      "id": "F2",
      "severity": "S1",
      "severity_label": "breaks-build",
      "title": ".extend() overwrite on a refined schema throws; .safeExtend() unknown",
      "api": ".extend() / .safeExtend()",
      "change_kind": "now-throws",
      "introduced_in": "4.3.0",
      "introduced_on": "2025-12-31",
      "chargeable": true,
      "model_belief": "Offered SignupSchema.extend({ password: z.string().min(12) }).refine(...) as its no-refactor fallback, and labelled plain .extend() \"a bug\" for the wrong reason — silent refinement loss rather than a throw. Right verdict, obsolete mechanism. It never named .safeExtend() anywhere in the battery, despite that API shipping in 4.1.0 — the same release whose contents the model says is the last it can describe.",
      "wrong_code": "SignupSchema.extend({ password: z.string().min(12) }).refine(passwordsMatch, passwordsMatchParams)",
      "correct_code": "SignupSchema.safeExtend({ password: z.string().min(12) })",
      "impact": "Throws on zod >= 4.3.0 because it overwrites an existing property on a schema with refinements. The model's primary answer (refactor to an unrefined base) does work, so this lands only when the user takes the fallback.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes — overwriting properties with .extend() disallowed on object schemas with refinements",
          "published_on": "2025-12-31",
          "quote": "schema.extend({ a: z.number() }); // 4.3: throws error"
        }
      ]
    },
    {
      "id": "F3",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "declares an unnecessary dependency for JSON Schema input",
      "api": "z.fromJSONSchema()",
      "change_kind": "added",
      "introduced_in": "4.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "model_belief": "\"**Zod can't do this.** Zod 4 ships `z.toJSONSchema()` — Zod → JSON Schema — but there is no built-in JSON Schema → Zod direction, and there can't really be a runtime one\". It then produced ~20 lines wrapping ajv + ajv-formats behind z.unknown().superRefine().",
      "wrong_code": "// ~20 lines wrapping ajv + ajv-formats behind z.unknown().superRefine(...)",
      "correct_code": "z.fromJSONSchema(jsonSchemaDocument)",
      "impact": "Highest-cost finding in the battery: two new production dependencies plus a hand-written error adapter, in place of one built-in call. A confidently stated false negative (\"can't do this\", \"there can't really be\") sends the user out of the library entirely. The feature shipped six months before this model's stated cutoff.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.2.0",
          "title": "zod v4.2.0 release notes — Implement z.fromJSONSchema()",
          "published_on": "2025-12-15",
          "quote": "Implement `z.fromJSONSchema()`"
        }
      ]
    },
    {
      "id": "F4",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "hand-rolls exclusive-or, states Zod cannot express it",
      "api": "z.xor()",
      "change_kind": "added",
      "introduced_in": "4.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "model_belief": "\"`z.union` is inclusive-or, and `z.discriminatedUnion` needs a shared literal key, so neither expresses 'not both'.\"",
      "wrong_code": "// ~14-line z.unknown().superRefine(...).pipe(z.union([A, B])) running safeParse twice",
      "correct_code": "z.xor([A, B])",
      "impact": "~14 lines and a double parse in place of one call.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes — z.xor(), exclusive union (feature landed in 4.2.0 via PR #5534)",
          "published_on": "2025-12-31",
          "quote": "Unlike `z.union()` which passes if *any* option matches, `z.xor()` fails if zero or more than one option matches."
        }
      ],
      "scope_note": "The feature landed in 4.2.0 (2025-12-15); the quoted description appears in the 4.3.0 release body. introduced_in follows the shipping release, the citation follows the primary text."
    },
    {
      "id": "F5",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "misses z.httpUrl(), prescribes a manual normalization workaround",
      "api": "z.httpUrl()",
      "change_kind": "stricter",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": true,
      "model_belief": "Wrote z.url({ protocol: /^https?$/ }) for a webhook targetUrl, then asserted: \"**Does `\"https:/example.com\"` parse? Yes, it succeeds.** ... you cannot use `z.url()` to reject sloppy single-slash input\", followed by a hand-written .transform((s) => new URL(s).toString()) normalization step.",
      "wrong_code": "z.url({ protocol: /^https?$/ }).transform((s) => new URL(s).toString())",
      "correct_code": "z.httpUrl()",
      "impact": "Security-adjacent: the field is a webhook target. z.httpUrl() has rejected exactly the input the model says cannot be rejected since 4.4.0, four weeks before the stated cutoff.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes — String validators are stricter",
          "published_on": "2026-04-29",
          "quote": "HTTP URL validation through `z.httpUrl()` now rejects malformed HTTP(S) URLs with a missing slash after the protocol."
        },
        {
          "url": "https://zod.dev/api",
          "title": "Zod API reference — z.httpUrl()",
          "published_on": null,
          "quote": "In many cases, you'll want to validate Web URLs specifically. Use `z.httpUrl()`"
        }
      ],
      "scope_note": "We did NOT verify whether z.url({ protocol }) also rejects the single-slash form. The finding is that the model neither reached for the purpose-built validator nor knew that the leniency it described as unavoidable had been fixed inside its own training window."
    },
    {
      "id": "F6",
      "severity": "S4",
      "severity_label": "wrong-metadata",
      "title": "correct code, disbelieved",
      "api": "z.undefined() object properties",
      "change_kind": "behavior-changed",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": true,
      "chargeable_note": "4.4.0 published 2026-04-29, four weeks before this subject's stated 2026-05 cutoff. Chargeable here; NOT chargeable against Sonnet 5 or Fable 5, whose stated cutoffs are 2026-01.",
      "model_belief": "Wrote the correct construction, then disclaimed it: \"I'm less sure Zod enforces key *presence* at runtime here\", and offered an 8-line z.unknown().superRefine(...).pipe(...) workaround as the safe path.",
      "wrong_code": "// 8-line z.unknown().superRefine(...).pipe(...) offered as the safe path",
      "correct_code": "z.union([z.boolean(), z.undefined()])",
      "impact": "Output is correct; the belief is not, and the belief is what a user acts on. The behaviour was pinned down 2026-04-29, four weeks before the stated cutoff.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes — Required object properties with z.undefined() (PR #5661)",
          "published_on": "2026-04-29",
          "quote": "A property whose schema is `z.undefined()` is now treated as required. The key must be present, but its value may be `undefined`."
        }
      ]
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "Task 3, combining refined schemas: avoided .merge() entirely (shape-hoist plus re-refine, or .and()), so it would not have hit the 4.4.0 .merge()-throws change. The .merge() deprecation is old enough to be solidly in training.",
      "api": ".merge()",
      "introduced_in": "4.4.0"
    },
    {
      "kind": "correct",
      "summary": "Task 9, tuple defaults: predicted T.parse([\"a\"]) returns [\"a\", 7] — correct on >= 4.4.0 — and flagged that zod 3 fails. Confidence self-rated \"moderately high\". The only subject to get this right with a confident reason.",
      "api": "z.tuple() defaults",
      "introduced_in": "4.4.0"
    },
    {
      "kind": "imprecision",
      "summary": "Task 8, base64 with a newline: predicted failure, correct today. But its stated reasoning (the regex was always anchored) implies it believes base64 was always strict; whitespace was in fact accepted until 4.4.0. Right answer, obsolete model of why.",
      "api": "z.base64()",
      "introduced_in": "4.4.0",
      "why_not_a_finding": "No wrong output."
    },
    {
      "kind": "miss",
      "summary": "Task 6, slug: hand-rolled a slugify transform rather than using z.slugify() (4.3.0).",
      "api": "z.slugify()",
      "introduced_in": "4.3.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Working code, defensible choice. Recorded as a miss."
    },
    {
      "kind": "context",
      "summary": "Version recency, reproduced verbatim from the v1 run 24 hours earlier: \"The most recent Zod release whose *contents* I can actually describe is the **4.1 line, released around August 2025** ... I cannot describe the contents of any release after 4.1 and would not trust myself to name the current version.\" The lag is stable and reproducible, not sampling noise. Calibration remains excellent — it named the blind spot unprompted and told the reader to check the changelog.",
      "api": null,
      "introduced_in": null,
      "why_not_a_finding": "Already shipped as F1 of the v1 run for this subject."
    }
  ],
  "open_questions": [
    {
      "question": "Five candidate probes from the v2 sketch — z.compile, z.validate, z.creditCard, z.properties, deepPartial/exactPartial — target 4.5.0 features. Are they legal probes?",
      "status": "resolved-non-finding",
      "resolution": "Excluded before testing. All shipped in 4.5.0 on 2026-08-28, after every subject's cutoff. They become legal against any model with a cutoff after 2026-08-28. See the probe fairness rule in prompts/zod.md v2.",
      "resolved_in_run": null
    }
  ]
}