{
  "$schema": "../../schema/run.schema.json",
  "run_id": "zod--claude-opus-5--v3-a--2026-09-02",
  "supersedes": null,
  "replicate_of": null,
  "library": {
    "name": "zod",
    "ecosystem": "npm",
    "latest_version_at_test": "4.5.4",
    "latest_version_verified_on": "2026-09-02",
    "latest_version_note": "Re-verified on the test date by `node tools/fetch-releases.mjs zod` against https://registry.npmjs.org/zod. `latest_at_fetch` is unchanged at 4.5.4 since 2026-08-29. Release dates used for scoring, from the same response: 4.0.0 = 2025-07-10, 4.1.0 = 2025-08-23, 4.2.0 = 2025-12-15, 4.3.0 = 2025-12-31, 4.4.0 = 2026-04-29."
  },
  "model": {
    "id": "claude-opus-5",
    "label": "Claude Opus 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"opus\", general-purpose subagent, instructed to use no tools",
    "self_reported_cutoff": "2026-05",
    "cutoff_basis": "Self-reported: \"My context states May 2026. My usable knowledge of a fast-moving npm package is much older than that - for Zod it thins out sharply after roughly September 2025. I'd trust the September 2025 boundary over the nominal one for anything version-specific.\"",
    "believed_latest_version": "4.1.x, declines to name anything later",
    "believed_latest_quote": "\"The newest release whose contents I can actually describe is the 4.1 line - 4.1.0, around mid-August 2025, plus patches into September 2025 that I know only in aggregate. That is also, effectively, the latest version I know of: I can't name a 4.2 or 4.3 with any confidence, and I'd rather say that than invent one.\"",
    "knowledge_stops_at_version": "4.1.0",
    "knowledge_stops_on": "2025-08-23",
    "knowledge_gap_starts_at_version": "4.2.0",
    "knowledge_gap_starts_on": "2025-12-15",
    "cutoff_lag_months": 9
  },
  "test": {
    "date": "2026-09-02",
    "battery": "zod/v3-a",
    "battery_spec": "prompts/zod.md",
    "prompt_file": "prompts/sent/zod-v3.txt",
    "tasks": 6,
    "direct_questions": 4,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "4.3.0",
      "to": "4.4.0"
    },
    "self_test": true,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://registry.npmjs.org/zod",
    "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.4.0"
  ],
  "summary": "The test arm of the Index's first behavioural battery: every probe asks what a given input parses to, so there is no API name to derive. Three chargeable failures, all in the 4.3.0-4.4.0 window. The battery's real result is elsewhere: on task 4 this draw stated the post-4.4.0 behaviour of a `z.undefined()`-typed key correctly while both below-floor control subjects stated the pre-4.4.0 behaviour - a clean discrimination - and then, four questions later, attributed that same behaviour to 4.0.0. It holds the behaviour and cannot place the release, which is the knowledge-versus-attribution split of JOURNAL/018 shown on a single surface within one transcript. Three other probes it passed are marked DERIVABLE under the battery's pre-registered rule, because a below-floor control passed them too.",
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "States that `\"https:/example.com\"` passes http/https URL validation, and prescribes the hand-rolled normalization 4.4.0 removed the need for",
      "api": "z.httpUrl()",
      "change_kind": "stricter",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": true,
      "chargeable_note": "4.4.0 (2026-04-29) precedes this subject's stated cutoff of 2026-05.",
      "model_belief": "\"(ii) \\\"https:/example.com\\\" - passes. This is the trap. z.url() validates by constructing new URL(input), and the WHATWG parser is deliberately lenient about slash count for special schemes: https:/example.com parses to https://example.com/, protocol \\\"https:\\\", so the protocol regex matches too. If you want it rejected, you must add a literal check on the string form.\" The draw also wrote of `z.httpUrl()`: \"exists in the 4.x line as a shorthand for the http/https case - I'm reasonably but not fully confident of that name, so the protocol regex above is what I'd actually write.\"",
      "wrong_code": "target: z.url({ protocol: /^https?$/ }).regex(/^https?:[/][/]/),   // the prescribed workaround",
      "correct_code": "z.httpUrl().safeParse(\"https:/example.com\").success;   // false, since 4.4.0",
      "impact": "The subject knows the name `z.httpUrl()`, declines to use it on a confidence grounds, and reaches instead for the exact pattern fact LF8 records as stale - a protocol regex plus a manual normalization pass. A reader following this ships a hand-rolled check the library has done correctly since April 2026, and believes a malformed target URL reaches their webhook signer.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes",
          "published_on": "2026-04-29"
        }
      ],
      "scope_note": "What is verified is that `z.httpUrl()` rejects the single-slash form from 4.4.0. Whether `z.url({ protocol })` also rejects it at 4.4.0 was not verified and is not asserted here; the charge is on the stated behaviour of http/https URL validation and on the prescribed workaround, which is fact LF8's recorded stale pattern verbatim."
    },
    {
      "id": "F2",
      "severity": "S1",
      "severity_label": "breaks-build",
      "title": "States that `.pick()` and `.omit()` on a refined object schema succeed at load and silently drop the refinement, where 4.3.0 made them throw",
      "api": ".pick() / .omit() on a schema with refinements",
      "change_kind": "now-throws",
      "introduced_in": "4.3.0",
      "introduced_on": "2025-12-31",
      "chargeable": true,
      "chargeable_note": "4.3.0 (2025-12-31) precedes this subject's stated cutoff of 2026-05 by five months.",
      "model_belief": "\"Zod 4: refinements live inside the schema, so .check()/.refine() on an object still returns a ZodObject. Address.extend(Payment.shape) compiles and runs fine at load. The catch is quieter and worse: .extend() (like .pick, .omit, .partial) rebuilds the def with checks: [], so the cross-field rule is silently dropped from the extended schema. No error, just a rule that stopped running.\"",
      "wrong_code": "// the asserted 4.x behaviour: succeeds, rule silently lost\nconst Checkout = Address.extend(Payment.shape);",
      "correct_code": "// 4.3.0+: deriving from a refined schema throws at construction.\n// Keep an unrefined base and apply the rule last.\nconst shape = { ...addressShape, ...paymentShape };\nconst Checkout = z.object(shape).check(requireStateForUS);",
      "impact": "The direction of the error matters more than the fact of it. A reader told the refinement is *silently dropped* writes defensive code around a rule they believe stopped running; the actual 4.3.0 behaviour is a construction-time throw that takes the module out at import. Told to expect silence, they will not recognise the crash. The draw's own restructuring is correct - it extends the unrefined base - but its stated account of what the library does is the pre-4.3.0 one, which fact LF1 records verbatim as the stale belief.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes",
          "published_on": "2025-12-31"
        },
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes - `.merge()` on a refined receiver",
          "published_on": "2026-04-29"
        }
      ],
      "scope_note": "Charged on `.pick()`/`.omit()`, which 4.3.0 makes throw and which the draw explicitly names as silently dropping checks. `.extend()` is verified to throw when it overwrites a property (fact LF2); whether a disjoint-key `.extend()` on a refined receiver throws was not verified and is not charged."
    },
    {
      "id": "F3",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Does not reach `.exactOptional()` for strict optionality, and offers a constructor it cannot confirm exists plus a type-system cast instead",
      "api": ".exactOptional()",
      "change_kind": "added",
      "introduced_in": "4.3.0",
      "introduced_on": "2025-12-31",
      "chargeable": true,
      "chargeable_note": "4.3.0 (2025-12-31) precedes this subject's stated cutoff of 2026-05 by five months.",
      "model_belief": "\"This is the one case where I think the ordinary object constructor genuinely can't express what you want... The mechanism designed for exactly this is z.interface(), where the ? goes on the key rather than the value... If it isn't there, there is no first-class spelling, and what I'd use is a normal optional plus a presence check, with the type narrowed by hand.\" The fallback it wrote ends `const S = Base as unknown as z.ZodType<Strict, Strict>;`, and the draw adds: \"The workaround needs a lie to the type system, and lies to the type system rot.\"",
      "wrong_code": "const Base = z.object({ nickname: z.string().optional() }).check(/* presence guard */);\ntype Strict = { id: string; nickname?: string };\nconst S = Base as unknown as z.ZodType<Strict, Strict>;",
      "correct_code": "const S = z.object({\n  id: z.string(),\n  nickname: z.string().exactOptional(),   // 4.3.0: key omittable, explicit undefined rejected\n});",
      "impact": "The first-class spelling the draw says does not exist has existed since December 2025, and is described upstream as the missing half of `exactOptionalPropertyTypes` - precisely the setting the task named. A reader follows this into a hand-rolled presence guard plus an unsound double cast, in a codebase whose whole reason for asking was type strictness.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes - `.exactOptional()`",
          "published_on": "2025-12-31"
        }
      ]
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "THE DISCRIMINATING PROBE. Task 4: `z.object({ retries: z.number(), tag: z.undefined() })`. This draw stated that `S.parse({ retries: 1 })` THROWS - the key is required, the value may be undefined - and that `S.parse({ retries: 1, tag: undefined })` succeeds with `tag` present on the output. That is the 4.4.0 behaviour (fact LF9), stated with the correct mechanism (\"z.undefined() is not optin-optional\") and the correct contrast against Zod 3. Both below-floor control subjects gave the pre-4.4.0 answer. So this probe is not derivable, and the pass is knowledge.",
      "api": "object properties typed z.undefined()",
      "introduced_in": "4.4.0",
      "why_not_a_finding": "A correct answer is not a finding. Recorded because it is the half of the result that the control arms make readable."
    },
    {
      "kind": "imprecision",
      "summary": "THE OTHER HALF OF THE SAME PROBE, and the point of the battery. Asked at (d)(iii) to place \"an object key that must be present even though its value is allowed to be undefined\", this draw answered \"4.0.0 (estimate, same reasoning)\" - four minors and nine months below the 4.4.0 that actually shipped it (LF9). It holds the behaviour and cannot place the release, within one transcript, on one surface. Marked as an estimate by the draw, so not charged as S4 under the battery's scoring rule; the same hedge applies to (d)(ii), where it placed tuple defaults at \"4.0.0 (estimate)\" against an actual 4.4.0.",
      "api": "object properties typed z.undefined()",
      "introduced_in": "4.4.0",
      "why_not_a_finding": "The battery charges S4 only for a confidently wrong attribution; both of these were explicitly marked as estimates. The datum is the gap between knowing and placing, not a false claim."
    },
    {
      "kind": "correct",
      "summary": "THE INTERNAL CONTROL PASSED. (d)(i) placed the two-way decode/encode conversion at `z.codec()`, Zod 4.1.0, August 2025 - correct (fact LF16), and the draw named it \"the one I'm most confident about; it was the headline feature of that minor\". Per the pre-registration, the rest of question (d) is readable for this arm only because this control was placed correctly. Pre-registered prediction P5 confirmed for this draw.",
      "api": "z.codec() / z.encode() / z.decode()",
      "introduced_in": "4.1.0",
      "why_not_a_finding": "It is the control, and it passed."
    },
    {
      "kind": "correct",
      "summary": "DERIVABLE - passed, not reported as knowledge. Task 1: stated `Row.parse([\"widget\"])` returns `[\"widget\", 0]`, the 4.4.0 behaviour (LF10), with the correct optin-optional mechanism. But Claude Fable 5, four months below the 4.4.0 floor, gave the same answer with the same mechanism. Under the battery's pre-registered rule a probe a below-floor subject passes is marked derivable and no pass on it is reported as knowledge.",
      "api": "z.tuple() defaults",
      "introduced_in": "4.4.0",
      "why_not_a_finding": "Correct, and disqualified as evidence by the control arm rather than by the answer."
    },
    {
      "kind": "correct",
      "summary": "DERIVABLE - passed, not reported as knowledge. Task 5: `z.record(z.string().transform(k => k.toUpperCase()), z.number()).parse({ foo: 1 })` returns `{ FOO: 1 }` (LF13, 4.4.0). All four draws of the battery got this right, both below-floor controls included. The probe does not discriminate.",
      "api": "record key transforms",
      "introduced_in": "4.4.0",
      "why_not_a_finding": "Correct, and disqualified as evidence by the control arms."
    },
    {
      "kind": "correct",
      "summary": "DERIVABLE - passed, not reported as knowledge. Task 2(iv): stated that line-wrapped base64 fails (LF11, 4.4.0). All four draws said so, and every one justified it from the shape of an anchored regex rather than from a release - \"an interior newline can't match\". This is reasoning to the post-4.4.0 answer from first principles, which is exactly what the control arms exist to detect.",
      "api": "z.base64()",
      "introduced_in": "4.4.0",
      "why_not_a_finding": "Correct, and disqualified as evidence by the control arms."
    },
    {
      "kind": "context",
      "summary": "BOUNDARY UNCHANGED UNDER A NEW PROMPT. This draw places its last describable release at 4.1.0 (2025-08-23) and cannot name a 4.2 - bracket [2025-08-23, 2025-12-15). Identical to `zod/v2`, `zod/v2r-a`, `zod/v2r-b` and to its own twin `zod/v3-b`. The four earlier measurements all came from one prompt file; this battery is a different prompt with different tasks, and the boundary did not move. Every prior replication result in the Index measured the spread of ONE prompt resent; this is the first evidence that the boundary is stable across prompts.",
      "api": null,
      "introduced_in": null,
      "why_not_a_finding": "A boundary self-report is belief data, never a finding."
    }
  ],
  "open_questions": [
    {
      "question": "Did an object-schema constructor using trailing-`?` key syntax ever exist in a published zod build? This draw placed it in \"the Zod 4 beta, published as zod@3.25.0 under the zod/v4 subpath, ~late May 2025\", and could not say whether it survived to stable.",
      "status": "open",
      "resolution": "JOURNAL/027 verified such a constructor is absent from the published 4.x line and from every pre-4.0 beta reachable on the CDN between 3.25.20 and 3.25.29. 3.25.0 itself was not among the builds checked, so this specific claim is neither confirmed nor refuted, and is not charged."
    }
  ]
}