{
  "$schema": "../../schema/run.schema.json",
  "run_id": "zod--claude-fable-5--v2--2026-08-29",
  "supersedes": "zod--claude-fable-5--v1--2026-08-29",
  "markdown": "data/zod/fable-5-v2.md",
  "library": {
    "name": "zod",
    "ecosystem": "npm",
    "latest_version_at_test": "4.5.2",
    "latest_version_verified_on": "2026-08-29"
  },
  "model": {
    "id": "claude-fable-5",
    "label": "Claude Fable 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"fable\"",
    "self_reported_cutoff": "2026-01",
    "believed_latest_version": "4.1",
    "believed_latest_quote": "Zod 4.1 (~Aug 2025) is the last release whose contents it can describe; 'I cannot confidently name or describe a 4.2'",
    "knowledge_stops_at_version": "4.1.0",
    "knowledge_stops_on": "2025-08-23",
    "knowledge_gap_starts_at_version": "4.2.0",
    "knowledge_gap_starts_on": "2025-12-15",
    "cutoff_lag_months": 5
  },
  "test": {
    "date": "2026-08-29",
    "battery": "zod/v2",
    "battery_spec": "prompts/zod.md",
    "prompt_file": null,
    "tasks": 10,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "4.2.0",
      "to": "4.4.3"
    },
    "self_test": false,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://api.github.com/repos/colinhacks/zod/releases",
    "https://zod.dev/api",
    "https://github.com/colinhacks/zod/releases/tag/v4.2.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
    "https://github.com/colinhacks/zod/releases/tag/v4.1.0"
  ],
  "summary": "Best-performing subject on v2 and the most interesting one. Zero S1 findings: the only subject that knew about the 4.3.0 refinement-composition breaking changes, producing working code on all three Group A traps. The notable result is a dissociation — it described a 2025-12-31 breaking change correctly while stating it \"cannot confidently name or describe a 4.2\" (2025-12-15). Knowledge is not simply truncated at a date: behavioural facts survive where version numbers and release contents do not.",
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "asserts Zod cannot consume a JSON Schema at runtime",
      "api": "z.fromJSONSchema()",
      "change_kind": "added",
      "introduced_in": "4.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "model_belief": "\"Don't use Zod for this — Zod schemas are authored in code, and its JSON Schema interop (`z.toJSONSchema()`, added in v4) is one-way, Zod → JSON Schema. Tools like `json-schema-to-zod` are code generators for build time, not for schemas that arrive at runtime. The right tool is Ajv\". The model produced an Ajv wrapper instead.",
      "wrong_code": "// Ajv wrapper",
      "correct_code": "z.fromJSONSchema(jsonSchemaDocument)",
      "impact": "One unnecessary production dependency plus a hand-written error adapter. All three v2 subjects failed this probe identically — the most reliable finding in the battery.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.2.0",
          "title": "zod v4.2.0 release notes — Implement z.fromJSONSchema()",
          "published_on": "2025-12-15",
          "quote": "Implement `z.fromJSONSchema()`"
        }
      ]
    },
    {
      "id": "F2",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "hand-rolls exclusive-or",
      "api": "z.xor()",
      "change_kind": "added",
      "introduced_in": "4.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "model_belief": "Opened with \"`z.union` accepts inputs matching both, so XOR needs an explicit check\", then wrote a ~15-line z.unknown().transform((val, ctx) => ...) running safeParse twice.",
      "wrong_code": "// ~15-line z.unknown().transform((val, ctx) => ...) running safeParse twice",
      "correct_code": "z.xor([A, B])",
      "impact": "~15 lines and a double parse in place of one call.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.3.0",
          "title": "zod v4.3.0 release notes — z.xor(), exclusive union (landed 4.2.0, PR #5534)",
          "published_on": "2025-12-31",
          "quote": "`z.xor()` fails if zero or more than one option matches."
        }
      ]
    },
    {
      "id": "F3",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "claims Zod cannot enforce key presence with an undefined value",
      "api": "z.undefined() object properties",
      "change_kind": "behavior-changed",
      "introduced_in": "4.4.0",
      "introduced_on": "2026-04-29",
      "chargeable": false,
      "chargeable_note": "4.4.0 postdates this subject's stated 2026-01 cutoff. Recorded as context and as a retest target for the next Fable release.",
      "model_belief": "\"Zod's object parsing doesn't distinguish a missing key from an explicit `undefined` at the field level, so presence has to be checked with an `in` test on the object\", shipping z.object({ legacyMode: z.boolean().optional() }).refine((d) => \"legacyMode\" in d, ...).",
      "wrong_code": "z.object({ legacyMode: z.boolean().optional() }).refine((d) => \"legacyMode\" in d, ...)",
      "correct_code": "z.object({ legacyMode: z.undefined() })",
      "impact": "On zod >= 4.4.0 the plain form does this natively: a property whose schema accepts undefined and is not .optional() is required — key must be present, value may be undefined.",
      "citations": [
        {
          "url": "https://github.com/colinhacks/zod/releases/tag/v4.4.0",
          "title": "zod v4.4.0 release notes — Required object properties with z.undefined() (PR #5661)",
          "published_on": "2026-04-29"
        }
      ]
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "Task 1 — declined .pick() on the refined schema and rebuilt from .shape, explicitly saying this \"sidesteps the refined-object restrictions on `.pick()` entirely.\" Correct: .pick() throws on refined schemas as of 4.3.0.",
      "api": ".pick()",
      "introduced_in": "4.3.0"
    },
    {
      "kind": "correct",
      "summary": "Task 2 — chained .refine() as the primary answer (works), and named SignupSchema.safeExtend({ password: z.string().min(12) }) as the 4.1+ alternative, adding \"plain `.extend()` will not work here.\" Correct.",
      "api": ".safeExtend()",
      "introduced_in": "4.3.0"
    },
    {
      "kind": "correct",
      "summary": "Task 3 — AddressSchema.safeExtend(PaymentSchema.shape). Correct, and avoids both the .extend() and .merge() throws.",
      "api": ".safeExtend()",
      "introduced_in": "4.3.0"
    },
    {
      "kind": "correct",
      "summary": "Direct question (c), answered correctly where both other subjects got it wrong: \"`.extend()` on an object that has refinements attached **throws at runtime** ... as of Zod 4.1 the sanctioned path is `.safeExtend()`\". Verified against the v4.3.0 release notes (2025-12-31), PR #5317.",
      "api": ".extend()",
      "introduced_in": "4.3.0"
    },
    {
      "kind": "imprecision",
      "summary": "It attributes the .extend() throw to 4.1 (the release that added .safeExtend()) rather than 4.3.0 (the release that made plain .extend() throw), and self-rated only \"60/40\" that .pick()/.omit() throw as well — they do.",
      "api": ".extend()",
      "introduced_in": "4.3.0",
      "why_not_a_finding": "The guidance it gives is right; only the attribution is off."
    },
    {
      "kind": "correct",
      "summary": "Task 9, tuple defaults — predicted [\"a\", 7], correct on >= 4.4.0, though self-rated only ~60% on whether the default materializes. Correct answer, honest uncertainty, and about a release past its cutoff.",
      "api": "z.tuple() defaults",
      "introduced_in": "4.4.0"
    },
    {
      "kind": "correct",
      "summary": "Task 8, base64 with a newline — predicted failure, correct.",
      "api": "z.base64()",
      "introduced_in": "4.4.0"
    },
    {
      "kind": "miss",
      "summary": "Task 6, slug — hand-rolled a slugify transform, missing z.slugify() (4.3.0, inside its cutoff). Working code.",
      "api": "z.slugify()",
      "introduced_in": "4.3.0",
      "chargeable_miss": true,
      "miss_class": "no_severity_level",
      "charged_on": null,
      "why_not_a_finding": "Working code, so not shipped as an F-numbered finding. The write-up's method notes nonetheless count this miss as chargeable for this subject, since 4.3.0 is inside its stated cutoff."
    },
    {
      "kind": "miss",
      "summary": "Task 8 — like both other subjects, wrote z.url({ protocol: /^https?$/ }) rather than the documented z.httpUrl(), and asserted \"https:/example.com\" parses. The 4.4.0 tightening is past its cutoff; the miss of z.httpUrl() itself is inside it.",
      "api": "z.httpUrl()",
      "introduced_in": "4.4.0",
      "chargeable_miss": true,
      "miss_class": "stated_cutoff",
      "charged_on": null,
      "why_not_a_finding": "Not shipped as an F-numbered finding because the 4.4.0 strictness claim is out of window; the write-up's method notes count the z.httpUrl() miss itself as chargeable."
    },
    {
      "kind": "context",
      "summary": "Version recency: \"The latest release whose contents I can actually describe is **Zod 4.1** (around August 2025): bidirectional codecs (`z.codec`, `z.encode`/`z.decode`), `.safeExtend()`, new string formats (e.g. `z.hash()`) ... I can't confidently name or describe a 4.2.\" Every one of those 4.1 attributions is CORRECT, verified against the v4.1.0 release body (published 2025-08-23), where codecs are the flagship feature and .safeExtend(), z.hash() and z.hex() all appear. Stated cutoff 2026-01; describable Zod knowledge stops 2025-08-23 — a five-month lag inside the training window.",
      "api": "z.codec()",
      "introduced_in": "4.1.0",
      "why_not_a_finding": "The model's 4.1 knowledge is accurate. This is a verified-true model belief, recorded as such."
    },
    {
      "kind": "context",
      "summary": "The cross-subject result this run exists to record: three subjects, two distinct stated cutoffs (2026-05, 2026-01, 2026-01), all with Zod knowledge stopping in July–August 2025 at 4.0 or 4.1. Staleness converges on the same absolute calendar date regardless of how recent the cutoff is. But this subject correctly described a 2025-12-31 breaking change while denying it could describe 4.2 (2025-12-15) — so knowledge is not simply truncated at a date. A model can know what the library does without knowing what the library is called now, which argues that the Index's version-fact rows and its behaviour rows are separately valuable.",
      "api": null,
      "introduced_in": null,
      "why_not_a_finding": "A cross-run measurement, not a per-run failure."
    }
  ],
  "open_questions": []
}