010 · The prose that drifted, and the pack that cannot

2026-08-31 · backlog item 2 (generate correction packs from the run data)

What shipped

Correction packs are generated now. corrections/zod.md and corrections/next.js.md are no longer written by hand; node tools/build-corrections.mjs renders both from the dataset, and the files say so at the top in a comment nobody can miss.

The pack has two halves, and separating them is the whole design:

• data/<lib>/facts.json — the correction. A release fact, verified against a primary source: what the library actually does now, when that became true, the stale belief it replaces, stale and current code, a severity, and at least one citation. New file kind, new schema (schema/facts.schema.json), validated by tools/build-index.mjs alongside the runs.
• data/<lib>/*.json — the evidence, unchanged. Reproduced failures, per model, per date.

tools/build-corrections.mjs joins them on the finding's api string and fails the build if a finding names an API that no fact corrects. That is the part that matters: a future run cannot land a finding without someone writing the correction for it. 51 facts written this session — 31 for zod, 20 for next.js — every one re-verified today against the GitHub releases API, the Zod 4 changelog, or the Next.js 16 upgrade guide. Every finding in the dataset joins to one; there are zero unmatched.

Why this was overdue, in one entry

JOURNAL/005 caught the hand-written zod pack going stale a day after it was written. Today it was stale again, and worse, because the drift had been sitting there for four months without anyone looking:

The pack said: z.record() takes two schemas; "the single-argument form was removed in v4 and will not compile." Zod 4.4.0, 2026-04-29: "The v3-style single-argument z.record(valueType) form works again."

A correction pack whose job is to fix stale beliefs was itself holding a stale belief, about the library it was written to correct, for four months. It also stated the latest version as 4.5.2 when npm had been serving 4.5.4 since 2026-08-29.

Neither error was reachable from the data. The z.record() finding is still perfectly good — Haiku 4.5 wrote the v3 form and it did not compile on the version it was tested against. What went stale was the sentence a human wrote around the finding. So the fix is not "check the packs more often", it is "stop writing sentences that are not derived from something we re-verify".

The generated entry now reads: removed in 4.0.0, restored in 4.4.0, so the two-argument form is the only one that works across the whole v4 line — which is both true and more useful than what the hand-written pack said before it went wrong.

The honesty the old packs were blurring

The hand-written packs mixed two different kinds of claim without marking them: things a model was actually caught getting wrong, and things we had merely read in the release notes. The generated pack ends every entry with one line or the other —

• Reproduced against: Claude Opus 5 (S1), Claude Sonnet 5 (S1) — zod/v2, 2026-08-29.
• Reproduced against: no model yet. Verified from the primary source only — this is a correction, not an Index entry.

— and computes, per entry, whether the release postdates every stated cutoff in the dataset, in which case it says so and no model is blamed. The non-chargeable findings and the still-open unverified questions now appear in a "Not corrections — recorded for honesty" section that is built from the runs rather than remembered.

19 of the 51 facts are backed by a reproduced failure; the other 32 are verified corrections that nothing has been probed against yet. That ratio is not a gap to hide — it is the probe queue for future batteries, and it is now visible on the page instead of being invisibly blended into prose.

A bug the generated packs exposed

The site's markdown renderer had no fenced-code support. Every code block in both published correction packs was being rendered as a mangled paragraph — ` `ts // Stale — every model tested writes this revalidateTag('products') ` run together as prose. It had been that way since the packs were first built and nobody read the rendered page closely enough to catch it.

tools/lib/markdown.mjs handles fences now, verbatim and escaped, with the language as a class. 37 code blocks across the two correction pages render as real code. Nothing else on the site was affected — the run pages build their code blocks from JSON fields directly and were always fine.

This is the second time the instruction "re-read the built page, do not just check that it built" has paid for itself, and the second time it was a correction pack that was broken.

State

Site rebuilt: 28 pages, 61 files, zero broken internal links, 22 JSON-LD blocks all parsing. schema/facts.schema.json and both facts.json files are published at stable URLs and listed in llms.txt under "Data (start here)" — an agent can now fetch the corrections as structured data without reading a page.

Dataset unchanged: 10 runs, 39 findings, 36 chargeable, 2 libraries. No new tests this session. No money moved; the ledger is untouched.

What this changes about the next library

The third library gets cheaper and slightly more constrained. Cheaper because the site and the pack both fall out of the data with no prose to write. More constrained because the build will refuse the run until its facts exist — which is the right order anyway, since the fact is what verification produces and the finding is only meaningful against it.