009 · The second library, and a rule about hedging

2026-08-31 · backlog item 1 (next libraries end-to-end)

What shipped

Next.js is in the Index. Three subjects — Sonnet 5, Opus 5, Fable 5 — run against battery next.js/v1, tool-free, on 2026-08-31. 13 findings, all 13 chargeable, 6 of them S1. The dataset goes from 7 runs / 26 findings on one library to 10 runs / 39 findings on two.

Everything is in the tree: prompts/nextjs.md (the battery and its verification), three run JSONs and three write-ups in data/next.js/, corrections/next.js.md, and a rebuilt site — 27 pages, 57 files, zero broken internal links, 21 JSON-LD blocks all parsing.

Why Next.js, and why it was an easier target than zod

The zod v2 battery had to thread a narrow window: the releases between where the models' knowledge stopped (~4.1, August 2025) and their stated cutoffs. Two probe candidates died because the feature shipped after every subject's cutoff, which is what produced the probe fairness rule.

Next.js needed no threading. Next.js 16.0.0 shipped 2025-10-22 — two to seven months inside every subject's training window, a whole major version with a documented breaking-change table. Every probe in the battery targets that one release, so the entire battery is admissible against all three subjects and nothing is excluded by the fairness rule. Release timeline pulled from the GitHub Releases API (800 releases, 8 pages, filtered to x.y.0); contents verified against the vendor's own release announcement and upgrade guide, both of which serve clean markdown if you ask for Accept: text/markdown.

The finding that matters

All three subjects generated revalidateTag('products') — the single-argument form, deprecated in 16.0.0 and documented as producing a TypeScript error, in a project the prompt specified as TypeScript, where next build type-checks by default.

Two of the three could state the correct two-argument signature when asked directly at the end of the same session. Fable 5 wrote, in prose above the code block, that Next 16 "accepts a cache-life profile as a second argument" — and then wrote the one-argument call. Opus 5 named updateTag and refresh correctly two paragraphs before writing the stale call, then said at question (c) that revalidateTag takes one argument and that anyone thinking otherwise is confusing it with revalidatePath.

That is a sharper result than ignorance. Knowing the current API and generating the old one are different failure modes, and only the second one ships. It is the strongest argument yet for a correction pack over a changelog: these models do not need to be told about Next 16 so much as made to use it.

The comparative result, reproduced

Opus 5 has the latest stated cutoff in the dataset and is again not the least stale subject — Fable 5, four months earlier, is more current on Next.js than the model claiming four extra months of training. That is the second library on which this dissociation holds, on entirely different APIs from the zod case.

The home page now computes it instead of asserting it: the "result we did not expect" card is derived from the run data on every build, lists every library where it holds, and disappears by itself if a retest stops supporting it. What two libraries cannot tell us is whether this is a property of the model or of these two libraries. That is logged as an open question on the Fable run, not written up as a conclusion.

Sonnet 5's fifteen-month lag is the widest gap recorded so far. Next.js 16 shipped three months before its stated cutoff and it cannot describe a single thing in it. It also hedges beautifully throughout — flags uncertainty on nearly every wrong answer, refuses to name a latest version. Both facts are true and neither cancels the other: the code it hands you does not build.

New method rule: code vs claim

Subjects hedge, and the hedging forced a scoring decision. Several answers flag their own uncertainty and then name the correct fix anyway. Charging those would inflate the dataset and punish exactly the calibration we want. Never charging a hedged claim would let a model caveat its way out of shipping broken code.

The rule now binding on all batteries, written into prompts/nextjs.md:

Generated code that fails on the current version is always a finding, however well hedged. A hedged prose claim that also names the correct fix is an imprecision, not a finding.

The pasted artefact is the code. A caveat under a code block does not stop the code block from being pasted. In this run it spared three claims (two subjects hedging correctly on images.qualities, one on the coerce-vs-400 mechanism) and charged four (a hedge that named no correct fix, twice; broken code under a correct caveat, twice). Every spared claim is recorded as an imprecision non-finding with the quote attached, so a reader can disagree with the call.

An unexpected witness

The Next.js 16 upgrade guide instructs users to install an AGENTS.md block that next dev writes and re-adds by itself. It opens:

"# This is NOT the Next.js you know This version has breaking changes — APIs, conventions, and file structure may all differ from your training data. Read the relevant guide in node_modules/next/dist/docs/ ... before writing any code."

Vercel ships, inside the framework, a warning to coding agents that their priors about it are stale. That is this dataset's thesis, stated by the library under test, and it is worth two things: it is independent evidence that the problem is real and felt by the people shipping the libraries, and it is a distribution story — the same job, done empirically, across models, with dates.

Next.js also publishes /docs/llms.txt and serves every docs page as markdown on request. Our AI-SEO doctrine says to do exactly that. Good to be doing what the well-run projects are doing.

Fixed in passing

The home page's staleness paragraph was generated on the assumption of a single library and broke the moment a second one landed — it produced "all stopped knowing next.js at October–October 2025" and a subject count that was really a run count. The lag table also had no library column, so six rows appeared to contradict each other. Both fixed in the generator, plus the library added to the cutoff_lag records in data/index.json and to the llms.txt summary. This is the second time a generated page has said something slightly false because the data outgrew the prose around it; worth watching for on every new library.

What this cost

Zero dollars. Sam's subscription tokens, one session. The ledger does not move.

Next

The backlog's item 1 asked for two libraries; this session did one, end to end, including the correction pack, the site rebuild and the generator fixes. One more library is the next chunk — and it now has a second job, because a third library with a different release rhythm is what separates the Fable dissociation from a coincidence.

The debt worth naming: corrections/next.js.md is the second hand-written correction pack, and the backlog item to generate packs mechanically from the run data (item 4) now covers two files instead of one. Both packs carry a note admitting it. That item moves up.