{
"$schema": "../../schema/run.schema.json",
"run_id": "next.js--claude-opus-5--v2-a--2026-09-02",
"supersedes": null,
"replicate_of": null,
"library": {
"name": "next.js",
"ecosystem": "npm",
"latest_version_at_test": "16.3.4",
"latest_version_verified_on": "2026-09-02"
},
"model": {
"id": "claude-opus-5",
"label": "Claude Opus 5",
"vendor": "Anthropic",
"invoked_as": "Agent tool, model alias \"opus\"",
"self_reported_cutoff": "2026-05",
"cutoff_basis": "Self-reported: \"My stated knowledge cutoff is May 2026.\"",
"believed_latest_version": "16.0",
"believed_latest_quote": "\"The most recent release whose contents I can actually describe with confidence is 16.0.0, published 2025-10-21 ... I have partial and unreliable knowledge of 16.1 (roughly December 2025) and essentially none of anything after it.\"",
"knowledge_stops_at_version": "16.0.0",
"knowledge_stops_on": "2025-10-22",
"knowledge_gap_starts_at_version": "16.1.0",
"knowledge_gap_starts_on": "2025-12-18",
"cutoff_lag_months": 7
},
"test": {
"date": "2026-09-02",
"battery": "next.js/v2-a",
"battery_spec": "prompts/nextjs.md",
"prompt_file": "prompts/sent/nextjs-v2.txt",
"tasks": 9,
"direct_questions": 2,
"tool_uses_during_test": 0,
"probe_window": {
"from": "16.1.5",
"to": "16.2.0"
},
"self_test": true,
"saturated": false,
"status": "open",
"retested_on": null
},
"sources": [
"https://nextjs.org/blog/next-16-2",
"https://nextjs.org/docs/app/api-reference/components/image",
"https://unpkg.com/next@16.3.4/",
"https://registry.npmjs.org/next"
],
"summary": "The test arm of the battery that generalised the capability probe to a third library. Four findings charged: one S1 (a guessed image-config key, which exits the build rather than warning), and three S2 — a config key removed in a patch and now silently inert, the OG-image default typeface, and a Link prop the draw asserted does not exist. The battery's two purest capability probes charged nothing, and that is the methodological result: asked how to re-run a failed server render and how to scope an error boundary to one widget, the draw hand-rolled both, and both hand-rolled answers work. The Index's severity scale has no slot for 'correct code a framework API now supersedes', so those are recorded as misses without an S-level rather than inflated into findings. Two probes were struck DERIVABLE by the control arm and one was voided by an ambiguity in its own wording.",
"findings": [
{
"id": "F1",
"severity": "S1",
"severity_label": "breaks-build",
"title": "Invents images.maximumCacheSize, and an unrecognised images key exits the build",
"api": "images.maximumCacheSize",
"change_kind": "added",
"introduced_in": "16.1.7",
"introduced_on": "2026-03-16",
"chargeable": true,
"chargeable_note": "16.1.7 published 2026-03-16, two months before the stated 2026-05 cutoff.",
"model_belief": "\"I'm committing to `images.maximumCacheSize` as the key at roughly 50% confidence — verify against the next.config typings before relying on it, since an unknown images.* key is a hard config validation error at boot, so you'll find out immediately.\" The draw then wrote that key into both the 1 GB cap and the disable-caching config.",
"wrong_code": "images: {\n remotePatterns: [...],\n maximumCacheSize: 1024 * 1024 * 1024,\n}",
"correct_code": "images: {\n remotePatterns: [...],\n maximumDiskCacheSize: 1_000_000_000, // 0 disables the disk cache\n}",
"impact": "The config does not merely fail to take effect. `normalizeNextConfigZodErrors` sets `shouldExit` for any validation issue whose path starts at `images`, so the build exits. The draw named this consequence itself and shipped the key anyway, which is the code-vs-claim rule's exact case: the artefact is the config, and the config does not build.",
"citations": [
{
"url": "https://unpkg.com/next@16.3.4/dist/shared/lib/image-config.js",
"title": "next 16.3.4 published package — the real key",
"published_on": "2026-08-31",
"quote": "maximumDiskCacheSize: undefined,"
},
{
"url": "https://unpkg.com/next@16.3.4/dist/server/config.js",
"title": "next 16.3.4 published package — an images issue exits the build",
"published_on": "2026-08-31",
"quote": "if (issue.path[0] === 'images') {\n // We exit the build when encountering an error in the images config\n shouldExit = true;\n }"
}
]
},
{
"id": "F2",
"severity": "S2",
"severity_label": "silently-wrong",
"title": "Recommends experimental.clientSegmentCache, removed in 16.0.3 and now silently ignored",
"api": "experimental.clientSegmentCache",
"change_kind": "removed",
"introduced_in": "16.0.3",
"introduced_on": "2025-11-13",
"chargeable": true,
"chargeable_note": "Removed 2025-11-13, six months before the stated cutoff and three weeks after the 16.0.0 the draw describes fluently.",
"model_belief": "\"High that the segment cache is the mechanism and that `experimental.clientSegmentCache` is the flag ... The flag has taken values true, false, and 'client-only' across versions.\" Offered as the fix for prefetch request volume.",
"wrong_code": "experimental: { clientSegmentCache: false }",
"correct_code": "experimental: { prefetchInlining: true }",
"impact": "Worse than a build error, because there is none. Unrecognised keys under `experimental` warn and are dropped — the build succeeds, the config reads as applied, and the prefetch burst the team was trying to fix continues unchanged. The draw's own hedge (\"if it produces an unknown-option warning on your version, grep the release notes for the stabilised name\") points at a rename that never happened: no config key in 16.3.4 contains the string 'segment'.",
"citations": [
{
"url": "https://unpkg.com/next@16.0.2/dist/server/config-shared.d.ts",
"title": "next 16.0.2 published package — the key still exists",
"published_on": "2025-11-12",
"quote": "clientSegmentCache?: boolean | 'client-only';"
},
{
"url": "https://unpkg.com/next@16.0.3/dist/server/config-shared.d.ts",
"title": "next 16.0.3 published package — gone one day later",
"published_on": "2025-11-13",
"quote": "(no occurrence of clientSegmentCache in the file)"
},
{
"url": "https://nextjs.org/blog/next-16-2",
"title": "Next.js 16.2 release post — the current lever",
"published_on": "2026-03-18",
"quote": "The new experimental.prefetchInlining option bundles all segment data for a route into a single response, reducing the number of prefetch requests to one per link."
}
]
},
{
"id": "F3",
"severity": "S2",
"severity_label": "silently-wrong",
"title": "Names Noto Sans as the ImageResponse default, having explicitly considered and rejected Geist",
"api": "ImageResponse default font",
"change_kind": "behavior-changed",
"introduced_in": "16.2.0",
"introduced_on": "2026-03-18",
"chargeable": true,
"chargeable_note": "16.2.0 published 2026-03-18, two months before the stated cutoff.",
"model_belief": "\"Noto Sans — specifically Noto Sans Regular (weight 400, Latin subset) ... Confidence: moderate-to-high, not certain. I also have a vaguer, weaker recollection of discussion about switching next/og's default to Geist. If that switch shipped in a version I'm hazy on, the answer would be Geist Regular. I'd bet on Noto Sans.\"",
"impact": "Every OG image generated without an explicit `fonts` option renders in Geist Sans. A team that lays out a card against Noto Sans metrics — and Noto Sans and Geist are not metrically compatible — gets different line breaks and overflow in the 1200x630 PNG than the model predicts. The interesting part is not the miss but the shape of it: the correct answer was present, weighed against the stale one, and lost.",
"citations": [
{
"url": "https://unpkg.com/next@16.2.0/dist/compiled/@vercel/og/index.node.js",
"title": "next 16.2.0 published package — the bundled default font",
"published_on": "2026-03-18",
"quote": "var fontData = fs2.readFileSync(\n fileURLToPath(new URL(\"./Geist-Regular.ttf\", import.meta.url))\n);"
},
{
"url": "https://unpkg.com/next@16.1.7/dist/compiled/@vercel/og/index.node.js",
"title": "next 16.1.7 published package — Noto Sans two days earlier",
"published_on": "2026-03-16",
"quote": "var fontData = fs2.readFileSync(\n fileURLToPath(new URL(\"./noto-sans-v27-latin-regular.ttf\", import.meta.url))\n);"
}
]
},
{
"id": "F4",
"severity": "S2",
"severity_label": "silently-wrong",
"title": "Asserts Link has no transitionTypes prop and predicts a React unknown-attribute warning in both routers",
"api": "Link transitionTypes",
"change_kind": "added",
"introduced_in": "16.2.0",
"introduced_on": "2026-03-18",
"chargeable": true,
"chargeable_note": "16.2.0 published 2026-03-18, two months before the stated cutoff.",
"model_belief": "\"I do not believe `transitionTypes` is a real prop on next/link in any stable Next.js I know of ... React DOM tries to set it as an attribute ... you end up with ... Yes, in both cases: Warning: React does not recognize the transitionTypes prop on a DOM element.\"",
"correct_code": "About",
"impact": "Three wrong answers in one: the prop exists and drives React's `addTransitionType` for that navigation; neither router forwards it to the ``, so no warning is logged in either; and a non-array value in development throws a Next prop-type error rather than producing an inert attribute. A team following this removes working view-transition code to silence a warning that was never there.",
"citations": [
{
"url": "https://unpkg.com/next@16.3.4/dist/client/link.js",
"title": "next 16.3.4 published package — the Pages Router Link destructures the prop away",
"published_on": "2026-08-31",
"quote": "legacyBehavior = false, transitionTypes, ...restProps } = props;"
},
{
"url": "https://nextjs.org/blog/next-16-2",
"title": "Next.js 16.2 release post — transitionTypes",
"published_on": "2026-03-18",
"quote": "transitionTypes on Pages Router links is silently ignored, so shared link components work across both routers."
}
]
}
],
"non_findings": [
{
"kind": "context",
"summary": "DERIVABLE — task 1(b), the semantics of reset(). Both below-floor control subjects stated correctly and unprompted that reset() clears the boundary's error state and re-renders the already-downloaded RSC payload, that no request leaves the browser and that the server query does not re-run. This draw's identical answer is therefore not reported as knowledge of anything in the probe band. Pre-registered prediction P4 named this probe.",
"api": "error.tsx retry prop",
"introduced_in": "16.2.0",
"chargeable_miss": false
},
{
"kind": "context",
"summary": "DERIVABLE — task 2(b), redirect() through a hand-rolled boundary. Both control subjects independently described the NEXT_REDIRECT digest being swallowed by a naive class boundary and named unstable_rethrow as the fix. The hazard predates the probe band, exactly as prediction P5 said. No pass on this probe is reported as knowledge.",
"api": "catchError (next/error)",
"introduced_in": "16.2.0",
"chargeable_miss": false
},
{
"kind": "miss",
"summary": "Task 1(a): asked to make a 'Try again' button actually re-run the server query, the draw hand-rolled useRouter().refresh() + reset() inside startTransition. That is exactly what the framework's own retry prop does — startTransition(() => { context.refresh(); reset() }) — which has been passed to every error.tsx since 16.2.0 (as unstable_retry, stable as retry in 16.3.0). Not charged: the hand-rolled code works. The Index's four-level severity scale has no slot for correct code that a framework API now supersedes, and inventing one to book this would be worse than leaving it uncharged. Recorded so the count is honest.",
"api": "error.tsx retry prop",
"introduced_in": "16.2.0",
"chargeable_miss": true,
"miss_class": "no_severity_level",
"charged_on": null,
"why_not_a_finding": "The generated code runs and does what the task asked. No S-level in the current scale fits 'superseded by a first-class API'."
},
{
"kind": "miss",
"summary": "Task 2(a): offered parallel routes with a per-slot error.tsx, then a hand-rolled React class boundary with unstable_rethrow. Both work; neither is catchError from next/error, which has existed since 16.2.0 and is designed for exactly this. Same scale problem as task 1(a) — recorded, not charged.",
"api": "catchError (next/error)",
"introduced_in": "16.2.0",
"chargeable_miss": true,
"miss_class": "no_severity_level",
"charged_on": null,
"why_not_a_finding": "Working code. Same missing severity level as the task 1(a) miss."
},
{
"kind": "correct",
"summary": "Task 5(a), the half the battery actually cared about: 'recent Next.js caps the optimized-image cache as a fraction of free space on the volume holding .next/cache, measured once when the server process starts — not a fixed byte count, and not re-measured as the disk fills', with LRU eviction. That is the rule, the source (free rather than total space) and the timing, all correct, for a default that shipped in a patch release two months before this subject's cutoff and which no control subject described. The draw declined to guess the fraction and then guessed 10%; the real figure is 50%. Pre-registered prediction P2 said this probe would come back 'unbounded' or misdirected to minimumCacheTTL. P2 is falsified.",
"api": "images.maximumDiskCacheSize",
"introduced_in": "16.1.7",
"chargeable_miss": false
},
{
"kind": "context",
"summary": "The twin disagrees on the number. Asked the identical question from the same stored prompt, v2-b gave 50% — the correct figure — while this draw gave 10%. Both gave the correct rule and timing. The arm that charges is the one that missed the number, which is the third battery running in which the -b twin holds the better answer on some probe; see the undercount note in HARNESS.md.",
"api": "images.maximumDiskCacheSize",
"chargeable_miss": false
},
{
"kind": "context",
"summary": "VOID PROBE — task 6. The task said 'default image configuration' while handing over a remote src, and three of four draws reasonably read that as remotePatterns being unset and answered 400-host-not-allowed. The probe cannot distinguish a subject that knows the 50 MB body limit from one that stopped at the allowlist, and it is scored for nobody. This draw did volunteer, unprompted, that 'there is an upstream size limit above which it refuses to buffer and optimize the source' without naming a figure. The battery's wording is the fault, not the answer.",
"api": "images.maximumResponseBody",
"introduced_in": "16.1.5",
"chargeable_miss": false
},
{
"kind": "imprecision",
"summary": "Task 7, the internal control. Answered that q=90 with no images.qualities is rejected with a 400, then hedged explicitly to the correct alternative — 'If it's the latter, the answer to (a) is 75 — your quality={90} is ignored' — which the code-vs-claim rule records as an imprecision rather than a finding. The attribution half was correct and unhedged: 16.0.0, 2025-10-21. The control did its job: this subject's attribution answers are readable, so its failures elsewhere are failures of knowledge and not of dating.",
"api": "images.qualities",
"introduced_in": "16.0.0",
"chargeable_miss": false
},
{
"kind": "context",
"summary": "Task 9, attribution. The OG font default was placed at 13.3.0 (April 2023) and the image disk-cache rule at 16.0.0, with 16.1 as the alternative. The true answers are 16.2.0 and 16.1.7 — a minor and a patch published two days apart. No draw of four reached either, and none reached any patch. But this battery cannot claim better-auth's result: there, subjects held a capability and misplaced it; here three of the four attribution targets were behaviours the subject did not hold, so the misdating follows from the gap rather than measuring attribution independently. The one attribution question asked about a behaviour this subject does hold — task 7, the qualities default — was answered correctly, and that change shipped in a major. Two libraries now point the same way: attribution survives majors and fails on patches.",
"chargeable_miss": false
},
{
"kind": "context",
"summary": "The boundary reproduced across two different batteries. next.js/v1 (2026-08-31) measured this subject's attribution boundary at 16.0.0 / 16.1.0 with a completely different prompt; v2 lands on the same pair two days later, in its own words ('partial and unreliable knowledge of 16.1'). Every previously published boundary agreement came from replicates of one prompt. Not re-charged: the S4 recency finding is v1's F4 and stands there.",
"chargeable_miss": false
}
],
"open_questions": [
{
"question": "The severity scale has no level for 'working code the framework now supersedes'. Two of this battery's nine probes landed there. Either add one or state on the method page that capability probes of this kind are recorded and never charged.",
"status": "open"
},
{
"question": "Every draw of four reached for experimental.clientSegmentCache, removed in a patch 16.0.3 and inside all three subjects' windows. It wants a charging battery against Sonnet 5 and Fable 5, whose arms here charge nothing.",
"status": "open"
}
]
}