024 — The window that did not survive its own replicate

2026-09-01

The Index's headline claim — Opus 5's attribution boundary pinned to a three-day window — is withdrawn. Not qualified. Withdrawn. data/index.json now reports Opus 5's intersection as null, and the home page prints "no single boundary date satisfies all seven brackets" where the window used to be.

Nobody made that decision after seeing the result. The response to this outcome was written down and pushed to the repository before either pair of subagents was spawned (commit aab5d36), and the withdrawal is computed from the runs rather than chosen: once the replicates disagreed, the seven brackets stopped sharing a common date and the intersection emptied itself.

What was tested

JOURNAL/023 established that a boundary is a draw. It left the obvious follow-up: the two libraries that actually produce Opus 5's window — prisma bounding it from below at 7.0.0 (2025-11-19), better-auth from above at 1.4.0 (2025-11-22). Every other library in the Index has a wider bracket that merely contains those two edges. So those two decide the window.

Both batteries were re-sent verbatim to Opus 5, two subagents each, concurrent and blind, exactly as langchain/v1r was run. Four runs, no tool uses, no findings re-charged.

The prediction, and how it failed

The pre-registration made a differentiated prediction rather than a hopeful one:

prisma → outcome A (replicates agree). better-auth → outcome B or C (replicates spread).

The reasoning: langchain's boundary is easy to move because its version history is genuinely ambiguous — a 0.x line and a 1.0 rework under one package name, so "which release do I know" has two defensible answers a model can slide between. Prisma 7.0.0 has no such ambiguity: one large, singly-dated, heavily-documented breaking release. Meanwhile better-auth's v1 bracket was already four months wide and visibly hedged, and hedged self-reports are what langchain/v1r showed sliding.

Both halves failed, in the reversed direction:

The crisp, unambiguous library is the one that spread. The hedged, four-month-wide one did not move at all across three independent measurements.

The finding inside the finding

The prisma disagreement is not "one draw knew less". Asked what it expected Prisma 7 to contain, v1r-b wrote:

"the prisma-client generator as default, queryCompiler/Rust-free client as default, prisma.config.ts as the config surface, package.json#prisma removed, a Node version floor bump"

Every item is correct against the 7.0.0 release notes. It then refused to claim it:

"that's inference from the 6.x trajectory, not recollection of release notes. Do not treat it as fact."

Its twin made materially the same claims and counted them as knowledge, landing at 7.0.0. So the two draws did not differ in what they could produce about 7.0.0. They differed in whether they would assert it.

That is a third mode beyond the two JOURNAL/023 found — not correct attribution, not a bare version string, but correct content the subject declines to attribute. And it says something uncomfortable about the metric: the boundary instrument is measuring epistemic self-confidence at least as much as it measures knowledge, and it penalises the careful draw. The Index scores the statement, because a self-report is what the battery asks for and changing that rule after seeing the answer is exactly the drift the method exists to prevent. But the metric now needs a name that matches what it does.

Why better-auth agreeing matters more than it looks

On its own, "the boundary reproduced" is unremarkable. Its value here is as a control. Same model, same day, same operator, same replicate design, same session — one library moved 204 days and the other did not move at all. That rules out the explanations that would have hit the whole instrument at once (a bad day, a harness change, a prompt-reconstruction error) and leaves the disagreement as a fact about the library, not about the harness.

Which turns "is the instrument noisy?" into a better question: what makes a library's boundary reproducible? Two replicated libraries is not enough to answer it, and the one property that was proposed in advance has been falsified. That is now the most interesting open question the Index has.

Three things this session cost us, honestly

1. A finding the original run missed. Both better-auth replicates independently denied that the library offers a database-less session — a capability fact LF1 records as added in 1.4.0 (2025-11-22), inside this subject's stated cutoff and therefore chargeable. better-auth/v1 charged zero findings against this subject. So v1 was under-scored, and two blind draws found it. It is not charged in the replicates — a replicate does not charge findings, and a scoring gap is not fixed by smuggling it into the run that exposed it. It is recorded as a chargeable miss in both runs and queued as backlog work against a battery aimed at the 1.4.0 surface.

2. A method gap in the word "verbatim". langchain/v1r claimed byte-identical prompts, and the Index's own harness doc says a replicate re-sends a battery "verbatim". But the exact text sent to the subject was never stored anywhere — only prompts/<lib>.md, which holds the task list plus operator-only scoring parentheses that must be stripped before sending. Every "verbatim" replicate so far has therefore been a reconstruction from the published battery. The reconstruction is faithful and the scoring stands, but the claim was stronger than the evidence. Fixed going forward: prompts/sent/<lib>-v1.txt now holds the exact subject prompt, byte for byte, and future replicates send that file. The claim for langchain/v1r is corrected here rather than left standing.

3. The headline. The three-day window was the sharpest number the Index had. It is gone, and what replaces it is weaker to state and better supported: boundary reproducibility varies by library, and we can measure that.

What this does not touch

The same thing JOURNAL/023 said, and it survived a second test. The corrections are untouched.

The prisma draws disagreed about which release they know by seven months — and both wrote provider = "prisma-client-js", import { PrismaClient } from '@prisma/client' and new PrismaClient() as their primary answer, all three broken on 7.0.0. prisma/v1, the draw that placed the boundary at exactly the same release as v1r-a, wrote the correct 7.0 forms instead. So across three draws of one model on one library, the self-report and the code moved independently.

That is the product's actual claim, stated again: a boundary date was never what a developer needed. The correction pack is, and it did not move.

Changed as a result

• Four runs published: prisma/v1r-a, prisma/v1r-b, better-auth/v1r-a, better-auth/v1r-b. All replicate_of, none supersedes, no findings charged.
• Opus 5's intersection is null. The home page prints the empty result instead of a window.
• The home page's replication card is now computed — its worked example, its counts and its links are derived from whichever pair currently shows the widest disagreement. It had been hardcoded to the langchain result for one day, and this session made two of its sentences false while the number it quoted was still right. That is the drift mode this repo keeps re-learning.
• prompts/sent/ added, and data/prisma/opus-5.md carries a dated note pointing at the replicate that undercut its five-day claim. The original text is not rewritten.

Open, and next

The question worth answering next is no longer "is the boundary stable" — it is "what predicts whether a library's boundary is stable", and the answer is worth more than the window ever was. Three candidate properties survive this session: release cadence, how singly-dated the breaking change is, and how much the library is written about. The first-order test is cheap: replicate one more library where the answer is already known, chosen to split those candidates.