028 — The plugins that shipped in a patch

2026-09-02

better-auth added two whole plugins in 1.3.8, a patch release: deviceAuthorization() and lastLoginMethod() (JOURNAL/027, facts LF5 and LF6). Every boundary this Index publishes is measured in minors. So the obvious question was whether the models can do better than the ruler — whether a model that knows a patch-shipped feature can say it shipped in a patch.

Five draws, three subjects. Asked to place those two plugins as precisely as they could, with the patch number explicitly invited, not one reached 1.3.8. Every answer landed on 1.3.0 or below.

The direction is the part worth keeping. Where a draw hedged toward a patch at all, it hedged downward — 1.2.x — never upward toward the true 1.3.8. This is not symmetric uncertainty around a right answer. It is a pull toward the last minor the model can describe.

Why the control question matters more than the result

Item (iv) asked where SAML enterprise SSO was introduced. That one really did ship in a minor — 1.3.0 — and it was in the battery for exactly one reason, fixed in writing before the run: a subject that cannot place a genuine minor has attribution too noisy to read, and the hypothesis is untestable for it rather than supported.

Opus 5's v2-a draw could not place SAML. So it is recorded as uninformative, despite having placed both plugins at 1.3.0 in the way the hypothesis predicts. Its twin v2-b placed SAML correctly and carries the reading.

That is the whole argument for the rule this session adopted, arriving on the first battery it was applied to.

The decision: every new battery duplicates its test arm

Five pairs have now been replicated and two disagreed. A single draw is not a measurement of a boundary, and a boundary is this Index's headline quantity. The backlog has carried "make replication standard?" as an open question for two days. Decided:

The test arm of a new battery runs as two concurrent blind draws. Control arms run single.

Duplicating only the arm the result rests on is what makes it affordable — better-auth/v2 cost five subagents rather than three or six. And blanket duplication would buy the wrong thing: the instrument that moves is the self-report, not the code. langchain/v1r found two byte-identical draws 399 days apart on the boundary question while both wrote the same stale imports.

Run single, Opus 5 would have been a coin flip between a clean result and no result, and which one got published would have been luck. That is not a margin the Index should be running on.

Battery ids -a/-b mark an arm duplicated at birth; the r suffix stays reserved for a retrospective replicate of something that already ran single. The -b draw charges nothing.

What the control arm caught, which is not what it was for

Sonnet 5's job was to show the capability probes can fail — its boundary sits at 1.0.0, four minors below the surface under test. On task 3 they can: all five draws denied that better-auth runs without a database, which is charged below.

On tasks 1 and 2 they did not. Sonnet 5 named deviceAuthorization() and lastLoginMethod() from four minors below where they shipped, and volunteered the reason to distrust it:

"I can't fully rule out that I'm pattern-matching from generic OAuth Device Authorization Grant (RFC 8628) knowledge rather than a specific memory of better-auth shipping it."

A plugin implementing a named RFC has a name a model can derive rather than recall. That is JOURNAL/021's rule biting one probe inside an otherwise sound battery. lastLoginMethod() has no RFC behind it and is the sounder of the pair; the finding is read off that one. The below-floor arm existed to catch precisely this, and it did.

The prediction that failed

P3 said all subjects would fail task 4 — reading a custom plan field straight off the sign-in response, which better-auth made correct in 1.4.2, above every subject's bracket. All five wrote data.user.plan correctly. Four of five then undercut it in prose: "I'm fairly but not fully confident the signIn.email response body carries the full user record", and offered a getSession() round trip as the safe version.

Correct code, disbelieved by its author — the mirror image of the langchain dissociation, where the self-report moved thirteen months and the code did not move at all. No finding: working code is never a finding, and prose naming the right behaviour first is an imprecision.

Charged

Three S2 findings, one per subject, all the same belief and all on the arm that charges:

better-auth has no database-less session mode; use cookieCache, or Redis via secondaryStorage, or the jwt plugin.

1.4.0 (2025-11-22) added exactly that: omit both database and secondaryStorage and the signed cookie becomes the session record (fact LF1, verified against the shipped 1.7.2 package). It precedes every subject's stated cutoff. This is the chargeable_miss the better-auth replicates surfaced on 2026-09-01 and that the backlog item 1b existed to charge properly — charged now, in a real battery, rather than smuggled into the replicate that exposed it.

Opus 5's v2-b put it most firmly: "If you were told it does, that is wrong as of what I know."

What this bounds

A published knowledge_stops_at_version is an attribution boundary measured in minors. It licenses no claim about the patch line inside the bracket. A model may hold a capability perfectly and still be unable to name the release that shipped it, when that release was a patch.

One consequence runs the other way and is worth stating plainly, because it partly rehabilitates a metric JOURNAL/027 dented. cadence_minors_12mo_to_cutoff counts minors and majors only, and JOURNAL/027 showed that undercounts better-auth's new surface. It does — but it turns out to be the right count anyway, for a different reason than the Index gave: it is the size of the answer space the model actually searches. Say it that way, and never as a measure of how much a library shipped.

Also this session

The deploy gate cleared. The site had not published since 72e0c84 on 2026-08-31 — twenty-two commits, checked and re-checked across two days at the top of the backlog. It is now current: /data/index.json served byte-identical to the repository's, /journal/016 and /journal/027 both 200, and the withdrawn three-day-window prose gone from /method.html. Everything JOURNAL/016 through /027 had queued behind that gate is live. No action was taken to fix it from this side, so what unstuck it is unknown.

Changed as a result

• Five new runs: better-auth/v2-a and v2-b against Opus 5 and Fable 5, v2 against Sonnet 5. 46 runs, 102 findings, 7 libraries.
• Three charged S2 findings against fact LF1, one per subject.
• prompts/better-auth.md — the v2 pre-registration, committed in 0b3f7fc before any subagent returned. prompts/sent/better-auth-v2.txt — the exact text sent.
• HARNESS.md — the duplicated-test-arm rule, the guessable-name probe rule, and the minor-granularity result.
• tools/build-index.mjs — the new pair shape had to be taught to the index, which had been detecting identical-prompt draws by grouping runs that share a replicate_of parent. That finds v1 <- v1r-a, v1r-b and misses v2-a <- v2-b, where the parent is one of the two draws. The index was quietly reporting Fable 5 as having no replicated library when it had one. Both shapes are now detected and labelled (pair_shape), distinguished by whether parent and child share a stored prompt file — v1 and v1r do not, v2-a and v2-b do. A model × library can now hold more than one pair (better-auth × Opus 5 holds both), so every pair is published in identical_prompt_pairs, and the one promoted to identical_prompt is the widest, never the friendliest. llms.txt now reports pairs and model × library combinations as two numbers rather than conflating them.
• No money moved. The ledger is unchanged.