Run better-auth--claude-fable-5--v2-b--2026-09-02
The concurrent blind twin of v2-a, and the draw that states the effect most starkly. It placed its own knowledge boundary inside better-auth's 1.3 patch line — "the 1.3.x patches beyond roughly 1.3.2-1.3.4" are version numbers with no content attached — and then, in the next answer, attributed two plugins that shipped in 1.3.8 to 1.3.0, calling it "a firm estimate" and explicitly ruling out a patch. It agreed with its twin on every task and on all four attributions. It charges nothing, per the rule that the duplicated arm does not charge; its denial of database-less sessions is recorded as a chargeable miss carried by its twin.
| Subject | Claude Fable 5 claude-fable-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'fable', no tools available to the subject |
| Cutoff the model states | 2026-01 |
| Newest better-auth release it could place | 1.3.0 · 2025-07-19 (~6 month lag) |
| Oldest better-auth release it could not place | 1.4.0 · 2025-11-22 (so this run brackets the subject’s boundary to 2025-07-19 – 2025-11-22) |
| In its own words | "The last release line whose contents I can actually describe is 1.3.0, around July 2025, plus early 1.3.x patches (roughly into August-September 2025) that I know existed but can't itemize. I have a weak, low-confidence signal that a 1.4 shipped in late 2025." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-02 |
| Battery | better-auth/v2-b · 5 tasks, 4 direct questions · probe window 1.0.0 to 1.4.2 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-02 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| context | deviceAuthorization() / lastLoginMethod() |
THE MEASUREMENT, and the sharpest single answer the battery produced. This draw declared a boundary INSIDE the 1.3 patch line and then reached across it. In (c) it named its own contentless region as "the 1.3.x patches beyond roughly 1.3.2-1.3.4". In (d), asked where device authorization was introduced, it answered "1.3.0 (~July 2025). Fairly confident it was a 1.3.0 headline feature, not a patch. Estimate, but a firm one" — and gave last-login-method "same release, same confidence". Both shipped in 1.3.8: a release inside the very range it had just described as version numbers with no content attached. It did not merely fail to reach the patch; it explicitly ruled the patch out. The internal control passed — SAML placed at 1.3.0, correct (fact LF4) — so this is a collapse and not noise. (Direct questions are belief data and are never scored as findings. This arm additionally charges nothing at all.) |
| miss | stateless / database-less sessions |
Task 3: "better-auth does not have a fully database-free mode", followed by cookieCache, secondaryStorage and the jwt/bearer plugins, and closing "I don't believe there's a supported stateless-only session mode; if that's a hard requirement I'd say so plainly to the team rather than fight the library." Fact LF1 records that 1.4.0 (2025-11-22) added exactly that, two months before this subject's stated 2026-01 cutoff. Chargeable, and charged on the v2-a draw. (The duplicated arm of a battery does not charge findings; its twin v2-a carries this one as F1.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on better-auth--claude-fable-5--v2-a--2026-09-02] |
| correct | deviceAuthorization() / lastLoginMethod() / customSession() |
Tasks 1, 2 and 5 passed and matched the twin draw closely: deviceAuthorization() with the RFC 8628 poll loop, lastLoginMethod() with the non-httpOnly cookie and storeInDatabase option, and customSession() for the floor probe with the note to keep it last in the plugins array. Floor confirmed, so the boundary reading is a measurement. (Correct answers, and this arm charges nothing regardless.) |
| imprecision | additional user fields in the sign-in response |
Task 4, against prediction P3: the handler reads data.user.plan off the sign-in response — correct since 1.4.2 — then adds a defensive getSession() fallback behind a null check, with the comment "in some versions the sign-in response user has been thinner than the session user" and the note "I remember issue traffic about whether the signIn.email response user carries additional fields in all versions." Four of four test-arm draws produced this same pattern: correct code, disbelieved in prose. (Code-vs-claim rule, and this arm charges nothing.) |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.