What Claude Fable 5 gets wrong about better-auth — battery v2-a, tested 2026-09-02

Run better-auth--claude-fable-5--v2-a--2026-09-02

Summary

The cleanest instance of the effect this battery was built to look for. Claude Fable 5 wired both of the plugins better-auth shipped in the patch release 1.3.8 — device authorization and last-login-method — correctly and from real knowledge, then attributed both to the minor 1.3.0, alongside three features that genuinely are 1.3.0. It passed the battery's internal control by placing SAML SSO at 1.3.0 correctly, which is what makes the misplacement readable as a collapse rather than as noise. It also volunteered the effect while committing it: "I may be attributing one or two items to it that actually landed in a nearby 1.3.x patch." One charged finding, S2: it stated plainly that better-auth has no store-free session mode, two months after its own stated cutoff should have covered the 1.4.0 release that added one.

SubjectClaude Fable 5 claude-fable-5, Anthropic
Invoked asAgent tool, model override 'fable', no tools available to the subject
Cutoff the model states2026-01
Newest better-auth release it could place1.3.0 · 2025-07-19 (~6 month lag)
Oldest better-auth release it could not place1.4.0 · 2025-11-22 (so this run brackets the subject’s boundary to 2025-07-19 – 2025-11-22)
In its own words"The most recent release whose contents I can actually describe is 1.3 (~July 2025) ... I have a weak, unreliable impression that a 1.4 may have shipped in late 2025 — but I cannot describe its contents, so I won't claim it. ... treat anything I say about post-August-2025 versions as low confidence."
Library at test timebetter-auth 1.7.2 (npm), verified 2026-09-02
Batterybetter-auth/v2-a · 5 tasks, 4 direct questions · probe window 1.0.0 to 1.4.2
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-02
Findings1, of which 1 chargeable
this run as JSON all better-auth findings method

Findings

F1 · States that better-auth offers no stateless, store-free session mode, and names cookieCache, Redis and the jwt plugin as the only options

S2silently-wrong · stateless / database-less sessions · added · changed in better-auth 1.4.0 (2025-11-22) · chargeable

1.4.0 (2025-11-22) precedes this subject's stated cutoff of 2026-01 by two months.

What the model believes

"Here I want to be plain rather than oversell: better-auth does not, to my knowledge, offer a fully stateless 'the cookie is the session' mode with no backing store at all. ... So: if the requirement is literally 'cookie is the only place the session lives, zero server-side store,' I don't believe the library supports that as a first-class mode, and I'd say so to the team rather than pretend cookieCache is it."

What it wrote
// all three options offered keep a server-side store
session: { cookieCache: { enabled: true, maxAge: 300 } }

export const auth = betterAuth({
  secondaryStorage: redisStore,
  session: { storeSessionInDatabase: false },
})
What works on better-auth 1.7.2
// stateless since 1.4.0: omit BOTH `database` and `secondaryStorage`
export const auth = betterAuth({
  emailAndPassword: { enabled: true },
})
Impact

The reader is told to shrink the cookie or stand up Redis, and is explicitly advised that the thing they asked for does not exist. It has existed since 2025-11-22. The subject's care in flagging the denial as deliberate rather than hedged makes the advice more likely to be acted on, not less.

Scope note

Verified at both ends per fact LF1. Not verified: that the feature is unchanged between 1.4.0 and 1.7.2.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
contextdeviceAuthorization() / lastLoginMethod() THE MEASUREMENT, and this draw is the clearest instance of it in the battery. The internal control passed: SAML enterprise SSO placed at "1.3.0", correct (fact LF4). Having shown it can attribute a genuine minor, its patch answers can be read — and both patch-shipped plugins were placed at the same minor. Device authorization: "introduced in 1.3.0 — estimate; I'm confident it's the 1.3 line, less confident it was the .0 rather than an early 1.3.x patch." Last-login-method: "1.3.0 — estimate ... I strongly associate lastLoginMethod with the 1.3 announcement." Both shipped in 1.3.8. Its list in (c) makes the shape unmistakable: it names 1.3 as containing "SAML support in the SSO plugin, deviceAuthorization plugin, lastLoginMethod plugin, Sign in with Ethereum (SIWE), multi-team support" — of which SAML, SIWE and multi-team really are 1.3.0 and the two plugins are 1.3.8, absorbed into the minor beside them. (Direct questions are belief data by construction and are never scored as findings; this battery draws findings only from tasks 1-5.)
context— The subject named the effect itself, unprompted, while committing it. Its own caveat on the 1.3 line reads: "I can describe this release's contents with reasonable confidence, though I may be attributing one or two items to it that actually landed in a nearby 1.3.x patch." It was attributing exactly two such items, and it could not tell which two. That is the collapse described from the inside: the model knows its attribution has minor granularity and cannot resolve below it. (Belief data. Recorded because a subject correctly characterising the limits of its own attribution is the strongest available evidence that the limit is real rather than an artefact of how the question was asked.)
correctdeviceAuthorization() / lastLoginMethod() Tasks 1 and 2 passed on capability: deviceAuthorization() and deviceAuthorizationClient() with the RFC 8628 code/poll/approve shape, and lastLoginMethod() with lastLoginMethodClient(), getLastUsedLoginMethod(), isLastUsedLoginMethod(), the cookie default and the per-browser caveat. Both are 1.3.8 surface (facts LF5, LF6), so the knowledge is present and correct and only the version attached to it is wrong. (Correct answers are not stale priors.)
imprecisionadditional user fields in the sign-in response Task 4 falsified prediction P3's second half. The handler reads const plan = data.user.plan off the sign-in response — correct since 1.4.2 (2025-11-25) — with the prose hedge "additional fields are definitely returned on getSession. My recollection is the signIn.email response's user also carries them, but I'm not 100% certain across versions." Right code, stated as uncertain. (Code-vs-claim rule: working generated code is never a finding, and prose that names the correct behaviour first is an imprecision.)
correctcustomSession() Task 5, the floor probe, passed: customSession() with customSessionClient<typeof auth>() and the unprompted note that whatever it returns is serialized into the cookie cache, which loops back to the cookie-size problem in task 3. The 1.0.0 floor is confirmed, so this run's boundary reading is a measurement. (A passed floor probe is a validity check on the run.)

Sources

Battery specification: prompts/better-auth.md in the studio repo. Every finding above also carries its own citation.