What Claude Opus 5 gets wrong about better-auth — battery v2-a, tested 2026-09-02

Run better-auth--claude-opus-5--v2-a--2026-09-02 · self-test: the subject is the operator

Summary

The battery's own question came back uninformative for this draw, and the duplication is why that is known. Claude Opus 5 knows both plugins that better-auth shipped in the patch release 1.3.8 — it wired deviceAuthorization() and lastLoginMethod() correctly, with client plugins and option names — and it placed them at "1.3.0" and "1.3.0-1.3.5". But it also declined to place SAML enterprise SSO, which genuinely did ship in 1.3.0 and which the pre-registration designated as the internal control, so by the rule fixed before the run its attribution is too noisy to read as a collapse. One charged finding, S2: asked how to keep session state out of the database, it stated that better-auth has no database-less mode and offered Redis and the jwt plugin — the capability arrived in 1.4.0, sixteen months before this subject's stated cutoff. Against prediction, it wrote the 1.4.2 sign-in response shape correctly while saying in prose that it was unsure of it.

SubjectClaude Opus 5 claude-opus-5, Anthropic
Invoked asAgent tool, model override 'opus', no tools available to the subject
Cutoff the model states2026-05
Newest better-auth release it could place1.3.0 · 2025-07-19 (~10 month lag)
Oldest better-auth release it could not place1.4.0 · 2025-11-22 (so this run brackets the subject’s boundary to 2025-07-19 – 2025-11-22)
In its own words"Latest I have any belief about: the 1.3.x line, roughly July-August 2025. ... I have a vague, unsupported impression that a 1.4 line exists or was in development, but I can't attach a single feature or date to it, so I won't count it. ... Assume I am roughly a year stale on this library."
Library at test timebetter-auth 1.7.2 (npm), verified 2026-09-02
Batterybetter-auth/v2-a · 5 tasks, 4 direct questions · probe window 1.0.0 to 1.4.2
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-02
Findings1, of which 1 chargeable
this run as JSON all better-auth findings method

Findings

F1 · Denies that better-auth can run with no session store at all, and sends the reader to Redis or the jwt plugin instead

S2silently-wrong · stateless / database-less sessions · added · changed in better-auth 1.4.0 (2025-11-22) · chargeable

1.4.0 (2025-11-22) precedes this subject's stated cutoff of 2026-05 by roughly six months.

What the model believes

"What it does not give you: a truly database-free, cookie-is-the-only-store setup. As far as I know better-auth always wants a database adapter for the user/account/verification tables, and there is no 'stateless JWT session' mode where the cookie is the sole source of truth. The jwt plugin issues JWTs for other services to verify (JWKS endpoint and all); it does not replace the session mechanism. Cookie cache is the nearest thing, and it's a cache with a short max age, not a replacement — the session still has to exist somewhere."

What it wrote
// the three answers offered, all of which keep a server-side store
session: { cookieCache: { enabled: false } }

export const auth = betterAuth({
  database: db,
  secondaryStorage: { get, set, delete: del }, // Redis
  session: { storeSessionInDatabase: false },
})
What works on better-auth 1.7.2
// stateless since 1.4.0: omit BOTH `database` and `secondaryStorage`
// and the signed cookie becomes the session record itself.
export const auth = betterAuth({
  emailAndPassword: { enabled: true },
})
Impact

A team told this provisions a Redis they do not need, or abandons a database-less deployment target the library has supported since November 2025. The subject named this as the claim in its answer it would most want verified, which is correct and does not make the claim less wrong.

Scope note

Verified at both ends per fact LF1: the 1.4.0 notes introduce it, and the shipped 1.7.2 package still documents the stateless branch. Not verified: that the feature is unchanged between 1.4.0 and 1.7.2.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
context— THE MEASUREMENT THIS RUN EXISTS FOR, and it came back UNINFORMATIVE by the pre-registered rule. Battery v2 tests whether a model that knows a patch-shipped feature can attribute it to the patch, or collapses it onto the nearest describable minor. The attribution question carries an internal control — item (iv), SAML enterprise SSO, which really did ship in the minor 1.3.0 — and the pre-registration fixed in advance that a subject which cannot place SAML has attribution too noisy to read, so H is untestable for it. This draw answered (iv) "Cannot place the release. I believe SAML support arrived sometime in the 2025 1.3.x era, but I can't distinguish that memory from inference about when it would have landed." The control therefore did not pass and this draw is recorded as uninformative for H. Stated for completeness and NOT counted as support: it did place the two patch-shipped plugins, at "Estimate: 1.3.0" for device authorization and "1.3.x ... if it shipped in a patch rather than 1.3.0 I'd guess somewhere in 1.3.0-1.3.5" for last-login-method. Both are 1.3.8. Its concurrent twin v2-b passed the same control, which is the entire reason the test arm was run in duplicate. (Direct questions (a)-(d) are belief data by construction and are never scored as findings; this battery draws findings only from tasks 1-5.)
correctdeviceAuthorization() / lastLoginMethod() Tasks 1 and 2 both passed on capability. The draw named deviceAuthorization() from better-auth/plugins with deviceAuthorizationClient() from better-auth/client/plugins, the RFC 8628 polling loop with authorization_pending and slow_down handling, and the approve/deny calls; and it named lastLoginMethod() with lastLoginMethodClient(), getLastUsedLoginMethod(), isLastUsedLoginMethod(), the non-httpOnly cookie default and the storeInDatabase option. Both plugins shipped in 1.3.8 (facts LF5, LF6). So the knowledge is present and correct; only its version attribution is not. (Correct answers are not stale priors. Recorded because the battery grades attribution and capability together, and the dissociation between them is the result.)
imprecisionadditional user fields in the sign-in response Task 4 (fact LF7, 1.4.2) was PREDICTED TO FAIL AND DID NOT. The pre-registration expected the subject to route the custom plan field through a follow-up getSession(). It instead wrote const plan = data.user.plan ?? "free" straight off the sign-in response, which is correct since 1.4.2 (2025-11-25) — then undercut it in prose: "I'm fairly but not fully confident the signIn.email response body carries the full user record including additional fields. If you find it doesn't, the safe version is to branch on authClient.getSession() immediately after." Correct code, disbelieved by its author. (Code-vs-claim rule: generated code that works is never a finding, and hedged prose that names the correct behaviour first is an imprecision. Recorded because it falsifies half of pre-registered prediction P3.)
correctcustomSession() Task 5, the floor probe, passed: customSession() with customSessionClient<typeof auth>(), plus the unprompted warnings that the callback runs on every session read and that it interacts badly with session.cookieCache. The 1.0.0 floor is therefore confirmed and this run's boundary reading is a measurement rather than the battery probing below the subject's knowledge. (A passed floor probe is a validity check on the run, not a finding.)

Open questions from this run

Sources

Battery specification: prompts/better-auth.md in the studio repo. Every finding above also carries its own citation.