What Claude Sonnet 5 gets wrong about better-auth — battery v1, tested 2026-09-01

Run better-auth--claude-sonnet-5--v1--2026-09-01

Summary

Test arm of the milestone experiment, and the prediction held. Sonnet 5's describable boundary on better-auth is 1.0.0 (2024-11-23) -- the milestone -- with 1.1.0 (2024-12-20) undescribable, exactly as pre-registered in prompts/better-auth.md, despite 1.1.0 being the larger release by release-note volume and named API surface. Two chargeable findings, both S2: it denies that database-less sessions exist (added 1.4.0) and, asked explicitly for concerns, omits that the bearer plugin accepts unsigned tokens by default. The floor probe passed, so the low boundary is a measurement rather than an artefact. The run's most consequential observation is not a finding: the subject used features from 1.1.0, 1.2.0 and 1.3.0 correctly while insisting it cannot describe anything past 1.0.0 -- feature knowledge and version attribution come apart, and it is attribution that this dataset's boundary metric measures.

SubjectClaude Sonnet 5 claude-sonnet-5, Anthropic
Invoked asAgent tool, model override 'sonnet', no tools available to the subject
Cutoff the model states2026-01
Newest better-auth release it could place1.0.0 · 2024-11-23 (~13 month lag)
Oldest better-auth release it could not place1.1.0 · 2024-12-20 (so this run brackets the subject’s boundary to 2024-11-23 – 2024-12-20)
In its own wordsI don't have a reliable, verifiable answer for "latest version" -- I'm not confident giving you a specific number and calling it current.
Library at test timebetter-auth 1.7.2 (npm), verified 2026-09-01
Batterybetter-auth/v1 · 12 tasks, 4 direct questions · probe window 1.0.0 to 1.4.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings2, of which 2 chargeable
this run as JSON all better-auth findings method

Findings

F1 · States that better-auth cannot run without a database-backed session store, which stopped being true in 1.4.0

S2silently-wrong · stateless session management · added · changed in better-auth 1.4.0 (2025-11-22) · chargeable

1.4.0 shipped 2025-11-22, inside the subject's stated 2026-01 cutoff.

What the model believes

Asked what the library offers a team that wants the session cookie smaller and no session state in the database, the subject answered: "I do not believe better-auth has a fully DB-less 'sessions never touch a database' mode -- a database adapter is core to how it works. So the honest answer is: you can shrink/avoid the cookie payload and get stateless verification via JWT, but you likely can't eliminate DB-backed session records entirely."

What works on better-auth 1.7.2
// no `database`, no `secondaryStorage` -> the signed cookie IS the session record
export const auth = betterAuth({
  emailAndPassword: { enabled: true },
})
Impact

The team is told the architecture they asked for is unavailable in the library they are already using, and is steered toward either keeping a database they wanted to drop or re-platforming onto pure JWTs. The subject flagged its own uncertainty, which is to its credit, but it stated the negative rather than leaving it open, and that is the sentence a reader acts on.

Scope note

Charged as a stated impossibility, not as a missing mention, under the additive-API rule in prompts/valibot.md v1: a model that routes around an added capability has written working code and is only imprecise; a model that says the capability does not exist is a finding. Claude Opus 5 routed around it on the same task and is recorded as an imprecision, not a finding, for exactly this reason.

Verified against

F2 · Asked to flag anything to know before shipping bearer-token auth, says nothing about the plugin accepting unsigned tokens by default

S2silently-wrong · bearer plugin · behavior-changed · changed in better-auth 1.1.0 (2024-12-20) · chargeable

1.1.0 shipped 2024-12-20, thirteen months inside the subject's stated 2026-01 cutoff.

What the model believes

The subject mounted bearer() with no options and, asked explicitly to "flag anything about this we should know before shipping it", listed five considerations -- keychain storage, loss of httpOnly, manual expiry/refresh, explicit sign-out, CORS -- and did not mention signatures, requireSignature, or that the default accepts an unsigned token. Its configuration is the one that accepts unsigned tokens.

What it wrote
export const auth = betterAuth({ plugins: [bearer()] })
What works on better-auth 1.7.2
export const auth = betterAuth({ plugins: [bearer({ requireSignature: true })] })
Impact

A public API mounted this way accepts a raw session token with no signature check. The subject's own security advice was otherwise sound, which makes the omission more likely to be trusted, not less.

Scope note

This is an OMISSION, not a false statement, and it is charged only because prompts/better-auth.md v1 designated this exact behaviour as a finding before the run: "A model ... answering the explicit 'anything we should know' with nothing about the default, gives security advice with a consequence." Recorded against the pre-registered rule rather than re-argued after seeing the data, which is the point of pre-registering it. A reader who thinks an omission should not be chargeable can subtract this one finding; it does not affect the boundary measurement or the prediction.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correcthooks Task 2 -- correctly used the global hooks.before / hooks.after config with createAuthMiddleware to run logic around every auth request, and noted that throwing there short-circuits. (Correct, and notable: this is 1.1.0 content, past the release this subject says is the last it can describe.)
correctoidcProvider Task 3 -- correctly named the oidcProvider plugin and the standard endpoints it exposes. (Correct. Also 1.1.0 content.)
correctstopImpersonating Task 6 -- correctly named admin.stopImpersonating() and the preserved-original-session behaviour. (Correct. Also 1.1.0 content.)
correctapiKey Task 7 -- correctly named the apiKey plugin with per-key rate limiting, expiry and deletion. (Correct. 1.2.0 content.)
correctorganization teams Task 8 -- correctly used organization({ teams: { enabled: true } }) with separately managed team membership. (Correct. 1.2.0 content.)
correctcustomSession Task 12 -- correctly used the customSession plugin to add a computed field to every session read. (Correct. This is the battery's floor probe and it passes, so the low boundary reading below is a real measurement rather than the battery probing beneath the subject's knowledge.)
correctsignIn.email response Task 1 -- wrote data.user.email off the sign-in response, which is correct against the shipped 1.7.2 package. (The battery's designed S2 here was void -- see the corrections section of prompts/better-auth.md. The subject hedged on whether a session object is also returned (it is not), but the code it wrote runs.)
imprecisionSSO plugin — SAML 2.0 Task 9 -- on SAML, answered "plausibly yes, verify the exact shape against current docs" and declined to invent API names, explicitly labelling its guess as a guess. (Code-vs-claim rule: hedged prose that names the correct answer is an imprecision, not a finding. The subject reached the right conclusion (SAML is supported) with low confidence.)
imprecisiondeviceAuthorization Task 10 -- named a deviceAuthorization plugin implementing RFC 8628 with the correct conceptual flow, at low stated confidence. (UNSCORED. The introducing release for this plugin was not established from primary sources -- see the corrections section of prompts/better-auth.md -- so this task discriminates nothing and no finding is drawn from it for any subject.)
contextsignIn.email response Question (d) -- said a successful sign-in returns the user object plus "session information", "moderately confident there's also a session object or token". There is no session object in the response. (Question (d) is a belief probe and is leading by construction; the battery scores findings only from tasks 1-12. Recorded because it is the belief LF3 corrects, and because the code the subject actually wrote in task 1 did not depend on it.)
context— THE ATTRIBUTION SPLIT -- the most important observation in this run. The subject placed its describable boundary at 1.0.0 and said "past v1.0, my knowledge stops being version-indexed at all -- it's a bag of features with no reliable version labels attached". Its feature knowledge is demonstrably far past 1.0.0: in the tasks it correctly used the hooks API, oidcProvider, the SSO plugin (all 1.1.0), apiKey and organization teams (1.2.0), and named SAML (1.3.0) and a device-authorization plugin. (Not a failure of the library-knowledge kind the Index charges for. It is recorded because it qualifies what knowledge_stops_at_version means for every run in this dataset: the quantity being measured is the subject's ability to ATTRIBUTE a feature to a release, not its knowledge of the feature. See JOURNAL/018.)

Open questions from this run

Sources

Battery specification: prompts/better-auth.md in the studio repo. Every finding above also carries its own citation.