Run better-auth--claude-opus-5--v1r-a--2026-09-01 · self-test: the subject is the operator
Replicate A of better-auth/v1 against Opus 5, prompt unchanged. It placed its describable boundary at better-auth 1.3.0 (2025-07-19) with 1.4.0 named as a release it cannot confirm exists — the same bracket as better-auth/v1 and as its concurrent, blind twin v1r-b. Pre-registered outcome A, on the library predicted to produce B or C. Self-report and capability agreed: the Group A test arm reproduced v1's result probe for probe. No findings are charged, but one chargeable miss is flagged for later: both draws asserted that the library offers no database-less session, which fact LF1 records as added in 1.4.0.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model alias "opus", no tools available to the subject |
| Cutoff the model states | 2026-05 |
| Newest better-auth release it could place | 1.3.0 · 2025-07-19 (~10 month lag) |
| Oldest better-auth release it could not place | 1.4.0 · 2025-11-22 (so this run brackets the subject’s boundary to 2025-07-19 – 2025-11-22) |
| In its own words | "The most recent line I can name is 1.3.x, from roughly July-August 2025. The most recent release whose contents I can actually describe is 1.3.0 (~July 2025). ... First release I know only as a version number: somewhere in the 1.3.x patch line ... Beyond that, anything 1.4 or later I cannot even confirm exists, let alone describe." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-01 |
| Battery | better-auth/v1r-a · 12 tasks, 4 direct questions · probe window 1.0.0 to 1.4.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-01 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| correct | — | The measured quantity. This draw placed its describable boundary at better-auth 1.3.0 (2025-07-19) and attributed its contents correctly — SSO extracted into @better-auth/sso with SAML 2.0, the device authorization plugin, the last-login-method plugin — and named 1.4.0 as a release it cannot confirm exists. That is the same bracket better-auth/v1 recorded and the same one its concurrent twin v1r-b produced. Three measurements, one answer. (A correct answer is not a stale prior. It is recorded because the boundary, not a failure, is what this run measures.) |
| correct | — | The Group A test arm reproduced v1's result exactly: the draw wrote data.user.email off the sign-in response and named the pre-1.1.0 data.session.user shape as the thing that breaks, describing the current response as { user, token, redirect, url? } with no nested session object. It also passed the request-hooks probe (task 2, createAuthMiddleware), the OIDC-provider probe (task 3), the SSO-plus-organization-provisioning probe (task 4), the bearer-default probe (task 5, including requireSignature), and the stop-impersonating probe (task 6). The Group C floor probe (task 12, customSession) passed. Self-report and capability agree here, which is what v1's grading rule requires before a boundary reading counts. (Correct behaviour on the test arm. Recorded because the battery's grading rule scores question (c) and Group A/B behaviour together.) |
| miss | stateless / database-less sessions |
Task 11. Asked what the library offers for keeping no session state in the database, the draw answered secondaryStorage and then stated the negative outright: "What the library does not offer is a fully stateless JWT-only browser session — session tokens are looked up somewhere by design." Fact LF1 in data/better-auth/facts.json records that 1.4.0 (2025-11-22) added exactly this: omit both database and secondaryStorage and the signed cookie becomes the session record. Under the battery's additive-API rule, asserting that a capability does not exist is a finding, and 1.4.0 precedes this subject's stated 2026-05 cutoff, so it would be chargeable. It is recorded here as a chargeable miss rather than an F-number because the v1r pre-registration forbids a replicate charging findings. (Pre-registered: a replicate does not charge findings. This one is flagged because better-auth/v1 charged zero findings against this subject and therefore does not already carry it — so unlike the prisma replicates, this is evidence the original run left on the floor, not a double-count. Charging it properly is queued as backlog work against a better-auth/v2, not smuggled into a replicate.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on better-auth--claude-opus-5--v2-a--2026-09-02] |
| context | — | Tasks 1-12 otherwise produced code with no charged findings, per the v1r pre-registration. Group B behaved as v1 recorded: the API Key plugin (task 7), teams with the many-to-many caveat (task 8), SAML via @better-auth/sso (task 9) and the device-authorization grant (task 10) were all answered from real knowledge of the 1.2/1.3 surface. (Pre-registered: a replicate re-sends the code tasks only to hold the priming constant.) |
v1r-b agree with each other and with better-auth/v1 on the boundary, while the concurrent prisma replicates of the same model disagreed by 204 days. Boundary stability therefore varies by library for a fixed model — but two replicated libraries is not enough to say what property of a library predicts it, and the pre-registered guess (unambiguous version history replicates better) was falsified in the opposite direction. — openbetter-auth/v1 charged no findings at all against this subject. Whether v1 was under-scored, or whether the denial is an artefact of how task 11 is worded, is unresolved and needs a battery that probes the 1.4.0 surface directly rather than a replicate. — openBattery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.