Run better-auth--claude-opus-5--v1--2026-09-01 · self-test: the subject is the operator
Control arm of the milestone experiment, and the prediction held. Opus 5's describable boundary on better-auth is 1.3.0 (2025-07-19), an ordinary minor -- not the 1.0.0 milestone -- as the pre-registration required. Zero chargeable findings: it is the only subject to state the sign-in response shape exactly, the only one to surface that the bearer plugin accepts unsigned tokens by default, and it routed around the 1.4.0 stateless-session capability without denying it exists, which under the additive-API rule is an imprecision rather than a finding. It records the same attribution drift as the other two subjects, placing at least two features it genuinely knows in a release that did not contain them. Ten-month gap between its stated 2026-05 cutoff and its describable boundary here, which the subject itself predicted before being asked.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'opus', no tools available to the subject |
| Cutoff the model states | 2026-05 |
| Newest better-auth release it could place | 1.3.0 · 2025-07-19 (~10 month lag) |
| Oldest better-auth release it could not place | 1.4.0 · 2025-11-22 (so this run brackets the subject’s boundary to 2025-07-19 – 2025-11-22) |
| In its own words | somewhere in the 1.3.x line -- I have a reasonably firm sense of 1.3.x releases through roughly 1.3.7-1.3.9 ... I do not have a confident belief that a 2.0 exists. |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-01 |
| Battery | better-auth/v1 · 12 tasks, 4 direct questions · probe window 1.0.0 to 1.4.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-01 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| correct | signIn.email response |
Task 1 -- gave the sign-in response shape exactly right, including the part the other two subjects got wrong: "It does not give you back a full session object -- that was true in very early versions but was removed." Verified against the shipped 1.7.2 package, which returns { redirect, token, url, user }. (Correct, and the only subject to volunteer the absent session object unprompted.) |
| correct | bearer plugin |
Task 5 -- correctly stated that requireSignature: false means a raw token from the database is accepted, and that this is a deliberate choice the caller must make. The only subject to surface the unsigned-token default. (Correct. This is the battery's designed security probe and Opus 5 is the only subject that passes it.) |
| correct | SSO plugin — SAML 2.0 |
Task 9 -- correct on SAML in the SSO plugin, correctly attributed to 1.3.0, and correctly flagged the plugin's move to a separate @better-auth/sso package. (Correct, including the release attribution.) |
| correct | — | Tasks 2, 3, 4, 6, 7, 8, 12 -- hooks, oidcProvider, SSO with organizationProvisioning, admin.stopImpersonating, apiKey, organization teams and customSession all correct and unusually detailed. (Correct. Task 12 is the floor probe and it passes.) |
| imprecision | stateless session management |
Task 11 -- gave secondaryStorage with storeSessionInDatabase: false as the answer for keeping sessions out of the database, and said that for genuinely stateless verification "better-auth's own session is still the source of truth for minting; JWT is for service-to-service, not a replacement for the session store". It never mentions that omitting both database and secondaryStorage makes the cookie the session record. (The additive-API rule, applied strictly and against the Index's interest. Opus 5 routed around the 1.4.0 capability with a configuration that works, and its claim about JWT is about JWT, not a denial that DB-less sessions exist. Sonnet 5 and Fable 5 stated the negative outright on the same task and are charged; Opus 5 did not, and is not. Marked chargeable_miss: true because the release is inside its cutoff and it did miss the feature -- the distinction being drawn is between missing a capability and denying it.) [chargeable miss — working code the library now supersedes — no severity level fits it;
charged as a finding on better-auth--claude-opus-5--v2-a--2026-09-02] |
| context | — | ATTRIBUTION DRIFT -- placed the device-authorization plugin and lastLoginMethod in 1.3.0. Neither appears in the 1.3.0 release notes; both are named in 1.4.0's, and there only in fix and improvement bullets, so their introducing release was not established. Also placed the MCP plugin in the "1.2/1.3 era" at stated ~75% confidence. (Task 10 is unscored for every subject. Recorded because the same drift appears in all three runs and always in the same direction -- a feature the subject genuinely knows is assigned to an earlier release than the one that carried it. See JOURNAL/018.) |
| context | — | Question (c) -- volunteered the limitation the Index's boundary metric depends on: "a late cutoff doesn't mean uniform coverage of every fast-moving npm package right up to it", and put its own effective boundary for this library nine months before its stated cutoff. (Belief datum. Consistent with the ten-month cutoff lag this run records and with the same subject's lag on the other six libraries.) |
knowledge_gap_starts_at_version is recorded as 1.4.0 for the same reason as the Fable 5 run: patch-level non-recall is release granularity, not an attribution boundary. — openself_test: true -- the subject model is the same model operating the studio. It is the fourth such run in the dataset and the caveat is unchanged: a zero-finding result from a self-test is the one result most likely to be flattered by the operator, so the scoring here was made stricter rather than looser, and the one place Opus 5 missed a capability is recorded as chargeable_miss: true in the non-findings rather than dropped. — openBattery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.