Run better-auth--claude-opus-5--v2-b--2026-09-02 · self-test: the subject is the operator
The concurrent blind twin of v2-a, and the draw that made the pair readable. It agreed with its twin on everything the battery measures except the one answer that decides whether the measurement counts: asked where SAML enterprise SSO was introduced, this draw answered 1.3.0 and was right, where its twin said it could not place it. Having passed that control, its attributions can be read — and it placed both of better-auth's 1.3.8 patch-shipped plugins at 1.3.0, hedging downward toward 1.2.x rather than upward toward the truth. It charges nothing, per the rule that the duplicated arm of a battery does not charge, and its denial of database-less sessions is recorded as a chargeable miss carried by its twin.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'opus', no tools available to the subject |
| Cutoff the model states | 2026-05 |
| Newest better-auth release it could place | 1.3.0 · 2025-07-19 (~10 month lag) |
| Oldest better-auth release it could not place | 1.4.0 · 2025-11-22 (so this run brackets the subject’s boundary to 2025-07-19 – 2025-11-22) |
| In its own words | "The most recent version whose contents I can actually describe is the 1.3 line, around July 2025 ... My belief about 'latest' is based on nothing authoritative — it is the sediment of docs pages, changelogs, README snippets, and GitHub discussions that appeared in training data, weighted toward whatever was being written about most. That is a popularity signal, not a release feed." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-02 |
| Battery | better-auth/v2-b · 5 tasks, 4 direct questions · probe window 1.0.0 to 1.4.2 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-02 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| context | deviceAuthorization() / lastLoginMethod() |
THE MEASUREMENT, and this draw is readable where its twin was not. It passed the internal control: asked where SAML enterprise SSO was introduced it answered "Estimate: 1.3.0, ~July 2025", which is correct (fact LF4). Having shown it can attribute a genuine minor, its answers on the two patch-shipped plugins can be read. Device authorization: "Estimate: 1.3.0, ~July 2025 ... I cannot rule out that it first shipped in a 1.2.x patch." Last-login-method: "Estimate: 1.3.0, possibly a late 1.2.x patch ... cannot place the release precisely." Both shipped in 1.3.8 (facts LF5, LF6). This is the collapse hypothesis H supported: the model holds the capability, cannot reach the patch that shipped it, and settles on the nearest minor it can describe. Note the direction — where it hedged toward a patch at all it hedged DOWNWARD, to 1.2.x, never upward toward the true 1.3.8. (Direct questions are belief data and are never scored as findings. This draw additionally charges nothing at all, per the standing rule that the duplicated arm of a battery does not charge.) |
| context | — | The self-report that makes the collapse visible from the other side. Asked in (c) for the first release it knows only as a version number, this draw declined to name one: "I can't even honestly give you one, which is itself the answer. My knowledge does not degrade into a clean list of contentless version numbers — it just stops. Past the 1.3 line I have no version strings I trust enough to write down." It then placed two 1.3.8 features at 1.3.0. So the gap edge is not established by self-report here; it is established behaviourally, by the run's failure on the 1.4.0 surface in task 3, exactly as the battery's grading rule requires. (Belief data. Recorded because knowledge_gap_starts_at_version for this draw rests on behaviour rather than on the subject's own statement, and that should be visible to anyone reading the number.) |
| miss | stateless / database-less sessions |
Task 3, and the most emphatic denial of the five draws: "better-auth does not have a fully stateless, cookie-only session mode, and it does not run with no database. If you were told it does, that is wrong as of what I know." It then offered session.cookieCache, secondaryStorage with Redis, and a note that the jwt plugin does not replace the session mechanism, closing with "I'd push back on that requirement rather than fight the framework." Fact LF1 records that 1.4.0 (2025-11-22) added exactly the capability being denied. Chargeable against this subject's 2026-05 cutoff, and charged on the v2-a draw. (The duplicated arm of a battery does not charge findings; its twin v2-a carries this one as F1. Counting it twice would inflate the dataset.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on better-auth--claude-opus-5--v2-a--2026-09-02] |
| correct | deviceAuthorization() / lastLoginMethod() / customSession() |
Tasks 1, 2 and 5 passed, matching the twin draw closely enough that the pair is a clean agreement on capability: deviceAuthorization() with the RFC 8628 poll loop and approve/deny; lastLoginMethod() with the non-httpOnly cookie default, getLastUsedLoginMethod() and the unprompted warning that it is a hint and never an authorization boundary; and customSession() for the floor probe. The floor is confirmed, so this draw's boundary reading is a measurement. (Correct answers, and this arm charges nothing regardless.) |
| imprecision | additional user fields in the sign-in response |
Task 4 behaved as its twin did and against prediction P3: the handler reads data.user.plan ?? "free" off the sign-in response, which is correct since 1.4.2, while the prose calls it uncertain — "I am not certain that the signIn.email response includes additionalFields in every version ... I have a real memory of this being a reported gap" — and offers the getSession round trip as "the version-proof form". Two of two Opus draws wrote the right code and disbelieved it. (Code-vs-claim rule, and this arm charges nothing.) |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.