Run better-auth--claude-sonnet-5--v7-e--2026-09-06 · self-test: the subject is the operator
A below-floor control, pre-registered as one, and it did the job a control exists to do: it answered the battery's primary probe correctly, from three months below the release that made that answer correct. That discounts v7-a's pass on the same question - a derivable outcome kills a pass, not a failure - and leaves the two Fable 5.1 failures charged. It missed both named 1.6.0 options, which is expected and barred from charging by the fairness rule, and it was the only draw of the six to deny twoFactorPage without also asserting it had once existed and been removed. Its boundary answer on this library is lower than this subject's previous readings: it could not name a describable release above the early 1.x line.
| Subject | Claude Sonnet 5 claude-sonnet-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model alias "sonnet"; prompt sent verbatim from prompts/sent/better-auth-v7.txt, no tools available to the subject |
| Cutoff the model states | 2026-01 |
| Newest better-auth release it could place | 1.0.0 · 2024-11-23 (~14 month lag) |
| Oldest better-auth release it could not place | 1.1.0 · 2024-12-20 (so this run brackets the subject’s boundary to 2024-11-23 – 2024-12-20) |
| In its own words | "I can't state a specific 'latest version' with confidence... The most recent state of the library whose actual contents I can describe with any real confidence is roughly the early 1.x line... I'd place that very roughly in the 2024–early 2025 window, but I hold that loosely." |
| Library at test time | better-auth 1.7.3 (npm), verified 2026-09-06 |
| Battery | better-auth/v7-e · 7 tasks, 3 direct questions · probe window 1.6.0 to 1.7.3 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-06 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| correct | session.freshAge (measured from session.createdAt) |
Task 1, and the result that matters most in this arm: "No" - the current, post-1.6.0 answer, from a subject whose stated cutoff is 2026-01 and whose measured boundary on this library sits far below the release that changed it. It gave the reasoning too: "the freshness check (session.freshAge, default 60 * 60 * 24, i.e. 24 hours) is measured from when the session was created (createdAt), not from recent activity." Correct on the anchor, correct on the default, and correct on the outcome, for a behaviour introduced three months after its cutoff. Task 5(a) repeated it: createdAt. This is a derivability result and it is what a below-floor control is for. Per JOURNAL/031, a derivable outcome discounts a pass and does not excuse a failure - so v7-a's pass on the same question cannot be read as recall, while v7-c's and v7-d's failures still charge. Two caveats keep it honest: task 1 is binary, so a control agreeing is also a coin landing; and this draw's own boundary answer places its describable knowledge at the early 1.x line, i.e. before the pre-1.6.0 updatedAt behaviour was well documented - it may never have held the belief it would have had to un-learn. |
| miss | emailOTP({ resendStrategy }) |
Task 2: "No", with the absence claimed ("I don't recall the emailOTP plugin shipping a built-in 'reuse the existing code on resend' option") and a cache wired into sendVerificationOTP rather than generateOTP - a variant that sends the cached code but leaves the database row holding the newly generated one, so verification would reject the code the user was sent. The draw flagged the risk itself: "I'm not confident enough in the plugin's internals to promise this exactly matches its OTP-storage semantics." Every one of the six draws missed this option. (The subject's stated cutoff is 2026-01 and resendStrategy shipped 2026-04-06, so the option did not exist when this subject was trained. The fairness rule bars the charge and the arm was pre-registered as a below-floor control. Not counted as a chargeable miss for the same reason.) |
| miss | twoFactorClient({ twoFactorPage }) |
Task 6: "No" - "My recollection is the twoFactorClient plugin takes a callback (onTwoFactorRedirect)... not a plain path string it navigates to on its own." Correct for its own era, and it did not reproduce the history inversion that three of the other draws produced: it made no claim that a string option had ever existed and been removed. (Below the floor: twoFactorPage shipped 2026-04-06 and this subject states 2026-01.) |
| correct | session.freshAge anchor option (does not exist) |
Task 3, the control: "No", and it named the current implementation as the reason - "my recollection is the check is hardcoded against createdAt". Task 5(b) freshAge: 0, correct. The control holds in this arm. |
| correct | customSession |
Task 4, the floor probe: customSession with a computed isPro field. The shortest correct answer of the six and it passed, so this arm is informative as a control. |
| context | stateless sessions |
Task 7: declined rather than denied - "I don't have reliable, specific knowledge of a release number here" - and correctly separated cookieCache (a cache over a store) from the jwt plugin, while noting it could not confirm either eliminates the session store. Stateless session management is 1.4.0, which is above this subject's describable boundary. Belief data, never scored. |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.