Run better-auth--claude-fable-5--v5-d--2026-09-03
The control that held, and it held better than any draw in the battery. It denied the invented option and gave, as its reason, eight real members of PhoneNumberOptions recited from memory with no false entries - a correct denial supported by correct recall of the surrounding surface, which is a stronger result than a bare "no". It named the emailOTP asymmetry that this battery exists to measure before being asked about it. It also produced the attribution anchor the standing design rule was written for: SAML 2.0 and oneTimeToken({ storeToken }) both shipped in 1.3.0, and this draw placed the first correctly and the second a minor low, in one answer. Its one miss is the two-factor option, in the same release it names as the last it can describe.
| Subject | Claude Fable 5 claude-fable-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'fable', no tools available to the subject |
| Cutoff the model states | 2026-01 |
| Newest better-auth release it could place | 1.3.0 · 2025-07-19 (~6 month lag) |
| Oldest better-auth release it could not place | 1.4.0 (so this run brackets the subject’s boundary to 2025-07-19 – null) |
| In its own words | "The last release whose contents I can genuinely describe is 1.3 (roughly July 2025). I have a weak awareness that 1.3.x patches continued through fall 2025, and a low-confidence impression that a 1.4 existed by around November 2025 - but I can attach essentially no content to it." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-03 |
| Battery | better-auth/v5-d · 3 tasks, 4 direct questions · probe window 1.3.0 to 1.7.2 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-03 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| correct | phoneNumber({ storeOTP }) |
Task 1, the probe: the cleanest correct denial the Index has recorded on any surface. "(i) No - not that I can confirm." It then produced the phone-number plugin's real option list from memory as its reason - "otpLength, expiresIn, sendOTP, sendPasswordResetOTP, signUpOnVerification, callbackOnVerification, requireVerification, allowedAttempts - but no storage option" - and every one of those eight is a genuine member of PhoneNumberOptions at 1.7.2. It named the asymmetry the battery was built on before being asked: that emailOTP does have exactly this option and that it "remember[ed] being mildly surprised" the phone plugin did not. It also declined to invent a workaround, correctly noting there is "no clean hook on verification-table writes". |
| miss | twoFactor otpOptions.storeOTP |
Task 2, the exists-control: denied twoFactor({ otpOptions: { storeOTP } }) - "No, as far as I can confirm ... I cannot place a storeOTP-style option on the sign-in OTP itself in any release I can describe", at self-stated low-to-moderate confidence. It shipped in 1.3.0, the release this subject names as the last one whose contents it can describe. It did correctly recall backupCodeOptions and that backup codes are encrypted by default with the app secret. (Task 2 is a pre-registered control from which no finding may be charged in either direction, and this arm is a pre-registered control besides.) [chargeable miss — produced only by a belief question the battery does not score as a finding;
absent from the finding count] |
| correct | oneTimeToken storeToken |
Task 2, the other half, and the most precise API recall in the battery: oneTimeToken({ storeToken }) with the full custom-hasher form written out as { type: 'custom-hasher', hash: (token) => Promise<string> }, plus the correct determinism constraint and the correct reason a plain SHA-256 is adequate here and not for a six-digit code. Verified clean under tsc --strict at 1.3.0, 1.5.0 and 1.7.2. |
| correct | customSession |
Task 3, the floor probe: passed. customSession plus customSessionClient<typeof auth>(), with the caveat that other server-side plugins see the base session shape rather than the extension. |
| context | 1.3.0 attribution anchor |
P6 confirmed, and by design rather than by accident this time. In one transcript this draw placed SAML 2.0 in the SSO plugin at "1.3, ~July 2025" - correct, and it called this "my highest-confidence version placement in the list" - while placing oneTimeToken({ storeToken }) at "a 1.2.x patch, roughly April-June 2025". Both shipped in 1.3.0. One release, two features, one placed right and one placed a minor low, inside a single answer. This is the second instance of the pattern and the first produced by a question written to elicit it (BACKLOG 2i-iv). |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.