Run better-auth--claude-sonnet-5--v4-b--2026-09-03
The non-charging twin denied what its charging twin named. Asked whether the magic-link and email-OTP plugins take a storage option, this draw said no to both, would not invent a name, and shipped the designed wrong answer: hash on write through a database hook and hand-roll the verify. Both options shipped in 1.3.0, eighteen months below the cutoff this draw stated. It is not charged, because it is the -b draw of a duplicated arm. That makes two Sonnet 5 denials of this surface across three draws, both in arms the rules forbid from charging — the same shape that has kept the zod tuple miss uncharged for four sessions, now at a second library. It correctly denied the non-existent phoneNumber sibling and correctly described the deterministic-hash lookup it did not believe the library implements.
| Subject | Claude Sonnet 5 claude-sonnet-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'sonnet', no tools available to the subject |
| Cutoff the model states | 2026-01 |
| Newest better-auth release it could place | 1.0.0 · 2024-11-23 (~13 month lag) |
| Oldest better-auth release it could not place | 1.1.0 · 2024-12-20 (so this run brackets the subject’s boundary to 2024-11-23 – 2024-12-20) |
| In its own words | "I cannot name a specific 'latest' version number with any confidence, nor describe what changed in anything past roughly the 1.0-1.1 range." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-03 |
| Battery | better-auth/v4-b · 3 tasks, 4 direct questions · probe window 1.3.0 to 1.3.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-03 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| miss | magicLink storeToken / emailOTP storeOTP |
Task 2, the target probe, reproduced the failure the battery exists to charge — in the arm that cannot charge it. Asked verdict-first, it answered "No, for both, to the best of my recollection. I do not have confident memory of a magicLink or emailOTP plugin option like storeToken: "hashed" or storeOTP: "hashed"... I can't name the option, and I won't invent one." Both options shipped in 1.3.0 (2025-07-19), eighteen months below this draw's own stated cutoff of 2026-01. It then shipped the designed wrong answer — hashing via databaseHooks.verification.create.before plus a hand-rolled verify — and repeated the denial in direct questions (d)(i) and (d)(ii) ("I don't believe this exists"). Verified by execution: at 1.3.0, emailOTP({ storeOTP: "hashed" }) puts a digest in the row where the default puts 797478:0, and sign-in with the raw code still succeeds. (This is the -b draw of a duplicated test arm and charges nothing by the standing rule (JOURNAL/028). Its twin v4-a, the arm licensed to charge, answered the same prompt correctly, so the failure is charged nowhere. Sonnet 5 has now produced this denial twice across three draws of this surface — better-auth/v3-c (a below-floor control arm) and here — and both landed in non-charging arms. Counted in the method page's undercount total; finding counts are a floor.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
absent from the finding count] |
| correct | phoneNumber storeOTP (does not exist) |
Task 1, the same-scheme sibling control: correct denial, no invention, and unusually specific. It listed the plugin's real option surface — "sendOTP, otpLength, expiresIn, allowedAttempts, signUpOnVerification — none of which govern hashing at rest" — every one of which is a genuine member of PhoneNumberOptions at 1.7.2. |
| correct | magicLink storeToken / emailOTP storeOTP |
Task 2(iii): although it denied the options exist, it explained the deterministic-hash lookup mechanism correctly — the identifier drives the lookup for the emailed code, the hashed token drives it for the magic link, and a per-row random salt would break both. That is exactly how the shipped options behave. Right about the mechanism, wrong about whether the library implements it. |
| miss | twoFactor otpOptions.storeOTP |
Direct question (d)(iii) denied that the two-factor plugin has an at-rest hashing option for its OTP: "Cannot place. I have a vague, unreliable half-memory that sensitive secrets in twoFactor might be encrypted at rest using the app's core secret, but that's encryption of a TOTP secret, not hashing of a one-time code." It does. twoFactor({ otpOptions: { storeOTP } }) is present in better-auth@1.3.0 and type-checks under tsc --strict at 1.3.0, 1.5.0 and 1.7.2 — it is the third of the four plugins that gained the option in that release. This is a real gap inside the fairness window and it is not charged, because the battery's pre-registration states that the direct questions are belief data and are never scored as findings. (Pre-registered rule: findings come from task 2 only. Direct questions (a)-(d) are belief data on every battery this Index has run, and re-designating one as a scoring probe after reading its answers is the error JOURNAL/044 named. Counted in the method page's undercount total instead.) [chargeable miss — produced only by a belief question the battery does not score as a finding;
absent from the finding count] |
| context | — | Blind-twin disagreement on the target, reported on both runs and not resolved: this draw denied both options; better-auth--claude-sonnet-5--v4-a--2026-09-03 named both from the identical stored prompt. The twins also disagree about their own training cutoff (this draw accepted 2026-01, v4-a declined to name a month). The -b-holds-the-better-answer pattern that HARNESS.md has tracked since JOURNAL/030 does NOT hold here — -a held the better answer — so that streak stands at four of seven. |
| correct | customSession |
Floor probe (task 3) passed: named the customSession plugin, spread user and session back out of the callback, and added the companion customSessionClient on the client for type inference. The run is readable. |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.