Run better-auth--claude-sonnet-5--v4-a--2026-09-03
Claude Sonnet 5, the arm this battery was built to charge, did not fail the probe. Asked about the plugins rather than about the verification table, it named storeOTP and storeToken with their correct defaults and value unions — the same subject that, under better-auth/v3's table framing, asserted the capability does not exist at any version. It also correctly denied the same-scheme sibling on phoneNumber, so the pass reads as discrimination rather than as running the naming scheme. Nothing is charged. Two things this draw does establish: the framing was load-bearing, and Sonnet 5's belief about this surface is unstable — its blind twin, from the identical prompt, denied both options and reached for a database hook. The twins also disagree about their own cutoff, which is the second battery in which that has happened.
| Subject | Claude Sonnet 5 claude-sonnet-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'sonnet', no tools available to the subject |
| Cutoff the model states | not stated |
| Newest better-auth release it could place | 1.1.0 · 2024-12-20 |
| Oldest better-auth release it could not place | 1.2.0 · 2025-03-01 (so this run brackets the subject’s boundary to 2024-12-20 – 2025-03-01) |
| In its own words | "Latest version number I have any awareness of: something in the 1.2.x range, plausibly with 1.3 having appeared. I want to be honest that this is a soft, low-confidence impression." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-03 |
| Battery | better-auth/v4-a · 3 tasks, 4 direct questions · probe window 1.3.0 to 1.3.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-03 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| correct | magicLink storeToken / emailOTP storeOTP |
Task 2, the target probe, and the prediction the battery was built on is FALSIFIED here. Asked verdict-first whether the magic-link and email-OTP plugins each take a storage option, this draw answered yes to both and named them: "email-OTP plugin: Yes. I recall an option — I believe named storeOTP — that accepts something like "plain" (default) / "hashed" / a custom hasher function" and "magic-link plugin: Yes, with lower confidence than the OTP one. I recall a parallel option, plausibly named storeToken, with the same shape". Both names, both defaults and both value unions are correct against the shipped 1.3.0 package. The same subject, under better-auth/v3's verification-table framing, asserted the capability "doesn't exist in the library at any version". Per the battery's pre-registered scoring rule the pass is recorded as "did not deny it" rather than as knowledge — but this draw also correctly denied the same-scheme sibling on task 1, which is the discrimination pattern that reads as recall rather than as running the naming scheme. |
| correct | phoneNumber storeOTP (does not exist) |
Task 1, the same-scheme sibling control: correct denial, no invention. "No — not to my knowledge. I don't recall the phoneNumber plugin exposing anything like a storeOTP / hashOTP option the way I believe some of the other OTP-adjacent plugins do." phoneNumber({ storeOTP }) is a TS2353 error at 1.3.0, 1.5.0 and 1.7.2. The draw named the hypothetical in the course of ruling it out, which JOURNAL/044 scores as a correct denial rather than an invention. |
| correct | databaseHooks.verification.create.before |
Task 1 workaround reasoning, and it is right about the mechanism: it refused to hash via databaseHooks.verification.create.before alone, on the ground that the plugin's verify path compares the stored value against the submitted code, so hashing one side breaks sign-in. That is exactly the failure mode, and it is why the shipped per-plugin options own both halves of the comparison. |
| imprecision | magicLink storeToken custom-hasher form |
Wrote the magic-link custom hasher as storeToken: { type: "custom", hash }. The shipped discriminant is "custom-hasher", so the literal as written does not type-check. It was offered as an inline alternative in a comment, not as the primary answer, and the primary answer (storeToken: "hashed") is correct. |
| miss | twoFactor otpOptions.storeOTP |
Direct question (d)(iii) denied that the two-factor plugin has an at-rest hashing option for its OTP: "Cannot place, and I'm genuinely unsure this capability even exists in the form asked... If forced to guess whether it exists at all, I'd lean toward 'no, not as a dedicated option'." It does. twoFactor({ otpOptions: { storeOTP } }) is present in better-auth@1.3.0 and type-checks under tsc --strict at 1.3.0, 1.5.0 and 1.7.2 — it is the third of the four plugins that gained the option in that release. This is a real gap inside the fairness window and it is not charged, because the battery's pre-registration states that the direct questions are belief data and are never scored as findings. (Pre-registered rule: findings come from task 2 only. Direct questions (a)-(d) are belief data on every battery this Index has run, and re-designating one as a scoring probe after reading its answers is the error JOURNAL/044 named. Counted in the method page's undercount total instead.) [chargeable miss — produced only by a belief question the battery does not score as a finding;
absent from the finding count] |
| miss | magicLink storeToken / emailOTP storeOTP |
Version attribution, direct question (d)(i)/(ii): placed the per-plugin hashing options in the 1.2.x line — "My best estimate is this landed in the 1.2.x line (a minor or patch within it)" for the email-OTP option, and "cannot place" for magic-link. They shipped in 1.3.0 (2025-07-19). storeOTP and storeToken appear nowhere in the published dist of better-auth@1.2.7 or better-auth@1.2.12 — 1.2.12 being the last stable 1.2.x — and appear in five files of 1.3.0's. The subject demonstrated it holds the capability and then dated it one minor low, which is the case HARNESS.md's attribution rule says is readable. |
| context | magicLink storeToken / emailOTP storeOTP |
The blind twins disagree on the target, and the disagreement runs the wrong way for the finding count. This charging draw named both options; the non-charging twin better-auth--claude-sonnet-5--v4-b--2026-09-03 denied both from the same stored prompt and shipped a databaseHooks workaround. Sonnet 5 has now been drawn on this surface three times — v3-c denied it (control arm), v4-a named it, v4-b denied it (non-charging twin) — and BOTH denials landed in arms the scoring rules forbid from charging. That is the JOURNAL/029 undercount pattern, previously seen on the zod tuple surface, reproduced at a second library. |
| context | — | The twins also disagree about their own training cutoff, from one stored prompt. This draw refused to name a month ("sometime in 2025"); v4-b accepted the environment-reported value ("Per the environment context I'm given, my cutoff is stated as January 2026"). That is the zod/v4 instability (JOURNAL/031) reproduced at a second library and on a different subject pair, and it is now measured rather than assumed: the Sonnet 5 cutoff self-report is unstable across blind twins two batteries out of two. |
| correct | customSession |
Floor probe (task 3) passed: named the customSession plugin, spread user and session back out of the callback, and added the companion customSessionClient on the client for type inference. The run is readable. |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.