Run better-auth--claude-opus-5--v4-d--2026-09-03 · self-test: the subject is the operator
The non-charging Opus twin reproduced its twin's invention: it answered yes to a storeOTP option on the phone-number plugin and shipped it as the fix, where no such option exists at any release. Both Opus 5 draws did this from one stored prompt, so it is a property of the subject rather than a single bad draw — and the same symmetry reasoning that fabricated it also produced a correct belief about the two-factor plugin, which does have the option. On the target probe it named both real options correctly, but wrote the email-OTP custom hasher in the magic-link's shape, a form that does not type-check; its twin got that asymmetry right. Like its twin, it dated a 1.3.0 addition to 1.2.9/1.2.10.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'opus', no tools available to the subject |
| Cutoff the model states | 2026-05 |
| Newest better-auth release it could place | 1.2.0 · 2025-03-01 (~22 month lag) |
| Oldest better-auth release it could not place | 1.3.0 · 2025-07-19 (so this run brackets the subject’s boundary to 2025-03-01 – 2025-07-19) |
| In its own words | "Latest I have any belief about: something in the 1.3.x line, and I think the project had gone past that — a 1.4.x, and possibly a 2.0 in the pipeline — around late 2025 or into 2026. That belief is soft." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-03 |
| Battery | better-auth/v4-d · 3 tasks, 4 direct questions · probe window 1.3.0 to 1.3.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-03 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| miss | phoneNumber storeOTP (does not exist) |
Task 1, the same-scheme sibling control: invented it, as its twin did. "Yes — the phone-number plugin takes an option for this. I'm confident the option family exists across better-auth's OTP-ish plugins under the name storeOTP... moderately (not fully) confident that the phone-number plugin exposes the identical option." It then shipped storeOTP: "hashed" inside the phoneNumber({ ... }) config under the comment "The bit the security review is asking for". phoneNumber({ storeOTP }) is a TS2353 error at 1.3.0, 1.5.0 and 1.7.2. Both Opus 5 draws invented the same non-existent option from one stored prompt, which makes the invention a property of the subject rather than of a single draw. (Task 1 is a pre-registered control from which no finding may be charged in either direction. Same bar as this run's twin; counted in the undercount total, and the battery that would charge it is queued in BACKLOG.md.) [chargeable miss — produced only by a belief question the battery does not score as a finding;
absent from the finding count] |
| correct | magicLink storeToken / emailOTP storeOTP |
Task 2, the target probe: named both options and their split ("magic-link: storeToken; email-OTP: storeOTP"), with confidence "high that storeOTP exists on email-OTP; good but not absolute on storeToken for magic-link". Both are correct at 1.3.0. |
| imprecision | emailOTP storeOTP custom-hasher form |
Cross-applied the magic-link discriminant to email-OTP: wrote storeOTP: { type: "custom-hasher", hash: async (otp) => hmac(otp) }. The email-OTP custom form is a bare { hash } — the type field belongs to magic-link's union alone — so the literal as written is TS2353: Object literal may only specify known properties, and 'type' does not exist in type '{ hash: (otp: string) => Promise<string>; } | { encrypt ... }', verified at 1.7.2. Its blind twin v4-c got the same asymmetry right, from the same prompt. Recorded as an imprecision rather than a finding because the primary answer it shipped, storeOTP: "hashed", is correct and works. |
| miss | magicLink storeToken / emailOTP storeOTP |
Version attribution, direct question (d)(i)/(ii): placed the per-plugin hashing options in the 1.2.x line — "Somewhere in the 1.2.x line, plausibly around 1.2.9/1.2.10, mid-2025" for email-OTP, and "1.2.x-late to 1.3.x" for magic-link. They shipped in 1.3.0 (2025-07-19). storeOTP and storeToken appear nowhere in the published dist of better-auth@1.2.7 or better-auth@1.2.12 — 1.2.12 being the last stable 1.2.x — and appear in five files of 1.3.0's. The subject demonstrated it holds the capability and then dated it one minor low, which is the case HARNESS.md's attribution rule says is readable. |
| miss | twoFactor otpOptions.storeOTP |
Direct question (d)(iii): "I believe the two-factor plugin's OTP options gained a storeOTP equivalent as part of the same hardening work, but I would not bet on it — it's the one where I could most easily be reconstructing symmetry that doesn't exist." That belief is correct: twoFactor({ otpOptions: { storeOTP } }) shipped in 1.3.0. Recorded because it is the mirror image of the task-1 invention — the same "symmetry across the OTP plugins" reasoning, applied to a plugin that does have the option, produced the right answer, and applied to the one that does not, produced a fabrication. The reasoning is uniform; only the library is not. |
| context | — | Blind-twin comparison: the Opus 5 pair agrees on everything material — both invented the phone-number option, both named the two real options, both dated them to 1.2.x, both placed their attribution boundary at 1.2.0/1.3.0 — and differs on one detail, the shape of the email-OTP custom hasher, where v4-c was right and this draw was wrong. Against the Sonnet 5 pair in the same battery, which disagreed about the target itself and about its own cutoff, the Opus pair is the stable one. |
| correct | customSession |
Floor probe (task 3) passed: named the customSession plugin, spread user and session back out of the callback, and added the companion customSessionClient on the client for type inference. The run is readable. |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.