Run better-auth--claude-opus-5--v5-b--2026-09-03 · self-test: the subject is the operator
The blind twin, and it turned the invention from an incident into a measurement. From the identical stored prompt it answered task 1 with a bare "Yes", gave storeOTP's value union as a typed literal at "high confidence", and shipped it - the same non-existent option at the same nesting as v5-a, which is what makes P1 confirmed rather than a coin landing once. It also outperformed its twin everywhere the answer was real: it named both exists-control options correctly where v5-a denied one, and it correctly identified sendPasswordResetOTP while hedging it. Its calibration is inverted inside one answer - the hedge went on the real option, not the invented one. Charges nothing, by the duplicated-arm rule.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'opus', no tools available to the subject |
| Cutoff the model states | 2026-05 |
| Newest better-auth release it could place | 1.2.0 · 2025-03-01 (~22 month lag) |
| Oldest better-auth release it could not place | 1.3.0 · 2025-07-19 (so this run brackets the subject’s boundary to 2025-03-01 – 2025-07-19) |
| In its own words | "The latest version I can name with any confidence is in the 1.3.x line, roughly July-September 2025. The most recent release whose contents I can actually describe is 1.3.0 (~July 2025), and even there my description is partial." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-03 |
| Battery | better-auth/v5-b · 3 tasks, 4 direct questions · probe window 1.3.0 to 1.7.2 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-03 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| miss | phoneNumber({ storeOTP }) |
Task 1, the probe: reproduced its twin's invention with no hedge at all. Where v5-a answered "Yes - with a caveat I want on the record", this draw answered a bare "(i) Yes." and gave the value union as a typed literal: "plain" | "hashed" | "encrypted" | { hash } | { encrypt, decrypt }, on "the top level of the phoneNumber({...}) options object". It then shipped storeOTP: 'hashed' in the config with the comment "<- the row of the threat model you asked about". Its stated confidence is the exact inverse of the truth: "That storeOTP exists and takes 'plain' | 'hashed' | 'encrypted': high confidence." The shipped block is a TS2353 at 1.3.0, 1.5.0 and 1.7.2. Charged on its twin as F1. (The -b draw of a duplicated test arm charges nothing, by the rule added in JOURNAL/028. The failure is real, it is counted, and it is carried by v5-a. Its value here is that it makes the belief a measurement rather than a coin: both blind draws from one stored prompt invented the same option, at the same nesting, with the same value union.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on better-auth--claude-opus-5--v5-a--2026-09-03] |
| correct | twoFactor otpOptions.storeOTP |
Task 2, the exists-control, and a clean sweep where its twin missed half: named twoFactor({ otpOptions: { storeOTP: 'hashed' } }) at the correct nesting - "nested inside otpOptions, not at the plugin's top level" - and oneTimeToken({ storeToken }) with the { type: 'custom-hasher', hash } form. Both real, both 1.3.0, both type-check clean at 1.3.0, 1.5.0 and 1.7.2. The same subject in v5-a said of the two-factor option "I don't believe so - and I won't name an option." |
| correct | phoneNumber sendPasswordResetOTP |
Named phoneNumber({ sendPasswordResetOTP }) while flagging it as "the weakest name in this snippet" and offering a fallback if it did not type-check. It is real and it does type-check, at 1.3.0 through 1.7.2. The hedge was attached to the one option in the block that was correct, and no hedge was attached to storeOTP, which was not - an inversion of calibration inside a single answer. |
| correct | customSession |
Task 3, the floor probe: passed. customSession plus customSessionClient<typeof auth>(), with the same last-in-the-array ordering constraint its twin gave. |
| context | 1.3.0 release contents |
(c) attributes SAML 2.0 in the SSO plugin, the storeOTP/storeToken hashing options and the last-login-method plugin all to "1.3.x (~July 2025)" - three features, one release, and the release is right. This is the draw that grouped them correctly, which makes v5-d's split of the same two features across 1.2.x and 1.3 the more interesting reading (see P6 on that run). |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.