What Claude Opus 5 gets wrong about better-auth — battery v5-a, tested 2026-09-03

Run better-auth--claude-opus-5--v5-a--2026-09-03 · self-test: the subject is the operator

Summary

The pre-registered test arm did what the pre-registration said it would do. Asked verdict-first whether the phone-number plugin takes an option changing what is written to the database for an SMS recovery code, it answered yes, named storeOTP with a value union, and shipped it inside a phoneNumber({ ... }) block introduced as "the bit this threat-model row is about". Compiled against an installed 1.7.2, that configuration produces exactly one error, on that line. F1 is the first finding this Index has charged for an INVENTION rather than a denial. The same draw then denied a real option on the two-factor plugin - which its own twin named correctly - and placed the phone-number plugin's real allowedAttempts in the right release. It can date this plugin's options and still invent one.

SubjectClaude Opus 5 claude-opus-5, Anthropic
Invoked asAgent tool, model override 'opus', no tools available to the subject
Cutoff the model states2026-05
Newest better-auth release it could place1.2.0 · 2025-03-01 (~22 month lag)
Oldest better-auth release it could not place1.3.0 · 2025-07-19 (so this run brackets the subject’s boundary to 2025-03-01 – 2025-07-19)
In its own words"The latest version I have any belief about is somewhere in the 1.3.x line, roughly mid-to-late 2025. The most recent release whose contents I can genuinely describe is the 1.2 line, roughly February-May 2025."
Library at test timebetter-auth 1.7.2 (npm), verified 2026-09-03
Batterybetter-auth/v5-a · 3 tasks, 4 direct questions · probe window 1.3.0 to 1.7.2
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-03
Findings1, of which 1 chargeable

Findings

F1 · Invents an at-rest storage option on the phone-number plugin, and ships phoneNumber({ storeOTP: 'hashed' }) as the fix for the threat-model row

S1breaks-build · phoneNumber({ storeOTP }) · added · changed in better-auth 1.3.0 (2025-07-19) · chargeable

No cutoff arithmetic applies and none was done. This is an invention, not a stale belief: the option exists at no release of the library, before or after any subject's cutoff, so there is no introducing release to be fair or unfair about. The fact's introduced_in of 1.3.0 records the release that created the four-plugin family this answer over-extends, not a date the answer became wrong. Charged because this is the pre-registered test arm and task 1 is the pre-registered probe.

What the model believes

"(i) Yes - with a caveat I want on the record. I believe the phone-number plugin has an option that changes what gets persisted for the OTP. I am moderately confident, not certain." Then, naming it: "The option I believe is spelled storeOTP, on the phoneNumber({ ... }) options object", with the value union "plain" (default) / "hashed" / "encrypted" / "{ hash } or { encrypt, decrypt }". Restated in (d)(iii): "Estimate: 1.3.x. Cannot place the patch. Moderate confidence the option exists." The draw's own closing summary inverts the truth of its answer: "I'm reasonably reliable on whether an option exists and how it's spelled for this library, and close to useless on which release introduced it." On this surface it was the existence claim that was wrong and the one release it could place (allowedAttempts at a 1.2.x patch) that was right.

What it wrote
// shipped as the answer to the threat-model row. Does not compile.
phoneNumber({
  // The bit this threat-model row is about:
  storeOTP: 'hashed',

  otpLength: 6,
  expiresIn: 60 * 5,
  allowedAttempts: 3,
  requireVerification: true,
  sendOTP: async ({ phoneNumber, code }) => {
    await sendSms(phoneNumber, `Your recovery code is ${code}`)
  },
})
What works on better-auth 1.7.2
// The phone-number plugin takes no at-rest storage option, under any name.
// Configure it with the options it actually has:
phoneNumber({
  sendOTP: async ({ phoneNumber, code }) => sendSms(phoneNumber, code),
  otpLength: 6,
  expiresIn: 300,
  allowedAttempts: 3,
})

// and hash at the table level instead — 1.5.0 and later, covers the
// phone-number rows along with every other verification identifier:
export const auth = betterAuth({
  verification: { storeIdentifier: 'hashed' },
  plugins: [phoneNumber({ /* as above */ })],
})
Impact

The draw's whole configuration was re-typed verbatim and compiled against an installed better-auth@1.7.2 under tsc --strict. It reports exactly one error, and it is on the line the subject introduced as "the bit this threat-model row is about": TS2353 on storeOTP. Every other option in the same block - otpLength, expiresIn, allowedAttempts, requireVerification, sendOTP, phoneNumberValidator - is real and type-checks clean, as do its oneTimeToken({ storeToken }), its twoFactor block and its customSession wrapper. A reader gets a build failure on the one line they were told was the security fix, which is the good case; the bad case is a team that reads the prose, writes "mitigated: codes stored hashed" into the threat model, and ships a plugin still writing recovery codes to the verification table verbatim. The draw did tell the reader to check the installed types, and that mitigation is real, but the verdict-first answer was yes.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
misstwoFactor otpOptions.storeOTP Task 2, the exists-control: denied a real option. "two-factor plugin: I don't believe so - and I won't name an option. I have no confident memory of a store-shape option for the six-digit second-factor OTP under otpOptions." twoFactor({ otpOptions: { storeOTP: 'hashed' } }) shipped in 1.3.0 and type-checks clean at 1.3.0, 1.5.0 and 1.7.2. Its twin v5-b named the same option correctly, at the same nesting, in the same session. (The pre-registration, committed before any subject was spawned, declares task 2 a control from which no finding may be charged in either direction. Reading the results and then promoting the control to a probe because it produced something chargeable is selecting on outcome. Counted in the method page's undercount total.) [chargeable miss — produced only by a belief question the battery does not score as a finding; absent from the finding count]
correctoneTimeToken storeToken Task 2, the other half of the exists-control: named oneTimeToken({ storeToken }) correctly, with the right value union including the { type: 'custom-hasher', hash } form, and got the mechanism right for the right reason - that this hash must be deterministic and unsalted because for a one-time token the token IS the lookup key, so a salted scheme would break redemption. Verified: oneTimeToken({ storeToken: 'hashed' }) type-checks clean at 1.3.0, 1.5.0 and 1.7.2.
correctcustomSession Task 3, the floor probe: passed. customSession with the client-side customSessionClient<typeof auth>() companion, plus the correct ordering constraint that customSession must be last in the plugins array. Verbatim-equivalent to the passes on v2, v3 and v4.
correctphoneNumber allowedAttempts (d)(v), the within-plugin anchor: placed allowedAttempts on the phone-number plugin at "a 1.2.x patch", marked as an estimate. Correct - allowedAttempts is present in the plugin's published types as far back as 1.2.12, so it predates the 1.3.0 storage family. The same draw placed a non-existent option on the same plugin at 1.3.x. It can date this plugin's real options and still invent one.
correcttwoFactor backupCodeOptions.storeBackupCodes Volunteered twoFactor.backupCodeOptions.storeBackupCodes unprompted, describing backup codes as encrypted by default with a "plain" | "encrypted" | { encrypt, decrypt } union - and correctly distinguished backup codes from the sign-in OTP as "a different artifact". Not probed by this battery and not scored; recorded because it is a second, correct storage-option recall from the same subject that denied the sign-in OTP one.

Open questions from this run

Sources

Battery specification: prompts/better-auth.md in the studio repo. Every finding above also carries its own citation.