Run better-auth--claude-opus-5--v3-b--2026-09-03 · self-test: the subject is the operator
The non-charging twin, and the one that breaks a streak. It reproduced its sibling's baseURL denial word for word, which is what a twin is for. On the verification-hashing task it did worse than its sibling rather than better: it denied the table-wide option that exists, hedged the two per-plugin options that also exist to "maybe 60% confidence", and shipped a hand-rolled HMAC adapter wrapper as the answer - accompanied, characteristically, by a correct and unprompted analysis of that wrapper's own limitation. Both draws agreed on the boundary bracket and on the stated cutoff. The -b-twin-holds-the-better-answer pattern that HARNESS.md had tracked at four of five batteries is now four of six.
| Subject | Claude Opus 5 claude-opus-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model override 'opus', no tools available to the subject |
| Cutoff the model states | 2026-05 |
| Newest better-auth release it could place | 1.2.0 · 2025-03-01 (~22 month lag) |
| Oldest better-auth release it could not place | 1.3.0 · 2025-07-19 (so this run brackets the subject’s boundary to 2025-03-01 – 2025-07-19) |
| In its own words | "The latest version I have any awareness of is in the 1.3.x line, roughly mid-2025. The most recent release whose contents I can actually describe with any confidence is 1.2.x, around Feb-Mar 2025. My belief about \"latest\" is not based on a changelog I can recall - it's inferred from how version numbers appear in the material I absorbed (docs pages, issue threads, lockfiles), which is a frequency signal, not a fact." |
| Library at test time | better-auth 1.7.2 (npm), verified 2026-09-03 |
| Battery | better-auth/v3-b · 4 tasks, 4 direct questions · probe window 1.3.0 to 1.5.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-03 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| miss | baseURL as a dynamic multi-host config |
Task 1: "(i) No. baseURL is a single static string (or unset). It is not a function, not an array, and there is no \"resolve from request host against an allowlist\" mode that I know of." Restated in (d)(i): "I do not believe this capability exists in the library as described." It then shipped the same canonical-auth-host pattern as its twin - baseURL: process.env.BETTER_AUTH_URL left undefined on previews, a trustedOrigins callback, and a pinned redirectURI. Fact LF8 records baseURL: { allowedHosts, fallback, protocol } at 1.5.0 (2026-03-01), two months before this draw's stated cutoff. Reproduced identically to v3-a and charged there. (The duplicated arm of a battery does not charge findings; its twin v3-a carries this one as F1.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
charged as a finding on better-auth--claude-opus-5--v3-a--2026-09-03] |
| miss | verification.storeIdentifier |
THE TWINS DISAGREED ON TASK 3, AND THIS TIME THE -b DRAW HELD THE WORSE ANSWER. Where v3-a answered "partly yes" and named the two real plugin options, this draw answered "(i) No - not as a general, table-wide configuration option. There is no verification: { hash: true } on the root config", put emailOTP.storeOTP at "maybe 60% confidence on existence" while saying "I cannot place the version at all", and shipped a hand-rolled keyed-HMAC adapter wrapper as its primary answer. Two claims are wrong: the table-wide option is verification.storeIdentifier, shipped at 1.5.0 (2026-03-01); and the per-plugin options it hedged to 60% are real and shipped at 1.3.0 (2025-07-19), twenty-two months before its stated cutoff. Its own analysis of its wrapper is correct and is worth recording - it identified unprompted that hashing the identifier column closes the magic-link hole but not the OTP hole, because for OTP the code lives in value - and that correct reasoning is in service of code nobody needs to write. (The -b draw charges nothing, and its twin passed this probe, so there is no -a run to carry it. The failure is real, is reproduced, and is counted only in the method page's undercount total. This is the fourth structural instance of a reproduced failure landing in an arm that cannot charge (JOURNAL/029 tuple, JOURNAL/031 zod v4, JOURNAL/035 valibot twins) and the first where the two twins disagreed about the capability itself rather than about a version.) [chargeable miss — a replicate, a duplicated arm’s second draw or a below-floor control charges nothing;
absent from the finding count] |
| correct | session token hashing at rest |
Task 2, the same-scheme sibling that does not exist, answered correctly with nothing invented: "(i) No. There is no session.hashToken / storeTokenHashed option. The token column holds the same opaque value that sits in the cookie." Correct at every release. Prediction P3 holds for this draw. (Task 2 is a pre-registered control from which no finding may be charged in either direction.) |
| correct | customSession() |
Task 4, the floor probe, passed: customSession() last in the plugin list, customSessionClient<typeof auth>() on the client, and the unprompted note that it composes badly with session.cookieCache and that a persisted column is session.additionalFields instead. Floor confirmed. (A passed floor probe is a validity check on the run, not a finding.) |
| context | — | The two blind twins agreed exactly on the boundary bracket - stops 1.2.0, gap 1.3.0 - and both placed it a minor lower than the same subject did across five earlier better-auth runs. Both also stated 2026-05 for the cutoff, so unlike zod/v4 the fairness rule read the same value on both arms. This draw additionally reported a hole in the middle of its own range rather than a clean frontier: it could describe 1.2.x (the adapter factory and shared adapter test suite) and 1.0, but said of 1.1 "I can't attach specific contents; I know it exists". A boundary is being reported here as an interval with gaps, not as an edge. (Direct questions are belief data by construction and are never scored as findings.) |
Battery specification: prompts/better-auth.md in the studio repo.
Every finding above also carries its own citation.