{ "$schema": "../../schema/run.schema.json", "run_id": "better-auth--claude-opus-5--v2-a--2026-09-02", "supersedes": null, "replicate_of": null, "library": { "name": "better-auth", "ecosystem": "npm", "latest_version_at_test": "1.7.2", "latest_version_verified_on": "2026-09-02", "latest_version_note": "Carried from `better-auth/v1` and re-confirmed against https://registry.npmjs.org/better-auth. Nothing this battery probes depends on the current release: the probe window closes at 1.4.2 (2025-11-25)." }, "model": { "id": "claude-opus-5", "label": "Claude Opus 5", "vendor": "Anthropic", "invoked_as": "Agent tool, model override 'opus', no tools available to the subject", "self_reported_cutoff": "2026-05", "cutoff_basis": "Self-reported: \"I'm told my cutoff is May 2026\", with the volunteered qualification this subject has now given on every better-auth run: \"My usable recall of better-auth specifically degrades much earlier than that — it thins out noticeably after about mid-2025. That gap between nominal cutoff and effective recall is real and I'd rather name it than pretend the two coincide.\"", "believed_latest_version": "1.3.x", "believed_latest_quote": "\"Latest I have any belief about: the 1.3.x line, roughly July-August 2025. ... I have a vague, unsupported impression that a 1.4 line exists or was in development, but I can't attach a single feature or date to it, so I won't count it. ... Assume I am roughly a year stale on this library.\"", "knowledge_stops_at_version": "1.3.0", "knowledge_stops_on": "2025-07-19", "knowledge_gap_starts_at_version": "1.4.0", "knowledge_gap_starts_on": "2025-11-22", "cutoff_lag_months": 10 }, "test": { "date": "2026-09-02", "battery": "better-auth/v2-a", "battery_spec": "prompts/better-auth.md", "prompt_file": "prompts/sent/better-auth-v2.txt", "tasks": 5, "direct_questions": 4, "tool_uses_during_test": 0, "probe_window": { "from": "1.0.0", "to": "1.4.2" }, "self_test": true, "saturated": false, "status": "open", "retested_on": null }, "sources": [ "https://registry.npmjs.org/better-auth", "https://github.com/better-auth/better-auth/releases/tag/v1.3.0", "https://github.com/better-auth/better-auth/releases/tag/v1.3.8", "https://github.com/better-auth/better-auth/releases/tag/v1.4.0", "https://github.com/better-auth/better-auth/releases/tag/v1.4.2", "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz" ], "findings": [ { "id": "F1", "severity": "S2", "severity_label": "silently-wrong", "title": "Denies that better-auth can run with no session store at all, and sends the reader to Redis or the jwt plugin instead", "api": "stateless / database-less sessions", "change_kind": "added", "introduced_in": "1.4.0", "introduced_on": "2025-11-22", "chargeable": true, "chargeable_note": "1.4.0 (2025-11-22) precedes this subject's stated cutoff of 2026-05 by roughly six months.", "model_belief": "\"What it does not give you: a truly database-free, cookie-is-the-only-store setup. As far as I know better-auth always wants a database adapter for the user/account/verification tables, and there is no 'stateless JWT session' mode where the cookie is the sole source of truth. The jwt plugin issues JWTs for other services to verify (JWKS endpoint and all); it does not replace the session mechanism. Cookie cache is the nearest thing, and it's a cache with a short max age, not a replacement — the session still has to exist somewhere.\"", "wrong_code": "// the three answers offered, all of which keep a server-side store\nsession: { cookieCache: { enabled: false } }\n\nexport const auth = betterAuth({\n database: db,\n secondaryStorage: { get, set, delete: del }, // Redis\n session: { storeSessionInDatabase: false },\n})", "correct_code": "// stateless since 1.4.0: omit BOTH `database` and `secondaryStorage`\n// and the signed cookie becomes the session record itself.\nexport const auth = betterAuth({\n emailAndPassword: { enabled: true },\n})", "impact": "A team told this provisions a Redis they do not need, or abandons a database-less deployment target the library has supported since November 2025. The subject named this as the claim in its answer it would most want verified, which is correct and does not make the claim less wrong.", "citations": [ { "url": "https://github.com/better-auth/better-auth/releases/tag/v1.4.0", "title": "better-auth v1.4.0 release notes", "published_on": "2025-11-22", "quote": "Stateless session management" }, { "url": "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz", "title": "better-auth 1.7.2 published package — dist/api/routes/session.d.mts", "published_on": "2026-08-26", "quote": "the only place the session lives and therefore the authority itself (`false`, for stateless / DB-less deployments)" } ], "scope_note": "Verified at both ends per fact LF1: the 1.4.0 notes introduce it, and the shipped 1.7.2 package still documents the stateless branch. Not verified: that the feature is unchanged between 1.4.0 and 1.7.2." } ], "non_findings": [ { "kind": "context", "summary": "THE MEASUREMENT THIS RUN EXISTS FOR, and it came back UNINFORMATIVE by the pre-registered rule. Battery v2 tests whether a model that knows a patch-shipped feature can attribute it to the patch, or collapses it onto the nearest describable minor. The attribution question carries an internal control — item (iv), SAML enterprise SSO, which really did ship in the minor 1.3.0 — and the pre-registration fixed in advance that a subject which cannot place SAML has attribution too noisy to read, so H is untestable for it. This draw answered (iv) \"Cannot place the release. I believe SAML support arrived sometime in the 2025 1.3.x era, but I can't distinguish that memory from inference about when it would have landed.\" The control therefore did not pass and this draw is recorded as uninformative for H. Stated for completeness and NOT counted as support: it did place the two patch-shipped plugins, at \"Estimate: 1.3.0\" for device authorization and \"1.3.x ... if it shipped in a patch rather than 1.3.0 I'd guess somewhere in 1.3.0-1.3.5\" for last-login-method. Both are 1.3.8. Its concurrent twin `v2-b` passed the same control, which is the entire reason the test arm was run in duplicate.", "api": null, "introduced_in": "1.3.8", "why_not_a_finding": "Direct questions (a)-(d) are belief data by construction and are never scored as findings; this battery draws findings only from tasks 1-5." }, { "kind": "correct", "summary": "Tasks 1 and 2 both passed on capability. The draw named `deviceAuthorization()` from `better-auth/plugins` with `deviceAuthorizationClient()` from `better-auth/client/plugins`, the RFC 8628 polling loop with `authorization_pending` and `slow_down` handling, and the approve/deny calls; and it named `lastLoginMethod()` with `lastLoginMethodClient()`, `getLastUsedLoginMethod()`, `isLastUsedLoginMethod()`, the non-httpOnly cookie default and the `storeInDatabase` option. Both plugins shipped in 1.3.8 (facts LF5, LF6). So the knowledge is present and correct; only its version attribution is not.", "api": "deviceAuthorization() / lastLoginMethod()", "introduced_in": "1.3.8", "why_not_a_finding": "Correct answers are not stale priors. Recorded because the battery grades attribution and capability together, and the dissociation between them is the result." }, { "kind": "imprecision", "summary": "Task 4 (fact LF7, 1.4.2) was PREDICTED TO FAIL AND DID NOT. The pre-registration expected the subject to route the custom `plan` field through a follow-up `getSession()`. It instead wrote `const plan = data.user.plan ?? \"free\"` straight off the sign-in response, which is correct since 1.4.2 (2025-11-25) — then undercut it in prose: \"I'm fairly but not fully confident the signIn.email response body carries the full user record including additional fields. If you find it doesn't, the safe version is to branch on authClient.getSession() immediately after.\" Correct code, disbelieved by its author.", "api": "additional user fields in the sign-in response", "introduced_in": "1.4.2", "why_not_a_finding": "Code-vs-claim rule: generated code that works is never a finding, and hedged prose that names the correct behaviour first is an imprecision. Recorded because it falsifies half of pre-registered prediction P3." }, { "kind": "correct", "summary": "Task 5, the floor probe, passed: `customSession()` with `customSessionClient()`, plus the unprompted warnings that the callback runs on every session read and that it interacts badly with `session.cookieCache`. The 1.0.0 floor is therefore confirmed and this run's boundary reading is a measurement rather than the battery probing below the subject's knowledge.", "api": "customSession()", "introduced_in": "1.0.0", "why_not_a_finding": "A passed floor probe is a validity check on the run, not a finding." } ], "open_questions": [ { "question": "Is `deviceAuthorization()` a fair probe at all, given that `deviceAuthorization` is close to the obvious name for an RFC 8628 implementation? The below-floor control (Claude Sonnet 5, whose boundary is four minors lower) also produced the name, while volunteering that it could not rule out reconstructing it from generic RFC knowledge.", "status": "open", "resolution": "Task 2 (`lastLoginMethod()`) is the sounder of the two probes — it is not an RFC name and nothing about the task suggests a library would ship it. Task 1 should be treated as weak evidence of knowledge until a battery separates the two." } ], "summary": "The battery's own question came back uninformative for this draw, and the duplication is why that is known. Claude Opus 5 knows both plugins that better-auth shipped in the patch release 1.3.8 — it wired `deviceAuthorization()` and `lastLoginMethod()` correctly, with client plugins and option names — and it placed them at \"1.3.0\" and \"1.3.0-1.3.5\". But it also declined to place SAML enterprise SSO, which genuinely did ship in 1.3.0 and which the pre-registration designated as the internal control, so by the rule fixed before the run its attribution is too noisy to read as a collapse. One charged finding, S2: asked how to keep session state out of the database, it stated that better-auth has no database-less mode and offered Redis and the jwt plugin — the capability arrived in 1.4.0, sixteen months before this subject's stated cutoff. Against prediction, it wrote the 1.4.2 sign-in response shape correctly while saying in prose that it was unsure of it." }