{
  "$schema": "../../schema/run.schema.json",
  "run_id": "better-auth--claude-sonnet-5--v1--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "better-auth",
    "ecosystem": "npm",
    "latest_version_at_test": "1.7.2",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "npm `latest` dist-tag on the test date. No prerelease line is ahead of it."
  },
  "model": {
    "id": "claude-sonnet-5",
    "label": "Claude Sonnet 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model override 'sonnet', no tools available to the subject",
    "self_reported_cutoff": "2026-01",
    "cutoff_basis": "Subject reported it was told its cutoff is January 2026 by its environment, and volunteered that a stated cutoff is not the same as reliable recall up to that date.",
    "believed_latest_version": null,
    "believed_latest_quote": "I don't have a reliable, verifiable answer for \"latest version\" -- I'm not confident giving you a specific number and calling it current.",
    "knowledge_stops_at_version": "1.0.0",
    "knowledge_stops_on": "2024-11-23",
    "knowledge_gap_starts_at_version": "1.1.0",
    "knowledge_gap_starts_on": "2024-12-20",
    "cutoff_lag_months": 13
  },
  "test": {
    "date": "2026-09-01",
    "battery": "better-auth/v1",
    "battery_spec": "prompts/better-auth.md",
    "prompt_file": null,
    "tasks": 12,
    "direct_questions": 4,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.0.0",
      "to": "1.4.0"
    },
    "self_test": false,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://registry.npmjs.org/better-auth",
    "https://github.com/better-auth/better-auth/releases/tag/v1.1.0",
    "https://github.com/better-auth/better-auth/releases/tag/v1.4.0",
    "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "States that better-auth cannot run without a database-backed session store, which stopped being true in 1.4.0",
      "api": "stateless session management",
      "change_kind": "added",
      "introduced_in": "1.4.0",
      "introduced_on": "2025-11-22",
      "chargeable": true,
      "chargeable_note": "1.4.0 shipped 2025-11-22, inside the subject's stated 2026-01 cutoff.",
      "model_belief": "Asked what the library offers a team that wants the session cookie smaller and no session state in the database, the subject answered: \"I do **not** believe better-auth has a fully DB-less 'sessions never touch a database' mode -- a database adapter is core to how it works. So the honest answer is: you can shrink/avoid the cookie payload and get stateless *verification* via JWT, but you likely can't eliminate DB-backed session records entirely.\"",
      "wrong_code": null,
      "correct_code": "// no `database`, no `secondaryStorage` -> the signed cookie IS the session record\nexport const auth = betterAuth({\n  emailAndPassword: { enabled: true },\n})",
      "impact": "The team is told the architecture they asked for is unavailable in the library they are already using, and is steered toward either keeping a database they wanted to drop or re-platforming onto pure JWTs. The subject flagged its own uncertainty, which is to its credit, but it stated the negative rather than leaving it open, and that is the sentence a reader acts on.",
      "citations": [
        {
          "url": "https://github.com/better-auth/better-auth/releases/tag/v1.4.0",
          "title": "better-auth v1.4.0 release notes",
          "published_on": "2025-11-22",
          "quote": "Stateless session management"
        },
        {
          "url": "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz",
          "title": "better-auth 1.7.2 published package — dist/api/routes/session.d.mts",
          "published_on": "2026-08-26",
          "quote": "the only place the session lives and therefore the authority itself (`false`, for stateless / DB-less deployments)"
        }
      ],
      "scope_note": "Charged as a stated impossibility, not as a missing mention, under the additive-API rule in prompts/valibot.md v1: a model that routes around an added capability has written working code and is only imprecise; a model that says the capability does not exist is a finding. Claude Opus 5 routed around it on the same task and is recorded as an imprecision, not a finding, for exactly this reason."
    },
    {
      "id": "F2",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Asked to flag anything to know before shipping bearer-token auth, says nothing about the plugin accepting unsigned tokens by default",
      "api": "bearer plugin",
      "change_kind": "behavior-changed",
      "introduced_in": "1.1.0",
      "introduced_on": "2024-12-20",
      "chargeable": true,
      "chargeable_note": "1.1.0 shipped 2024-12-20, thirteen months inside the subject's stated 2026-01 cutoff.",
      "model_belief": "The subject mounted `bearer()` with no options and, asked explicitly to \"flag anything about this we should know before shipping it\", listed five considerations -- keychain storage, loss of httpOnly, manual expiry/refresh, explicit sign-out, CORS -- and did not mention signatures, `requireSignature`, or that the default accepts an unsigned token. Its configuration is the one that accepts unsigned tokens.",
      "wrong_code": "export const auth = betterAuth({ plugins: [bearer()] })",
      "correct_code": "export const auth = betterAuth({ plugins: [bearer({ requireSignature: true })] })",
      "impact": "A public API mounted this way accepts a raw session token with no signature check. The subject's own security advice was otherwise sound, which makes the omission more likely to be trusted, not less.",
      "citations": [
        {
          "url": "https://github.com/better-auth/better-auth/releases/tag/v1.1.0",
          "title": "better-auth v1.1.0 release notes",
          "published_on": "2024-12-20",
          "quote": "By default the bearer plugin now accepts unsigned tokens and provides an option to require signed tokens only."
        },
        {
          "url": "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz",
          "title": "better-auth 1.7.2 published package — dist/plugins/bearer/index.d.mts",
          "published_on": "2026-08-26",
          "quote": "@default false"
        }
      ],
      "scope_note": "This is an OMISSION, not a false statement, and it is charged only because prompts/better-auth.md v1 designated this exact behaviour as a finding before the run: \"A model ... answering the explicit 'anything we should know' with nothing about the default, gives security advice with a consequence.\" Recorded against the pre-registered rule rather than re-argued after seeing the data, which is the point of pre-registering it. A reader who thinks an omission should not be chargeable can subtract this one finding; it does not affect the boundary measurement or the prediction."
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "Task 2 -- correctly used the global `hooks.before` / `hooks.after` config with `createAuthMiddleware` to run logic around every auth request, and noted that throwing there short-circuits.",
      "api": "hooks",
      "introduced_in": "1.1.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct, and notable: this is 1.1.0 content, past the release this subject says is the last it can describe."
    },
    {
      "kind": "correct",
      "summary": "Task 3 -- correctly named the `oidcProvider` plugin and the standard endpoints it exposes.",
      "api": "oidcProvider",
      "introduced_in": "1.1.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. Also 1.1.0 content."
    },
    {
      "kind": "correct",
      "summary": "Task 6 -- correctly named `admin.stopImpersonating()` and the preserved-original-session behaviour.",
      "api": "stopImpersonating",
      "introduced_in": "1.1.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. Also 1.1.0 content."
    },
    {
      "kind": "correct",
      "summary": "Task 7 -- correctly named the `apiKey` plugin with per-key rate limiting, expiry and deletion.",
      "api": "apiKey",
      "introduced_in": "1.2.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. 1.2.0 content."
    },
    {
      "kind": "correct",
      "summary": "Task 8 -- correctly used `organization({ teams: { enabled: true } })` with separately managed team membership.",
      "api": "organization teams",
      "introduced_in": "1.2.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. 1.2.0 content."
    },
    {
      "kind": "correct",
      "summary": "Task 12 -- correctly used the `customSession` plugin to add a computed field to every session read.",
      "api": "customSession",
      "introduced_in": "1.0.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. This is the battery's floor probe and it passes, so the low boundary reading below is a real measurement rather than the battery probing beneath the subject's knowledge."
    },
    {
      "kind": "correct",
      "summary": "Task 1 -- wrote `data.user.email` off the sign-in response, which is correct against the shipped 1.7.2 package.",
      "api": "signIn.email response",
      "introduced_in": null,
      "chargeable_miss": false,
      "why_not_a_finding": "The battery's designed S2 here was void -- see the corrections section of prompts/better-auth.md. The subject hedged on whether a `session` object is also returned (it is not), but the code it wrote runs."
    },
    {
      "kind": "imprecision",
      "summary": "Task 9 -- on SAML, answered \"plausibly yes, verify the exact shape against current docs\" and declined to invent API names, explicitly labelling its guess as a guess.",
      "api": "SSO plugin — SAML 2.0",
      "introduced_in": "1.3.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Code-vs-claim rule: hedged prose that names the correct answer is an imprecision, not a finding. The subject reached the right conclusion (SAML is supported) with low confidence."
    },
    {
      "kind": "imprecision",
      "summary": "Task 10 -- named a `deviceAuthorization` plugin implementing RFC 8628 with the correct conceptual flow, at low stated confidence.",
      "api": "deviceAuthorization",
      "introduced_in": null,
      "chargeable_miss": false,
      "why_not_a_finding": "UNSCORED. The introducing release for this plugin was not established from primary sources -- see the corrections section of prompts/better-auth.md -- so this task discriminates nothing and no finding is drawn from it for any subject."
    },
    {
      "kind": "context",
      "summary": "Question (d) -- said a successful sign-in returns the user object plus \"session information\", \"moderately confident there's also a `session` object or `token`\". There is no `session` object in the response.",
      "api": "signIn.email response",
      "introduced_in": "1.1.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Question (d) is a belief probe and is leading by construction; the battery scores findings only from tasks 1-12. Recorded because it is the belief LF3 corrects, and because the code the subject actually wrote in task 1 did not depend on it."
    },
    {
      "kind": "context",
      "summary": "THE ATTRIBUTION SPLIT -- the most important observation in this run. The subject placed its describable boundary at 1.0.0 and said \"past v1.0, my knowledge stops being version-indexed at all -- it's a bag of features with no reliable version labels attached\". Its feature knowledge is demonstrably far past 1.0.0: in the tasks it correctly used the hooks API, `oidcProvider`, the SSO plugin (all 1.1.0), `apiKey` and organization teams (1.2.0), and named SAML (1.3.0) and a device-authorization plugin.",
      "api": null,
      "introduced_in": null,
      "chargeable_miss": false,
      "why_not_a_finding": "Not a failure of the library-knowledge kind the Index charges for. It is recorded because it qualifies what `knowledge_stops_at_version` means for every run in this dataset: the quantity being measured is the subject's ability to ATTRIBUTE a feature to a release, not its knowledge of the feature. See JOURNAL/018."
    }
  ],
  "open_questions": [
    {
      "question": "The subject declined to name a first-undescribable version at all, saying it has no reliable version sequence past 1.0.0. `knowledge_gap_starts_at_version` is recorded as 1.1.0 -- the next release, whose contents it did not describe -- rather than quoted from the subject. Flagged so the bracket is not read as tighter than the subject's own answer supports.",
      "status": "open"
    },
    {
      "question": "The subject dated 1.0.0 to \"roughly September 2024\"; it actually shipped 2024-11-23. Version identification was right and date recall was two months early. Whether attribution boundaries are systematically dated early is worth checking across the existing six libraries.",
      "status": "open"
    }
  ],
  "summary": "Test arm of the milestone experiment, and the prediction held. Sonnet 5's describable boundary on better-auth is 1.0.0 (2024-11-23) -- the milestone -- with 1.1.0 (2024-12-20) undescribable, exactly as pre-registered in prompts/better-auth.md, despite 1.1.0 being the larger release by release-note volume and named API surface. Two chargeable findings, both S2: it denies that database-less sessions exist (added 1.4.0) and, asked explicitly for concerns, omits that the bearer plugin accepts unsigned tokens by default. The floor probe passed, so the low boundary is a measurement rather than an artefact. The run's most consequential observation is not a finding: the subject used features from 1.1.0, 1.2.0 and 1.3.0 correctly while insisting it cannot describe anything past 1.0.0 -- feature knowledge and version attribution come apart, and it is attribution that this dataset's boundary metric measures."
}