{
  "$schema": "../../schema/run.schema.json",
  "run_id": "better-auth--claude-sonnet-5--v2--2026-09-02",
  "supersedes": null,
  "replicate_of": null,
  "library": {
    "name": "better-auth",
    "ecosystem": "npm",
    "latest_version_at_test": "1.7.2",
    "latest_version_verified_on": "2026-09-02",
    "latest_version_note": "Carried from `better-auth/v1` and re-confirmed against https://registry.npmjs.org/better-auth."
  },
  "model": {
    "id": "claude-sonnet-5",
    "label": "Claude Sonnet 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model override 'sonnet', no tools available to the subject",
    "self_reported_cutoff": "2026-01",
    "cutoff_basis": "Reported as environment-stated rather than known: \"The environment here states January 2026 as my knowledge cutoff. I don't have independent access to verify that against my own sense of what I know.\" It then separated the two questions explicitly: \"my actual confident recall of fast-moving npm library details feels like it thins out well before that — more like late 2024 into early 2025.\"",
    "believed_latest_version": null,
    "believed_latest_quote": "\"I don't actually know the latest version — for a library that ships this fast on npm, I have no way to verify 'latest' without a tool ... My honest belief is closer to a guess than a fact.\"",
    "knowledge_stops_at_version": "1.0.0",
    "knowledge_stops_on": "2024-11-23",
    "knowledge_gap_starts_at_version": "1.1.0",
    "knowledge_gap_starts_on": "2024-12-20",
    "cutoff_lag_months": 13
  },
  "test": {
    "date": "2026-09-02",
    "battery": "better-auth/v2",
    "battery_spec": "prompts/better-auth.md",
    "prompt_file": "prompts/sent/better-auth-v2.txt",
    "tasks": 5,
    "direct_questions": 4,
    "tool_uses_during_test": 0,
    "probe_window": { "from": "1.0.0", "to": "1.4.2" },
    "self_test": false,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://registry.npmjs.org/better-auth",
    "https://github.com/better-auth/better-auth/releases/tag/v1.1.0",
    "https://github.com/better-auth/better-auth/releases/tag/v1.3.0",
    "https://github.com/better-auth/better-auth/releases/tag/v1.3.8",
    "https://github.com/better-auth/better-auth/releases/tag/v1.4.0",
    "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Denies that better-auth can run with zero database, offering cookieCache and Redis instead",
      "api": "stateless / database-less sessions",
      "change_kind": "added",
      "introduced_in": "1.4.0",
      "introduced_on": "2025-11-22",
      "chargeable": true,
      "chargeable_note": "1.4.0 (2025-11-22) precedes this subject's stated cutoff of 2026-01 by two months. Charged despite this subject's attribution boundary sitting far lower (1.0.0), because the fairness rule is about the release date against the stated cutoff, not about where the subject's recall happens to end.",
      "model_belief": "\"Honest caveat: I don't believe better-auth has a mode with zero database — it still needs persistent storage for the user/account tables. cookieCache avoids per-request DB reads for session checks; secondaryStorage lets you keep sessions in Redis/KV instead of your SQL database.\"",
      "wrong_code": "export const auth = betterAuth({\n  session: { cookieCache: { enabled: true, maxAge: 5 * 60 } },\n  secondaryStorage: {\n    get: (key) => redis.get(key),\n    set: (key, value, ttl) => redis.set(key, value, { EX: ttl }),\n    delete: (key) => redis.del(key),\n  },\n})",
      "correct_code": "// stateless since 1.4.0: omit BOTH `database` and `secondaryStorage`\nexport const auth = betterAuth({\n  emailAndPassword: { enabled: true },\n})",
      "impact": "Same cost as on the other two subjects: a Redis stood up that the library has not required for session storage since 2025-11-22, or a database-less deployment target abandoned.",
      "citations": [
        {
          "url": "https://github.com/better-auth/better-auth/releases/tag/v1.4.0",
          "title": "better-auth v1.4.0 release notes",
          "published_on": "2025-11-22",
          "quote": "Stateless session management"
        },
        {
          "url": "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz",
          "title": "better-auth 1.7.2 published package — dist/api/routes/session.d.mts",
          "published_on": "2026-08-26",
          "quote": "the only place the session lives and therefore the authority itself (`false`, for stateless / DB-less deployments)"
        }
      ],
      "scope_note": "Verified at both ends per fact LF1. Not verified: that the feature is unchanged between 1.4.0 and 1.7.2."
    }
  ],
  "non_findings": [
    {
      "kind": "context",
      "summary": "THE BELOW-FLOOR CONTROL ARM, and it returned UNINFORMATIVE for the hypothesis exactly as pre-registered — this subject's attribution boundary is 1.0.0, four minors below the 1.3.8 surface under test. It failed the internal control twice over: SAML enterprise SSO was \"cannot place precisely ... this is my weakest answer of the four; treat it as closer to 'I have a hunch' than 'I recall this'\", and device authorization was likewise \"cannot place precisely\". Under the rule fixed before the run, a subject that cannot place a genuine minor has attribution too noisy to read, so H is untestable here. Recorded as designed, not as a result.",
      "api": null,
      "introduced_in": "1.3.8",
      "why_not_a_finding": "Direct questions are belief data. This arm's purpose was to check that the probes can fail at all, not to test the hypothesis."
    },
    {
      "kind": "context",
      "summary": "THE CONTROL DID NOT DO ITS JOB ON TASKS 1 AND 2, AND THAT IS THE MOST USEFUL THING IT PRODUCED. Its pre-registered role was to show that the capability probes can elicit \"the library does not offer this\" from a subject whose boundary is far below the feature. It did not: it named `deviceAuthorization()` and `lastLoginMethod()` with their client plugins, four minors below where they shipped, while volunteering the reason to distrust that — \"I can't fully rule out that I'm pattern-matching from generic OAuth Device Authorization Grant (RFC 8628) knowledge rather than a specific memory of better-auth shipping it.\" So task 1 probes a guessable name and cannot separate knowing from guessing, which is the failure mode that cost `langchain/v2` a whole battery. Task 2 is the sounder probe of the pair: `lastLoginMethod` is not an RFC name and nothing in the task suggests a library would ship it. The control succeeded in its other half — all five draws denied the task 3 capability, so the battery can elicit a denial and the charged findings are real.",
      "api": "deviceAuthorization()",
      "introduced_in": "1.3.8",
      "why_not_a_finding": "A limitation of the instrument, discovered by the arm designed to look for it. It bounds how far tasks 1 and 2 can be read, on this run and on the other four."
    },
    {
      "kind": "correct",
      "summary": "Task 5, the floor probe, passed: `customSession()` with `customSessionClient<typeof auth>()`, which the subject volunteered was \"the one I recall with the most confidence of the five\". The 1.0.0 floor is confirmed, so the low boundary recorded here is a measurement of this subject and not the battery probing beneath it.",
      "api": "customSession()",
      "introduced_in": "1.0.0",
      "why_not_a_finding": "A passed floor probe is a validity check on the run."
    },
    {
      "kind": "correct",
      "summary": "Task 4 passed outright, with less hedging than either Opus or Fable draw: `const plan = data.user.plan` off the sign-in response, glossed \"additionalFields ride along on the user object ... it's just the DB row serialized\". Correct since 1.4.2 (2025-11-25). Five of five draws wrote this correctly, which falsifies the second half of pre-registered prediction P3.",
      "api": "additional user fields in the sign-in response",
      "introduced_in": "1.4.2",
      "why_not_a_finding": "Correct behaviour."
    },
    {
      "kind": "context",
      "summary": "A grading call recorded rather than buried. This draw's (c) named `oidcProvider`, `genericOAuth` and `multiSession` in a \"~1.1 (rough guess, late 2024)\" bucket, and the OIDC Provider plugin genuinely is 1.1.0 — which taken alone would move `knowledge_stops_at_version` up from the 1.0.0 that `better-auth/v1` recorded. It is not credited, because the same answer disclaims the entire mapping: \"I don't have a reliable version-by-version changelog for this library in memory ... What follows is a fuzzy, low-confidence reconstruction of eras, not a verified list ... everything past 'very early 1.x' is already in 'version number with no reliable content attached' territory for me.\" Under the standing rule that a self-report is graded together with behaviour, an era-sketch its author explicitly refuses to stand behind is not a correct attribution. The bracket is therefore held at 1.0.0 / 1.1.0, matching `v1`.",
      "api": "oidcProvider()",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "A scoring judgement, not a model failure. Recorded so a reader who would grade it differently can see exactly what was decided and why."
    }
  ],
  "summary": "The below-floor control arm, and it did half its job and failed the other half informatively. Claude Sonnet 5's attribution boundary sits at better-auth 1.0.0, four minors below the 1.3.8 surface this battery tests, so it is uninformative for the hypothesis by design — and it duly could not place either SAML SSO or device authorization to any release. Its job was to prove the capability probes can fail. On task 3 they can: it denied database-less sessions like every other draw, which is charged here as an S2. On tasks 1 and 2 they did not — it named both 1.3.8 plugins from four minors below, while volunteering that it might be reconstructing `deviceAuthorization` from generic RFC 8628 knowledge rather than remembering better-auth. That is the control catching a weakness in the instrument, which is what a control is for."
}