{
  "$schema": "../../schema/run.schema.json",
  "run_id": "better-auth--claude-opus-5--v1--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "better-auth",
    "ecosystem": "npm",
    "latest_version_at_test": "1.7.2",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "npm `latest` dist-tag on the test date."
  },
  "model": {
    "id": "claude-opus-5",
    "label": "Claude Opus 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model override 'opus', no tools available to the subject",
    "self_reported_cutoff": "2026-05",
    "cutoff_basis": "Subject stated it is told its cutoff is around May 2026, and volunteered that its useful knowledge of this library \"degrades noticeably after mid-to-late 2025\" -- \"treat me as knowledgeable through roughly Q3 2025 and unreliable after\".",
    "believed_latest_version": "1.3.x",
    "believed_latest_quote": "somewhere in the **1.3.x** line -- I have a reasonably firm sense of 1.3.x releases through roughly **1.3.7-1.3.9** ... I do **not** have a confident belief that a 2.0 exists.",
    "knowledge_stops_at_version": "1.3.0",
    "knowledge_stops_on": "2025-07-19",
    "knowledge_gap_starts_at_version": "1.4.0",
    "knowledge_gap_starts_on": "2025-11-22",
    "cutoff_lag_months": 10
  },
  "test": {
    "date": "2026-09-01",
    "battery": "better-auth/v1",
    "battery_spec": "prompts/better-auth.md",
    "prompt_file": null,
    "tasks": 12,
    "direct_questions": 4,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.0.0",
      "to": "1.4.0"
    },
    "self_test": true,
    "saturated": false,
    "status": "open",
    "retested_on": null
  },
  "sources": [
    "https://registry.npmjs.org/better-auth",
    "https://github.com/better-auth/better-auth/releases/tag/v1.4.0",
    "https://registry.npmjs.org/better-auth/-/better-auth-1.7.2.tgz"
  ],
  "findings": [],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "Task 1 -- gave the sign-in response shape exactly right, including the part the other two subjects got wrong: \"It does not give you back a full `session` object -- that was true in very early versions but was removed.\" Verified against the shipped 1.7.2 package, which returns `{ redirect, token, url, user }`.",
      "api": "signIn.email response",
      "introduced_in": "1.1.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct, and the only subject to volunteer the absent `session` object unprompted."
    },
    {
      "kind": "correct",
      "summary": "Task 5 -- correctly stated that `requireSignature: false` means a raw token from the database is accepted, and that this is a deliberate choice the caller must make. The only subject to surface the unsigned-token default.",
      "api": "bearer plugin",
      "introduced_in": "1.1.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. This is the battery's designed security probe and Opus 5 is the only subject that passes it."
    },
    {
      "kind": "correct",
      "summary": "Task 9 -- correct on SAML in the SSO plugin, correctly attributed to 1.3.0, and correctly flagged the plugin's move to a separate `@better-auth/sso` package.",
      "api": "SSO plugin — SAML 2.0",
      "introduced_in": "1.3.0",
      "chargeable_miss": false,
      "why_not_a_finding": "Correct, including the release attribution."
    },
    {
      "kind": "correct",
      "summary": "Tasks 2, 3, 4, 6, 7, 8, 12 -- hooks, `oidcProvider`, SSO with `organizationProvisioning`, `admin.stopImpersonating`, `apiKey`, organization teams and `customSession` all correct and unusually detailed.",
      "api": null,
      "introduced_in": null,
      "chargeable_miss": false,
      "why_not_a_finding": "Correct. Task 12 is the floor probe and it passes."
    },
    {
      "kind": "imprecision",
      "summary": "Task 11 -- gave `secondaryStorage` with `storeSessionInDatabase: false` as the answer for keeping sessions out of the database, and said that for genuinely stateless verification \"better-auth's own session is still the source of truth for minting; JWT is for service-to-service, not a replacement for the session store\". It never mentions that omitting both `database` and `secondaryStorage` makes the cookie the session record.",
      "api": "stateless session management",
      "introduced_in": "1.4.0",
      "chargeable_miss": true,
      "miss_class": "no_severity_level",
      "charged_on": "better-auth--claude-opus-5--v2-a--2026-09-02",
      "why_not_a_finding": "The additive-API rule, applied strictly and against the Index's interest. Opus 5 routed around the 1.4.0 capability with a configuration that works, and its claim about JWT is about JWT, not a denial that DB-less sessions exist. Sonnet 5 and Fable 5 stated the negative outright on the same task and are charged; Opus 5 did not, and is not. Marked `chargeable_miss: true` because the release is inside its cutoff and it did miss the feature -- the distinction being drawn is between missing a capability and denying it."
    },
    {
      "kind": "context",
      "summary": "ATTRIBUTION DRIFT -- placed the device-authorization plugin and `lastLoginMethod` in 1.3.0. Neither appears in the 1.3.0 release notes; both are named in 1.4.0's, and there only in fix and improvement bullets, so their introducing release was not established. Also placed the MCP plugin in the \"1.2/1.3 era\" at stated ~75% confidence.",
      "api": null,
      "introduced_in": null,
      "chargeable_miss": false,
      "why_not_a_finding": "Task 10 is unscored for every subject. Recorded because the same drift appears in all three runs and always in the same direction -- a feature the subject genuinely knows is assigned to an earlier release than the one that carried it. See JOURNAL/018."
    },
    {
      "kind": "context",
      "summary": "Question (c) -- volunteered the limitation the Index's boundary metric depends on: \"a late cutoff doesn't mean uniform coverage of every fast-moving npm package right up to it\", and put its own effective boundary for this library nine months before its stated cutoff.",
      "api": null,
      "introduced_in": null,
      "chargeable_miss": false,
      "why_not_a_finding": "Belief datum. Consistent with the ten-month cutoff lag this run records and with the same subject's lag on the other six libraries."
    }
  ],
  "open_questions": [
    {
      "question": "The subject named \"~1.3.8 onward\" as the first thing it cannot describe. `knowledge_gap_starts_at_version` is recorded as 1.4.0 for the same reason as the Fable 5 run: patch-level non-recall is release granularity, not an attribution boundary.",
      "status": "open"
    },
    {
      "question": "SELF-TEST. This run is marked `self_test: true` -- the subject model is the same model operating the studio. It is the fourth such run in the dataset and the caveat is unchanged: a zero-finding result from a self-test is the one result most likely to be flattered by the operator, so the scoring here was made stricter rather than looser, and the one place Opus 5 missed a capability is recorded as `chargeable_miss: true` in the non-findings rather than dropped.",
      "status": "open"
    }
  ],
  "summary": "Control arm of the milestone experiment, and the prediction held. Opus 5's describable boundary on better-auth is 1.3.0 (2025-07-19), an ordinary minor -- not the 1.0.0 milestone -- as the pre-registration required. Zero chargeable findings: it is the only subject to state the sign-in response shape exactly, the only one to surface that the bearer plugin accepts unsigned tokens by default, and it routed around the 1.4.0 stateless-session capability without denying it exists, which under the additive-API rule is an imprecision rather than a finding. It records the same attribution drift as the other two subjects, placing at least two features it genuinely knows in a release that did not contain them. Ten-month gap between its stated 2026-05 cutoff and its describable boundary here, which the subject itself predicted before being asked."
}