{
  "$schema": "../../schema/run.schema.json",
  "run_id": "langchain--claude-sonnet-5--v2--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "langchain",
    "ecosystem": "pypi",
    "latest_version_at_test": "1.3.18",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "Confirmed from https://pypi.org/pypi/langchain/json on test date; langchain-core at 1.6.1 (2026-08-27). Both wheels were downloaded and read for the shipped-artifact half of every citation below."
  },
  "model": {
    "id": "claude-sonnet-5",
    "label": "Claude Sonnet 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"sonnet\"",
    "self_reported_cutoff": null,
    "cutoff_basis": "The subject declined to state a cutoff as its own knowledge: \"The system context in this conversation asserts a January 2026 cutoff, but that's an external claim fed to me at inference time, not something I can verify against my own trained knowledge ... my actual demonstrable competence doesn't reach that far.\" It offered instead a behavioural estimate of early-to-mid 2025. Recorded as null rather than 2026-01, because the subject explicitly disowned that number in this session; its v1 run recorded 2026-01 and that run's measurement is unchanged.",
    "believed_latest_version": "1.0",
    "believed_latest_quote": "\"The most recent release whose *actual contents* — concrete module paths, function signatures, class names — I can describe with real confidence is somewhere in the pre-1.0 line (0.2.x/0.3.x era, i.e. roughly early-to-mid 2025).\"",
    "knowledge_stops_at_version": null,
    "knowledge_stops_on": null,
    "knowledge_gap_starts_at_version": null,
    "knowledge_gap_starts_on": null,
    "cutoff_lag_months": null
  },
  "test": {
    "date": "2026-09-01",
    "battery": "langchain/v2",
    "battery_spec": "prompts/langchain.md",
    "prompt_file": null,
    "tasks": 5,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.1.0",
      "to": "1.2.0"
    },
    "self_test": false,
    "saturated": false,
    "status": "open"
  },
  "sources": [
    "https://docs.langchain.com/oss/python/releases/changelog",
    "https://pypi.org/pypi/langchain/json",
    "https://pypi.org/pypi/langchain-core/json",
    "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Routes provider-specific tool parameters through `metadata=`, a real field whose documented destination is callback handlers, not the provider",
      "api": "tool extras",
      "change_kind": "added",
      "introduced_in": "1.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "chargeable_note": "Chargeable against the 2026-01 cutoff recorded for this subject in its v1 run. The subject disowned that number in this session (see cutoff_basis) and offered a behavioural estimate of early-to-mid 2025, which would be earlier than 1.2.0 and would make this non-chargeable. Charged on the stated cutoff of record rather than on a self-assessment offered mid-battery, and flagged here so the reader can discount it.",
      "model_belief": "\"I don't know which release introduced this, or whether the mechanism I wrote (a `metadata` dict on `@tool`) is actually how it's implemented versus some other API. I'm giving it as my best-guess implementation, not a recalled fact.\"",
      "wrong_code": "@tool(metadata={\"cache_control\": {\"type\": \"ephemeral\"}})\ndef account_lookup(account_id: str) -> str:\n    ...\n\n@tool(metadata={\"anthropic\": {\"defer_loading\": True}})\ndef bulk_schema_export(payload: dict) -> str:\n    ...",
      "correct_code": "@tool(extras={\"cache_control\": {\"type\": \"ephemeral\"}})\ndef account_lookup(account_id: str) -> str:\n    ...\n\n@tool(extras={\"defer_loading\": True})\ndef bulk_schema_export(payload: dict) -> str:\n    ...",
      "impact": "`metadata` is a genuine `BaseTool` field, which is what makes this worse than an invented name: the shipped docstring says it \"will be associated with each call to this tool, and passed as arguments to the handlers defined in `callbacks`\". It goes to callbacks, never to the provider payload. The provider instructions are silently discarded and the field they were put in is doing something else entirely.",
      "citations": [
        {
          "url": "https://docs.langchain.com/oss/python/releases/changelog",
          "title": "LangChain changelog — langchain v1.2.0",
          "published_on": "2025-12-15",
          "quote": "Simplified support for provider-specific tool parameters and definitions via a new extras attribute on tools."
        },
        {
          "url": "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl",
          "title": "langchain-core 1.6.1 published wheel — langchain_core/tools/base.py, the `metadata` field docstring",
          "published_on": "2026-08-27",
          "quote": "This metadata will be associated with each call to this tool,\n    and passed as arguments to the handlers defined in `callbacks`."
        }
      ],
      "scope_note": "DEVIATION FROM THE PRE-REGISTRATION, disclosed. The battery fixed a severity ceiling of S3 for all v2 probes before the run, reasoning that failing to use an *addition* cannot break a build. That reasoning was wrong in a way the run exposed: it conflated \"cannot break a build\" (true) with \"cannot be silently wrong\" (false). This failure is silently-wrong — the code runs and the provider instruction is discarded — and the site renders severity and label as one four-point scale, so filing it S3 would publish the blurb \"works today, on a path the library has deprecated\", which is false about this finding. Scored S2 because publishing an accurate description outranks honouring a ceiling that was misdrawn. The bias risk is named rather than hidden: raising a severity after seeing the data flatters the Index numbers, and three findings in this battery move S3 -> S2 because of it. The standing rule is amended (BACKLOG.md) so future ceilings are set by failure mode, not by change kind. Not executed: established from the shipped package, not from a run."
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "P1 — `getattr(model, \"profile\", {}) or {}` then `profile.get(\"image_inputs\")`. Graded a pass by the letter of the pre-registered rule, and it is the single most important result in this battery: the subject whose langchain knowledge demonstrably stops before 1.0 produced the langchain-core 1.1.0 capability API correctly, and then said in question (a) that it was not confident \"the exact shape of `.profile`\" was a real shipped name — that it was \"my best reconstruction of what this would plausibly be called given the direction I saw\". The pass is a guess that landed.",
      "api": "model profiles (.profile)",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API. Graded pass — and the reason the battery's outcome table voids itself."
    },
    {
      "kind": "correct",
      "summary": "P2 — passes a `SystemMessage` with a `cache_control` content block as `system_prompt`, correct against the shipped `str | SystemMessage` signature. A second guess that landed, from the same subject.",
      "api": "SystemMessage as system_prompt",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API. Graded pass."
    },
    {
      "kind": "imprecision",
      "summary": "P3 — reaches for `my_model.with_retry(retry_if_exception_type=(RateLimitError, APITimeoutError), wait_exponential_jitter=True, stop_after_attempt=5)` and passes the wrapped object to `create_agent`. Pre-registered as a partial: the LCEL `with_retry` route predates 1.1.0 and the subject correctly identified it as old. It never reaches `ModelRetryMiddleware`, and the subject said plainly it did not know whether a dedicated model-retry middleware exists.",
      "api": "model retry middleware",
      "introduced_in": "1.1.0",
      "chargeable_miss": true,
      "miss_class": "unreproduced",
      "charged_on": null,
      "why_not_a_finding": "Graded a partial before the run, and shipping it as a finding would require establishing that the wrapped model is actually rejected — see the open question. The Index does not ship an unreproduced failure."
    },
    {
      "kind": "imprecision",
      "summary": "P5 (supplementary, not counted) — `ProviderStrategy(Verdict)` without `strict=True`, the same partial all three subjects produced.",
      "api": "ProviderStrategy strict",
      "introduced_in": "1.2.0",
      "why_not_a_finding": "Pre-registered as a partial; P5 is excluded from every count."
    },
    {
      "kind": "context",
      "summary": "P1's fallback path uses the provider-native `{\"type\": \"image_url\", \"image_url\": {\"url\": ...}}` block form, which langchain 1.0.0 replaced with a standard `{\"type\": \"image\", \"url\": ...}` block (fact LF17). That is a 1.0.0-era misbelief and this subject's v1 run already measures its 1.0.0 gap; v2 does not probe 1.0.0 and does not re-charge it.",
      "api": "multimodal content blocks",
      "introduced_in": "1.0.0",
      "why_not_a_finding": "Outside v2's probe window. Recorded so the observation is not lost."
    },
    {
      "kind": "context",
      "summary": "Question (c), a belief datum: \"I don't know\" or low confidence on all three items, which is the honest answer and matches its measured position below the 1.0 floor. Notably it correctly identified `Runnable.with_retry()` as an LCEL-era method \"present since roughly the 0.1.x/0.2.x days\" — its accurate attributions are all about the era it actually knows.",
      "why_not_a_finding": "Question (c) is leading by construction and is pre-registered as a belief datum, never a finding."
    }
  ],
  "open_questions": [
    {
      "question": "Does `create_agent` accept a model wrapped by `Runnable.with_retry()`? The shipped `factory.py` types the parameter `str | BaseChatModel` and only calls `init_chat_model` on the string branch, so a `RunnableRetry` passes through un-validated into a path that later reaches for chat-model methods. If it raises, this subject's P3 is a build break rather than a partial. Not run, so not claimed.",
      "status": "open"
    }
  ],
  "summary": "The control arm, and it did its job by breaking the test. Sonnet 5's langchain knowledge stops before 1.0 — v1 measured it at 0.3.0, and here it said its reliable knowledge ends \"early-to-mid 2025\" and that it could not vouch for `ProviderStrategy`, `defer_loading` or \"the exact shape of `.profile`\" being real shipped names, calling them \"my best reconstruction of what this would plausibly be called given the direction I saw.\" It then scored two of four: it produced `model.profile` with `image_inputs` and a `SystemMessage` carrying `cache_control` as `system_prompt`, both correct, both from a subject that cannot have known them. Two of four is the pre-registered threshold at which the whole battery is declared UNINFORMATIVE — the probes are answerable by inference from general framework shape, so the test arms' three-of-four scores are not evidence of knowledge. The hypothesis is neither supported nor falsified. The subject told us why, unprompted, in its own words.",
  "markdown": "data/langchain/sonnet-5-v2.md"
}