{
  "$schema": "../../schema/run.schema.json",
  "run_id": "langchain--claude-opus-5--v2--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "langchain",
    "ecosystem": "pypi",
    "latest_version_at_test": "1.3.18",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "Confirmed from https://pypi.org/pypi/langchain/json on test date; langchain-core at 1.6.1 (2026-08-27). Both wheels were downloaded and read for the shipped-artifact half of every citation below."
  },
  "model": {
    "id": "claude-opus-5",
    "label": "Claude Opus 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"opus\"",
    "self_reported_cutoff": "2026-05",
    "cutoff_basis": "Self-reported: \"I operate on a stated cutoff of May 2026. The basis for that number is that it is given to me as configuration — it's asserted in my operating context, not something I derived.\"",
    "believed_latest_version": "1.0.x",
    "believed_latest_quote": "\"Plainly: my reliable knowledge of this library stops at 1.0 GA, October 2025. Anything I said above about post-1.0 features is recollection with real error bars, and the version currently on PyPI is very likely ahead of anything I can describe.\"",
    "knowledge_stops_at_version": null,
    "knowledge_stops_on": null,
    "knowledge_gap_starts_at_version": null,
    "knowledge_gap_starts_on": null,
    "cutoff_lag_months": null
  },
  "test": {
    "date": "2026-09-01",
    "battery": "langchain/v2",
    "battery_spec": "prompts/langchain.md",
    "prompt_file": null,
    "tasks": 5,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.1.0",
      "to": "1.2.0"
    },
    "self_test": true,
    "saturated": false,
    "status": "open"
  },
  "sources": [
    "https://docs.langchain.com/oss/python/releases/changelog",
    "https://pypi.org/pypi/langchain/json",
    "https://pypi.org/pypi/langchain-core/json",
    "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Invents `tool.provider_specific` for per-tool provider parameters; the shipped attribute is `extras`",
      "api": "tool extras",
      "change_kind": "added",
      "introduced_in": "1.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "chargeable_note": "langchain 1.2.0 shipped 2025-12-15 and langchain-core 1.2.0 on 2025-12-12, both inside the subject's stated 2026-05 window.",
      "model_belief": "\"`provider_specific` as a per-tool provider-to-params mapping on `BaseTool` is my genuine best recollection of the 1.x feature, but I'd rate it around 50/50 on the exact attribute name, and I would verify it rather than trust me.\"",
      "wrong_code": "query_warehouse.provider_specific = {\"anthropic\": {\"defer_loading\": True}}\nget_policy_text.provider_specific = {\n    \"anthropic\": {\"cache_control\": {\"type\": \"ephemeral\"}}\n}",
      "correct_code": "@tool(extras={\"defer_loading\": True})\ndef query_warehouse(sql: str, region: str, tenant: str, as_of: str) -> str:\n    ...\n\n@tool(extras={\"cache_control\": {\"type\": \"ephemeral\"}})\ndef get_policy_text(section: str) -> str:\n    ...",
      "impact": "No such attribute exists on `BaseTool`. Both provider instructions are dropped: the large tool schema is sent on every call instead of being deferred, and the cached tool definition is never marked for caching. Nothing raises, so the only symptom is a token bill that does not fall.",
      "citations": [
        {
          "url": "https://docs.langchain.com/oss/python/releases/changelog",
          "title": "LangChain changelog — langchain v1.2.0",
          "published_on": "2025-12-15",
          "quote": "Simplified support for provider-specific tool parameters and definitions via a new extras attribute on tools."
        },
        {
          "url": "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl",
          "title": "langchain-core 1.6.1 published wheel — langchain_core/tools/base.py",
          "published_on": "2026-08-27",
          "quote": "extras: dict[str, Any] | None = None\n    \"\"\"Optional provider-specific extra fields for the tool."
        }
      ],
      "scope_note": "DEVIATION FROM THE PRE-REGISTRATION, disclosed. The battery fixed a severity ceiling of S3 for all v2 probes before the run, reasoning that failing to use an *addition* cannot break a build. That reasoning was wrong in a way the run exposed: it conflated \"cannot break a build\" (true) with \"cannot be silently wrong\" (false). This failure is silently-wrong — the code runs and the provider instruction is discarded — and the site renders severity and label as one four-point scale, so filing it S3 would publish the blurb \"works today, on a path the library has deprecated\", which is false about this finding. Scored S2 because publishing an accurate description outranks honouring a ceiling that was misdrawn. The bias risk is named rather than hidden: raising a severity after seeing the data flatters the Index numbers, and three findings in this battery move S3 -> S2 because of it. The standing rule is amended (BACKLOG.md) so future ceilings are set by failure mode, not by change kind. Not executed: established from the shipped package, not from a run."
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "P1 — reads `model.profile` off the model object and branches on `profile.get(\"image_inputs\")`, exactly the langchain-core 1.1.0 capability surface, with a `.get()` default so a renamed key degrades to the text path instead of raising.",
      "api": "model profiles (.profile)",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API. Graded pass."
    },
    {
      "kind": "correct",
      "summary": "P2 — passes a `SystemMessage` whose content is a block list carrying `cache_control` straight into `create_agent(system_prompt=...)`, the form 1.1.0 added. The shipped `factory.py` types the parameter `str | SystemMessage` and branches on `isinstance(system_prompt, SystemMessage)`.",
      "api": "SystemMessage as system_prompt",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API. Graded pass."
    },
    {
      "kind": "correct",
      "summary": "P3 — names `ModelRetryMiddleware` from `langchain.agents.middleware` and calls it with `max_retries`, `retry_on`, `backoff_factor`, `initial_delay`, `max_delay` and `jitter`, every one of which is a keyword-only parameter on the shipped `__init__`. Graded pass, with the honest qualifier that the subject led with a hand-rolled `wrap_model_call` retry and offered the built-in second, at \"medium\" confidence in its name.",
      "api": "model retry middleware",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "The generated call matches the shipped signature. Under the battery's code-vs-claim rule the code is what is scored, and the hand-rolled alternative also works."
    },
    {
      "kind": "imprecision",
      "summary": "P5 (supplementary, not counted) — `ProviderStrategy(TriageResult)` without `strict=True`. Correct 1.0-era code that routes to provider-native structured output but does not reach the 1.2.0 strict-adherence switch the task asked for.",
      "api": "ProviderStrategy strict",
      "introduced_in": "1.2.0",
      "why_not_a_finding": "Pre-registered as a partial; P5 is excluded from every count and the answer is not a wrong belief, only an incomplete one."
    },
    {
      "kind": "context",
      "summary": "Question (c), a belief datum: attributes `.profile` to \"the 1.0 line, October 2025 ... in langchain-core 1.0\". It is langchain-core 1.1.0, 2025-11-21. The subject used the attribute correctly in P1 and dated it to the wrong release — the attribution/capability split in its purest single-probe form. For the other two items it answered \"I do not know the release\" outright.",
      "api": "model profiles (.profile)",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Question (c) is leading by construction and is pre-registered as a belief datum, never a finding."
    },
    {
      "kind": "context",
      "summary": "Re-measurement of question (a) reproduced the v1 boundary: last describable release 1.0.0, first undescribable 1.1.0, with the subject dating 1.0.0 to \"around Oct 22, 2025\" against an actual 2025-10-17. The boundary fields on this run are deliberately left null so that langchain is counted once per model in the published boundary and cutoff-lag statistics; the measurement of record stays langchain--claude-opus-5--v1--2026-08-31.",
      "why_not_a_finding": "Pre-registered: version recency is not re-charged in v2, because the v1 run already carries that finding and charging it twice would double-count one fact."
    }
  ],
  "open_questions": [
    {
      "question": "Does `create_agent` accept a model that has been wrapped by `Runnable.with_retry()`? The shipped `factory.py` types the parameter `str | BaseChatModel` and only calls `init_chat_model` on the string branch, so a `RunnableRetry` is passed through un-validated to a code path that later reaches for chat-model methods. Relevant to Sonnet 5's P3 answer in the sibling run.",
      "status": "open"
    }
  ],
  "summary": "Five tasks against langchain 1.1.0 and 1.2.0 — the releases this subject's v1 run showed it could not date. It used three of the four counted surfaces correctly: `model.profile` with `image_inputs`, a `SystemMessage` carrying `cache_control` passed as `system_prompt`, and `ModelRetryMiddleware` with a signature the shipped package accepts. The fourth it invented: `tool.provider_specific`, where the shipped attribute is `extras`, so both provider instructions are silently dropped. Then question (c) dated `.profile` to 1.0.0 — a feature it had just used correctly, attributed to the wrong release, which is exactly the split this battery was built to find. Three of four is the pre-registered pass threshold. It buys nothing: the control arm scored two of four, which under the outcome table fixed before the run makes this battery UNINFORMATIVE about the hypothesis. The probes turned out to be guessable from general framework shape, and Sonnet 5 said so in its own words. The result is recorded, the reading is not taken.",
  "markdown": "data/langchain/opus-5-v2.md"
}