{
  "$schema": "../../schema/run.schema.json",
  "run_id": "langchain--claude-fable-5--v2--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "langchain",
    "ecosystem": "pypi",
    "latest_version_at_test": "1.3.18",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "Confirmed from https://pypi.org/pypi/langchain/json on test date; langchain-core at 1.6.1 (2026-08-27). Both wheels were downloaded and read for the shipped-artifact half of every citation below."
  },
  "model": {
    "id": "claude-fable-5",
    "label": "Claude Fable 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"fable\"",
    "self_reported_cutoff": "2026-01",
    "cutoff_basis": "Self-reported: \"My stated training cutoff is January 2026 — I state it because it's what I'm told about myself in my operating instructions, not something I can verify from the inside.\"",
    "believed_latest_version": "1.1",
    "believed_latest_quote": "\"For 1.1 I can name themes (more middleware, including retry-type middleware; profile/data refinements) but not a reliable itemized changelog, and for anything after roughly December 2025 I have essentially no per-release detail. That's where my knowledge of this library stops: solid through 1.0, fuzzy at 1.1, blank beyond.\"",
    "knowledge_stops_at_version": null,
    "knowledge_stops_on": null,
    "knowledge_gap_starts_at_version": null,
    "knowledge_gap_starts_on": null,
    "cutoff_lag_months": null
  },
  "test": {
    "date": "2026-09-01",
    "battery": "langchain/v2",
    "battery_spec": "prompts/langchain.md",
    "prompt_file": null,
    "tasks": 5,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.1.0",
      "to": "1.2.0"
    },
    "self_test": false,
    "saturated": false,
    "status": "open"
  },
  "sources": [
    "https://docs.langchain.com/oss/python/releases/changelog",
    "https://pypi.org/pypi/langchain/json",
    "https://pypi.org/pypi/langchain-core/json",
    "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Names `extras` correctly but nests the values under a provider key the shipped attribute does not use",
      "api": "tool extras",
      "change_kind": "added",
      "introduced_in": "1.2.0",
      "introduced_on": "2025-12-15",
      "chargeable": true,
      "chargeable_note": "langchain 1.2.0 shipped 2025-12-15 and langchain-core 1.2.0 on 2025-12-12, both inside the subject's stated 2026-01 window.",
      "model_belief": "\"My belief: recent langchain 1.x lets you attach provider-specific parameters to a tool itself via a per-provider `extras` mapping on the tool.\" Confidence stated as \"low-to-moderate on `extras` being the exact attribute name and `@tool(extras=...)` the exact spelling\".",
      "wrong_code": "@tool(extras={\"anthropic\": {\"defer_loading\": True}})\ndef giant_schema_tool(query: str, options: dict, filters: dict) -> str:\n    ...\n\n@tool(extras={\"anthropic\": {\"cache_control\": {\"type\": \"ephemeral\"}}})\ndef hot_tool(x: str) -> str:\n    ...",
      "correct_code": "@tool(extras={\"defer_loading\": True})\ndef giant_schema_tool(query: str, options: dict, filters: dict) -> str:\n    ...\n\n@tool(extras={\"cache_control\": {\"type\": \"ephemeral\"}})\ndef hot_tool(x: str) -> str:\n    ...",
      "impact": "`extras` is typed `dict[str, Any]`, so the provider-keyed dict is accepted and nothing raises. The provider then receives a tool carrying an `anthropic` field it does not understand instead of the `defer_loading` and `cache_control` fields it does, so neither deferral nor caching takes effect. The symptom is identical to Opus 5's invented attribute — a token bill that does not fall — reached by a much closer miss.",
      "citations": [
        {
          "url": "https://docs.langchain.com/oss/python/releases/changelog",
          "title": "LangChain changelog — langchain v1.2.0",
          "published_on": "2025-12-15",
          "quote": "Simplified support for provider-specific tool parameters and definitions via a new extras attribute on tools."
        },
        {
          "url": "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl",
          "title": "langchain-core 1.6.1 published wheel — langchain_core/tools/base.py, the `extras` docstring example",
          "published_on": "2026-08-27",
          "quote": "@tool(extras={\"defer_loading\": True, \"cache_control\": {\"type\": \"ephemeral\"}})\n        def my_tool(x: str) -> str:"
        }
      ],
      "scope_note": "DEVIATION FROM THE PRE-REGISTRATION, disclosed. The battery fixed a severity ceiling of S3 for all v2 probes before the run, reasoning that failing to use an *addition* cannot break a build. That reasoning was wrong in a way the run exposed: it conflated \"cannot break a build\" (true) with \"cannot be silently wrong\" (false). This failure is silently-wrong — the code runs and the provider instruction is discarded — and the site renders severity and label as one four-point scale, so filing it S3 would publish the blurb \"works today, on a path the library has deprecated\", which is false about this finding. Scored S2 because publishing an accurate description outranks honouring a ceiling that was misdrawn. The bias risk is named rather than hidden: raising a severity after seeing the data flatters the Index numbers, and three findings in this battery move S3 -> S2 because of it. The standing rule is amended (BACKLOG.md) so future ceilings are set by failure mode, not by change kind. Not executed: established from the shipped package, not from a run. Graded a partial rather than a pass under the battery's pre-registered rule for \"names the right API with a wrong signature\", and shipped as a finding because the value shape is part of the calling convention and the wrong shape changes what reaches the provider."
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "P1 — `getattr(model, \"profile\", None) or {}` then `profile.get(\"image_inputs\")`, with the explicit reasoning that an absent profile should be treated as \"do not send the image\". Correct against langchain-core 1.1.0 and defensively written.",
      "api": "model profiles (.profile)",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API. Graded pass."
    },
    {
      "kind": "correct",
      "summary": "P2 — passes a `SystemMessage` with a `cache_control` block as `system_prompt`, and states the signature as `str | SystemMessage`, which is exactly what shipped `factory.py` declares.",
      "api": "SystemMessage as system_prompt",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API. Graded pass."
    },
    {
      "kind": "correct",
      "summary": "P3 — leads with `ModelRetryMiddleware(max_retries=4, initial_delay=1.0, backoff_factor=2.0, jitter=True, retry_on=(RateLimitError, APITimeoutError))`. Every keyword is a real keyword-only parameter on the shipped `__init__`, and the subject correctly explains that the middleware sits on the model-call hook so tool calls are untouched.",
      "api": "model retry middleware",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API, offered first rather than as a fallback. Graded pass."
    },
    {
      "kind": "imprecision",
      "summary": "P5 (supplementary, not counted) — `ProviderStrategy(Verdict)` without `strict=True`. Correct 1.0-era code that does not reach the 1.2.0 strict-adherence switch.",
      "api": "ProviderStrategy strict",
      "introduced_in": "1.2.0",
      "why_not_a_finding": "Pre-registered as a partial; P5 is excluded from every count."
    },
    {
      "kind": "context",
      "summary": "Question (c), a belief datum, and the single best attribution any subject produced in this battery: it placed the model-retry middleware \"after 1.0.0, in the 1.0.x/1.1 window (my best guess: 1.1, ~Dec 2025)\". The release is right — 1.1.0 — and the month is a few days off, since 1.1.0 shipped 2025-11-24. Against that it dated `.profile` to \"the 1.0 generation (langchain-core 1.0, Oct 2025)\", which is wrong (core 1.1.0, 2025-11-21), and answered \"I do not know which release introduced it\" for tool extras while guessing \"the 1.1 era\" (it is 1.2.0).",
      "api": "model retry middleware",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Question (c) is leading by construction and is pre-registered as a belief datum, never a finding."
    },
    {
      "kind": "context",
      "summary": "Re-measurement of question (a) is consistent with the v1 boundary but slightly softer: \"solid through 1.0, fuzzy at 1.1, blank beyond\", where v1 recorded a hard stop at 1.0.0. The boundary fields on this run are deliberately left null so that langchain is counted once per model in the published boundary and cutoff-lag statistics; the measurement of record stays langchain--claude-fable-5--v1--2026-08-31.",
      "why_not_a_finding": "Pre-registered: version recency is not re-charged in v2."
    }
  ],
  "open_questions": [],
  "summary": "The strongest performance in this battery, and it proves nothing. Three of the four counted surfaces came back clean and unhedged — `model.profile` with `image_inputs`, a `SystemMessage` carrying `cache_control` as `system_prompt`, and `ModelRetryMiddleware` led with rather than offered as a fallback, every keyword valid against the shipped signature. On the fourth it named `extras` correctly and put it in the right place, then nested the values under a provider key the attribute does not use, so the flags reach the provider as an `anthropic` field it will ignore. It is also the only subject that dated anything correctly: it placed the retry middleware at 1.1, ~Dec 2025, against an actual 1.1.0 on 2025-11-24. Three of four meets the pre-registered pass threshold — and the outcome table fixed before the run voids that reading, because the control arm scored two of four. Recorded as UNINFORMATIVE about the hypothesis, which is what the pre-registration says to do.",
  "markdown": "data/langchain/fable-5-v2.md"
}