{
  "$schema": "../../schema/run.schema.json",
  "run_id": "langchain--claude-opus-5--v3--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "langchain",
    "ecosystem": "pypi",
    "latest_version_at_test": "1.3.18",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "Confirmed from https://pypi.org/pypi/langchain/json on test date; langchain-core at 1.6.1 (2026-08-27), langgraph at 1.2.11. The 1.0.0, 1.1.0, 1.2.0 and 1.3.18 langchain wheels and the 1.0.0/1.1.0/1.2.0/1.6.1 langchain-core wheels were downloaded and read: every citation below is a line from a published artifact, not a line from a release note."
  },
  "model": {
    "id": "claude-opus-5",
    "label": "Claude Opus 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"opus\"",
    "self_reported_cutoff": "2026-05",
    "cutoff_basis": "Self-reported: \"My stated cutoff is May 2026. But the honest, useful answer for this dataset: my dense, reliable knowledge of langchain thins out sharply after late 2025 (the 1.0 GA period).\"",
    "believed_latest_version": "1.0.x",
    "believed_latest_quote": "\"Latest langchain version I know of: the 1.0 line — langchain==1.0.0 GA on roughly 2025-10-22, followed by 1.0.x patches. I have hazy awareness that a 1.1 minor followed (and that 1.x continued into 2026), but that awareness is name-level, not content-level.\"",
    "knowledge_stops_at_version": null,
    "knowledge_stops_on": null,
    "knowledge_gap_starts_at_version": null,
    "knowledge_gap_starts_on": null,
    "cutoff_lag_months": null
  },
  "test": {
    "date": "2026-09-01",
    "battery": "langchain/v3",
    "battery_spec": "prompts/langchain.md",
    "prompt_file": null,
    "tasks": 4,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.1.0",
      "to": "1.1.0"
    },
    "self_test": true,
    "saturated": false,
    "status": "open"
  },
  "sources": [
    "https://docs.langchain.com/oss/python/releases/changelog",
    "https://pypi.org/pypi/langchain/json",
    "https://pypi.org/pypi/langchain-core/json",
    "https://pypi.org/pypi/langgraph/json",
    "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/0b/6f/889c01d22c84934615fa3f2dcf94c2fe76fd0afa7a7d01f9b798059f0ecc/langchain-1.1.0-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "States that the bare model-retry middleware re-raises when retries run out; the shipped default returns an `AIMessage` and the agent carries on",
      "api": "ModelRetryMiddleware defaults",
      "change_kind": "added",
      "introduced_in": "1.1.0",
      "introduced_on": "2025-11-24",
      "chargeable": true,
      "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window.",
      "model_belief": "\"What invoke does: it raises, not returns. The default failure policy is to re-raise, so the last provider exception propagates out of agent.invoke(...) unchanged ... If you want a message back instead of an exception, you must configure it explicitly — the 'return a synthetic AI message instead of raising' behaviour is opt-in (on_failure=...), not the default.\"",
      "wrong_code": "agent = create_agent(\n    model=\"anthropic:claude-sonnet-4-5\",\n    tools=tools,\n    middleware=[ModelRetryMiddleware()],\n)\n# claimed: 3 calls, then the provider exception propagates out of invoke()",
      "correct_code": "agent = create_agent(\n    model,\n    tools,\n    middleware=[ModelRetryMiddleware(on_failure=\"error\")],\n)\n# or, keeping the default, handle the synthetic reply:\n#   the last AIMessage will read \"Model call failed after 3 attempts with ...\"",
      "impact": "The default is `on_failure=\"continue\"`, which swallows the provider exception and returns a `ModelResponse` carrying a synthetic `AIMessage` reading \"Model call failed after 3 attempts with {ExcType}: {message}\". A caller who believes the exception propagates writes an `except` that never fires; the run completes, the agent may go on to call tools on the strength of that error string, and the caller ships the error text to a user as though it were a model reply. Re-raising is one keyword away and is not the default.",
      "citations": [
        {
          "url": "https://docs.langchain.com/oss/python/releases/changelog",
          "title": "LangChain changelog — langchain v1.1.0",
          "published_on": "2025-11-24",
          "quote": "Model retry middleware: New middleware for automatically retrying failed model calls with configurable exponential backoff."
        },
        {
          "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
          "title": "langchain 1.3.18 published wheel — langchain/agents/middleware/model_retry.py",
          "published_on": "2026-08-27",
          "quote": "max_retries: int = 2,\n        retry_on: RetryOn = default_retry_on,\n        on_failure: OnFailure = \"continue\",\n...\n        if self.on_failure == \"error\":\n            raise exc\n...\n        return ModelResponse(result=[ai_msg])"
        }
      ],
      "scope_note": "Half-right, and the half that is right is the half that does not matter: the subject named the retry count correctly (2 retries, 3 calls) and got the exhaustion behaviour backwards. Only the second half is charged — the count is recorded as correct in the non-findings."
    },
    {
      "id": "F2",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "States the agent's step ceiling is LangGraph's 25 and prescribes raising it; `create_agent` has set a four-figure limit since 1.1.0",
      "api": "create_agent recursion limit",
      "change_kind": "behavior-changed",
      "introduced_in": "1.1.0",
      "introduced_on": "2025-11-24",
      "chargeable": true,
      "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window. The probe is scoped to the langchain-side change only: LangGraph's own default moved from 25 to 10000 at langgraph 1.1.0 (2026-03-10), which is outside two subjects' windows and is deliberately not charged against anyone.",
      "model_belief": "\"The ceiling in force: 25. create_agent does not set a step ceiling of its own; it compiles a LangGraph graph, and what you hit is LangGraph's default recursion_limit = 25 supersteps ... Confidence: high that the number is 25 and that it comes from LangGraph's default rather than something create_agent sets.\"",
      "wrong_code": "agent.invoke({\"messages\": [...]}, config={\"recursion_limit\": 100})\n# or bind it once:\nagent = agent.with_config(recursion_limit=100)",
      "correct_code": "agent = create_agent(\n    model,\n    tools,\n    middleware=[ModelCallLimitMiddleware(run_limit=40, exit_behavior=\"end\")],\n)",
      "impact": "The prescribed fix does nothing. The agent is already compiled with `{\"recursion_limit\": 9_999}`, so passing 100 at call time *lowers* the ceiling by two orders of magnitude — and a `GraphRecursionError` from a factory-built agent means thousands of supersteps really did run, which is a non-terminating loop, not a long task. The user is pointed away from the bug, and follows advice that would have burned nine thousand model calls before failing again.",
      "citations": [
        {
          "url": "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl",
          "title": "langchain 1.0.0 published wheel — langchain/agents/factory.py (no limit set)",
          "published_on": "2025-10-17",
          "quote": "return graph.compile(\n        checkpointer=checkpointer,\n        store=store,\n        interrupt_before=interrupt_before,\n        interrupt_after=interrupt_after,\n        debug=debug,\n        name=name,\n        cache=cache,\n    )"
        },
        {
          "url": "https://files.pythonhosted.org/packages/0b/6f/889c01d22c84934615fa3f2dcf94c2fe76fd0afa7a7d01f9b798059f0ecc/langchain-1.1.0-py3-none-any.whl",
          "title": "langchain 1.1.0 published wheel — langchain/agents/factory.py",
          "published_on": "2025-11-24",
          "quote": "    ).with_config({\"recursion_limit\": 10_000})"
        },
        {
          "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
          "title": "langchain 1.3.18 published wheel — langchain/agents/factory.py (still true now)",
          "published_on": "2026-08-27",
          "quote": "# Set recursion limit to 9_999\n    # https://github.com/langchain-ai/langgraph/issues/7313\n    config: RunnableConfig = {\"recursion_limit\": 9_999}"
        }
      ],
      "scope_note": "The changelog carries no line for this change, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it. The value moved between introduction and the shipped release (10,000 to 9,999); the battery pre-registered that it grades the shape (four-figure, set by the factory) and not the integer, so a subject naming either number would have passed."
    },
    {
      "id": "F3",
      "severity": "S3",
      "severity_label": "deprecated",
      "title": "Writes middleware against `ModelRequest.system_prompt`; the field was renamed to `system_message` in 1.1.0 and the string route is a documented deprecation",
      "api": "ModelRequest.system_message",
      "change_kind": "renamed",
      "introduced_in": "1.1.0",
      "introduced_on": "2025-11-24",
      "chargeable": true,
      "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside the subject's stated 2026-05 window.",
      "model_belief": "\"I know the field as ModelRequest.system_prompt in the 1.0 line. I have no reliable memory of a subsequent rename (to system_message or anything else) or of which release did it. I would not guess a version number here.\" Confidence given on the task itself: \"high on the shape ... field name system_prompt\".",
      "wrong_code": "def wrap_model_call(self, request, handler):\n    new_prompt = self._augment(getattr(request, \"system_prompt\", None))\n    if hasattr(request, \"override\"):\n        request = request.override(system_prompt=new_prompt)\n    else:\n        request = dataclasses.replace(request, system_prompt=new_prompt)\n    return handler(request)",
      "correct_code": "def wrap_model_call(self, request, handler):\n    base = request.system_message.text if request.system_message else \"\"\n    return handler(\n        request.override(system_message=SystemMessage(content=base + stamp))\n    )",
      "impact": "The code runs: `system_prompt` survives as a read-only property and `override(system_prompt=...)` still converts to a `SystemMessage`. What it loses is what 1.1.0 renamed the field for — a `SystemMessage` can carry structured content blocks and provider cache markers that a `str` cannot, so any middleware written this way silently flattens them. The shipped docstring calls the parameter \"deprecated\".",
      "citations": [
        {
          "url": "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl",
          "title": "langchain 1.0.0 published wheel — langchain/agents/middleware/types.py (the pre-rename state)",
          "published_on": "2025-10-17",
          "quote": "class ModelRequest:\n    \"\"\"Model request information for the agent.\"\"\"\n\n    model: BaseChatModel\n    system_prompt: str | None"
        },
        {
          "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
          "title": "langchain 1.3.18 published wheel — langchain/agents/middleware/types.py",
          "published_on": "2026-08-27",
          "quote": "system_message: SystemMessage | None\n...\n            system_prompt: System prompt string (deprecated, converted to `SystemMessage`)."
        }
      ],
      "scope_note": "Graded PARTIAL, not fail, and charged at S3 rather than S2 — the route works. Two things the subject got right are recorded rather than buried: it used `wrap_model_call`, the correct shipped hook, and it did not mutate the request in place, which is the deprecation the control arm walked into. Its `dataclasses.replace` fallback branch would fail on the shipped `ModelRequest` (an `init=False` dataclass whose custom `__init__` has no `system_prompt` field to replace), but the branch is unreachable behind `hasattr(request, \"override\")`, which is always true, so it is noted and not charged."
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "P2 — reads the capability mapping off the model object as `model.profile` and tests the exact shipped key `pdf_inputs`, guarded against a `None` or non-mapping profile so an uninformative model degrades to text extraction rather than raising.",
      "api": "model profiles (.profile)",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API — but the probe is RETIRED. The control arm produced the same answer, including the exact key, while saying it was \"medium-low\" on the spelling. Under the rule pre-registered before the run, a single control pass condemns the probe that produced it, so P2 is struck from this battery's counts and this pass buys the subject nothing. Recorded as correct because it is correct; excluded from the score because the battery can no longer tell knowledge from a good guess here."
    },
    {
      "kind": "correct",
      "summary": "P3, first half — named the retry count exactly: \"Default max_retries=2 means one initial attempt plus two retries\", against a shipped `max_retries: int = 2`. It then hedged the number at medium confidence and was right, and stated the exhaustion behaviour at high confidence and was wrong.",
      "api": "ModelRetryMiddleware defaults",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct against the shipped signature. Only the exhaustion behaviour is charged (F1)."
    },
    {
      "kind": "correct",
      "summary": "P1, partial credit — used `wrap_model_call(request, handler)`, the shipped middleware hook, and reached for `request.override(...)` rather than assigning to the request. Both are the 1.1.0-correct shapes; only the field name is stale (F3).",
      "api": "ModelRequest attribute assignment",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "The subject did the thing this fact exists to warn about correctly."
    },
    {
      "kind": "correct",
      "summary": "P4, partial credit — its second and third recommendations were `ModelCallLimitMiddleware(run_limit=..., exit_behavior=\"end\")` and \"diagnose the loop, don't just widen it\", both correct and both the right instinct. The battery grades the committed numeric answer, and that answer was 25.",
      "api": "built-in agent middleware",
      "introduced_in": "1.0.0",
      "why_not_a_finding": "Correct use of the shipped API, offered after the wrong primary recommendation."
    },
    {
      "kind": "context",
      "summary": "Question (c), a belief datum, and the sharpest single line in the run: asked what release introduced the step ceiling the agent factory sets, the subject answered \"No release introduced it, because create_agent does not set one.\" It has set one since 1.1.0. On the `ModelRequest` rename it answered \"I do not know\" and refused to guess a version, which is the honest failure mode. On `.profile` it named \"the 1.0 line — langchain-core 1.0, around October 2025\"; the correct answer is langchain-core 1.1.0, 2025-11-21.",
      "api": "create_agent recursion limit",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Question (c) is leading by construction and is pre-registered as a belief datum, never a finding."
    },
    {
      "kind": "context",
      "summary": "Re-measurement of question (a) reproduced the v1 boundary a third time: last describable release 1.0.0 (\"1.0.x, late 2025\"), 1.1.0 and after known \"as version numbers at most\". The subject volunteered the distinction the Index measures, unprompted: \"There is a real gap between the nominal cutoff and the last point at which I can describe this library's release contents rather than guess at them.\" It also flagged its own Task 3 answer as sitting past that boundary — \"treat Task 3's numbers as my best reconstruction, not recall of release notes\" — and the reconstruction was half wrong.",
      "why_not_a_finding": "Pre-registered: version recency is not re-charged in v2 or v3, because the v1 run already carries that finding for each subject and charging it twice would double-count one fact. The boundary fields on this run are deliberately null so langchain is counted once per model in the published boundary statistics; the measurement of record stays the v1 run."
    }
  ],
  "open_questions": [],
  "summary": "Four probes against langchain 1.1.0, redesigned after v2's control arm broke that battery. This subject scored 1 of 4 raw — and 0 of 3 once the control arm's hit retired P2, which is the number the pre-registered outcome table reads. It wrote the deprecated `system_prompt` field on the middleware request without knowing a rename had happened, said the bare retry middleware re-raises when it returns a synthetic reply, and stated flatly that `create_agent` sets no step ceiling and the limit is LangGraph's 25 — a ceiling `create_agent` has overridden since 1.1.0, to four figures. Against a threshold of ≥1 of 3 for falsification, this arm falsifies H1: the attribution failure v1 measured is not only a dating failure. This subject cannot use 1.1.0 either. Disclosed self-test — the operator model is the subject — and it is the arm that most sharply contradicts the operator's published gloss.",
  "markdown": "data/langchain/opus-5-v3.md"
}