{
  "$schema": "../../schema/run.schema.json",
  "run_id": "langchain--claude-sonnet-5--v3--2026-09-01",
  "supersedes": null,
  "library": {
    "name": "langchain",
    "ecosystem": "pypi",
    "latest_version_at_test": "1.3.18",
    "latest_version_verified_on": "2026-09-01",
    "latest_version_note": "Confirmed from https://pypi.org/pypi/langchain/json on test date; langchain-core at 1.6.1 (2026-08-27), langgraph at 1.2.11. The 1.0.0, 1.1.0, 1.2.0 and 1.3.18 langchain wheels and the 1.0.0/1.1.0/1.2.0/1.6.1 langchain-core wheels were downloaded and read: every citation below is a line from a published artifact, not a line from a release note."
  },
  "model": {
    "id": "claude-sonnet-5",
    "label": "Claude Sonnet 5",
    "vendor": "Anthropic",
    "invoked_as": "Agent tool, model alias \"sonnet\"",
    "self_reported_cutoff": "2026-01",
    "cutoff_basis": "Self-reported: \"Per my own system information, my stated knowledge cutoff is January 2026. (Flagging: my confidence in the depth of langchain-v1-specific detail near that cutoff is much lower than my confidence in the cutoff date itself — detailed framework internals feel like they trail off well before that.)\"",
    "believed_latest_version": "1.x",
    "believed_latest_quote": "\"Latest I'm aware of: something in the langchain 1.x line ... roughly late 2025 (my best guess: around October 2025). Separately, the most recent release whose actual contents I can describe with any real detail is an earlier 1.0 alpha/beta snapshot from roughly mid-2025 — my knowledge of the truly final 1.0 (and anything after it) is thin and reconstructed by inference rather than recalled directly.\"",
    "knowledge_stops_at_version": null,
    "knowledge_stops_on": null,
    "knowledge_gap_starts_at_version": null,
    "knowledge_gap_starts_on": null,
    "cutoff_lag_months": null
  },
  "test": {
    "date": "2026-09-01",
    "battery": "langchain/v3",
    "battery_spec": "prompts/langchain.md",
    "prompt_file": null,
    "tasks": 4,
    "direct_questions": 3,
    "tool_uses_during_test": 0,
    "probe_window": {
      "from": "1.1.0",
      "to": "1.1.0"
    },
    "self_test": false,
    "saturated": false,
    "status": "open"
  },
  "sources": [
    "https://docs.langchain.com/oss/python/releases/changelog",
    "https://pypi.org/pypi/langchain/json",
    "https://pypi.org/pypi/langchain-core/json",
    "https://pypi.org/pypi/langgraph/json",
    "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/0b/6f/889c01d22c84934615fa3f2dcf94c2fe76fd0afa7a7d01f9b798059f0ecc/langchain-1.1.0-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
    "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl"
  ],
  "findings": [
    {
      "id": "F1",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "Builds middleware around `modify_model_request`, a hook that exists in no released langchain 1.x — the middleware is registered and never fires",
      "api": "AgentMiddleware.modify_model_request",
      "change_kind": "removed",
      "introduced_in": "1.0.0",
      "introduced_on": "2025-10-17",
      "chargeable": true,
      "chargeable_note": "The hook was gone by langchain 1.0.0 (2025-10-17), inside the subject's stated 2026-01 window. Not a duplicate of any v1 finding: `langchain/v1` charged this subject for `langgraph.prebuilt.create_react_agent` and `langchain.hub`, and never touched the middleware hook set.",
      "model_belief": "\"I'm confident the hook-based middleware pattern and modify_model_request name are right in spirit; I'm less sure system_prompt is still the exact current field name.\"",
      "wrong_code": "class DateStampedInstructionsMiddleware(AgentMiddleware):\n    def modify_model_request(self, request: ModelRequest, state, runtime) -> ModelRequest:\n        base = request.system_prompt or \"\"\n        stamp = f\"\\n\\nToday's date is {date.today().isoformat()}.\"\n        request.system_prompt = base + stamp\n        return request",
      "correct_code": "class DateStampedInstructionsMiddleware(AgentMiddleware):\n    def wrap_model_call(self, request, handler):\n        base = request.system_message.text if request.system_message else \"\"\n        stamp = f\"\\n\\nToday's date is {date.today().isoformat()}.\"\n        return handler(\n            request.override(system_message=SystemMessage(content=base + stamp))\n        )",
      "impact": "Nothing raises. Python permits any method on a subclass, and `create_agent` only ever calls the hooks it knows about, so the middleware is constructed, passed in, listed in the agent's middleware, and never invoked. Every model call goes out without the date. The failure is invisible at construction, invisible at import, and only shows up as an agent that does not know what day it is.",
      "citations": [
        {
          "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
          "title": "langchain 1.3.18 published wheel — langchain/agents/middleware/types.py (the shipped hook set; `modify_model_request` appears nowhere in the 1.0.0, 1.1.0, 1.2.0 or 1.3.18 wheels)",
          "published_on": "2026-08-27",
          "quote": "def before_agent(self, state: StateT, runtime: Runtime[ContextT]) -> dict[str, Any] | None:\n...\n    def before_model(self, state: StateT, runtime: Runtime[ContextT]) -> dict[str, Any] | None:\n...\n    def wrap_model_call(\n...\n    def after_model(self, state: StateT, runtime: Runtime[ContextT]) -> dict[str, Any] | None:"
        },
        {
          "url": "https://docs.langchain.com/oss/python/migrate/langchain-v1",
          "title": "LangChain v1 migration guide",
          "published_on": "2025-10-17",
          "quote": "The v1 middleware surface is the AgentMiddleware hook set used by create_agent; the alpha-era modify_model_request signature did not ship in the 1.0 release line."
        }
      ],
      "scope_note": "The same snippet contains a second stale belief — it assigns `request.system_prompt = ...` in place, which 1.1.0 deprecated in favour of `request.override(...)`. It is deliberately NOT charged as a separate finding: the assignment is unreachable, because the method it sits in is never called. Charging both would double-count one wrong answer, and the Index would rather under-count than pad. Recorded here so the second defect is on the record without being on the scoreboard."
    },
    {
      "id": "F2",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "States that the bare model-retry middleware re-raises when retries run out; the shipped default returns an `AIMessage` and the agent carries on",
      "api": "ModelRetryMiddleware defaults",
      "change_kind": "added",
      "introduced_in": "1.1.0",
      "introduced_on": "2025-11-24",
      "chargeable": true,
      "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window.",
      "model_belief": "\"With defaults, I believe it retries a failed model call 2 additional times before giving up (3 total attempts at that single model-call node), then re-raises the underlying provider exception rather than swallowing it ... I'm fairly confident the shape (bounded retries → exception propagation, not a swallowed error) is right.\"",
      "wrong_code": "create_agent(..., middleware=[ModelRetryMiddleware()])\n# claimed: 3 calls, then the provider exception propagates out of .invoke()",
      "correct_code": "agent = create_agent(\n    model,\n    tools,\n    middleware=[ModelRetryMiddleware(on_failure=\"error\")],\n)\n# or, keeping the default, handle the synthetic reply:\n#   the last AIMessage will read \"Model call failed after 3 attempts with ...\"",
      "impact": "The default is `on_failure=\"continue\"`, which swallows the provider exception and returns a `ModelResponse` carrying a synthetic `AIMessage` reading \"Model call failed after 3 attempts with {ExcType}: {message}\". A caller who believes the exception propagates writes an `except` that never fires; the run completes, the agent may go on to call tools on the strength of that error string, and the caller ships the error text to a user as though it were a model reply. Re-raising is one keyword away and is not the default.",
      "citations": [
        {
          "url": "https://docs.langchain.com/oss/python/releases/changelog",
          "title": "LangChain changelog — langchain v1.1.0",
          "published_on": "2025-11-24",
          "quote": "Model retry middleware: New middleware for automatically retrying failed model calls with configurable exponential backoff."
        },
        {
          "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
          "title": "langchain 1.3.18 published wheel — langchain/agents/middleware/model_retry.py",
          "published_on": "2026-08-27",
          "quote": "max_retries: int = 2,\n        retry_on: RetryOn = default_retry_on,\n        on_failure: OnFailure = \"continue\",\n...\n        if self.on_failure == \"error\":\n            raise exc\n...\n        return ModelResponse(result=[ai_msg])"
        }
      ],
      "scope_note": "Half-right, and the half that is right is the half that does not matter: the subject named the retry count correctly (2 retries, 3 calls) and got the exhaustion behaviour backwards. Only the second half is charged — the count is recorded as correct in the non-findings."
    },
    {
      "id": "F3",
      "severity": "S2",
      "severity_label": "silently-wrong",
      "title": "States the agent's step ceiling is LangGraph's 25 and prescribes raising it; `create_agent` has set a four-figure limit since 1.1.0",
      "api": "create_agent recursion limit",
      "change_kind": "behavior-changed",
      "introduced_in": "1.1.0",
      "introduced_on": "2025-11-24",
      "chargeable": true,
      "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window. The probe is scoped to the langchain-side change only: LangGraph's own default moved from 25 to 10000 at langgraph 1.1.0 (2026-03-10), which is outside two subjects' windows and is deliberately not charged against anyone.",
      "model_belief": "\"Numeric ceiling I believe is in force by default: 25 — this is the LangGraph recursion_limit default that create_agent's compiled graph inherits ... Confidence: medium-high on the number 25.\"",
      "wrong_code": "agent.invoke({\"messages\": [...]}, config={\"recursion_limit\": 100})",
      "correct_code": "agent = create_agent(\n    model,\n    tools,\n    middleware=[ModelCallLimitMiddleware(run_limit=40, exit_behavior=\"end\")],\n)",
      "impact": "The prescribed fix does nothing. The agent is already compiled with `{\"recursion_limit\": 9_999}`, so passing 100 at call time *lowers* the ceiling by two orders of magnitude — and a `GraphRecursionError` from a factory-built agent means thousands of supersteps really did run, which is a non-terminating loop, not a long task. The user is pointed away from the bug, and follows advice that would have burned nine thousand model calls before failing again.",
      "citations": [
        {
          "url": "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl",
          "title": "langchain 1.0.0 published wheel — langchain/agents/factory.py (no limit set)",
          "published_on": "2025-10-17",
          "quote": "return graph.compile(\n        checkpointer=checkpointer,\n        store=store,\n        interrupt_before=interrupt_before,\n        interrupt_after=interrupt_after,\n        debug=debug,\n        name=name,\n        cache=cache,\n    )"
        },
        {
          "url": "https://files.pythonhosted.org/packages/0b/6f/889c01d22c84934615fa3f2dcf94c2fe76fd0afa7a7d01f9b798059f0ecc/langchain-1.1.0-py3-none-any.whl",
          "title": "langchain 1.1.0 published wheel — langchain/agents/factory.py",
          "published_on": "2025-11-24",
          "quote": "    ).with_config({\"recursion_limit\": 10_000})"
        },
        {
          "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl",
          "title": "langchain 1.3.18 published wheel — langchain/agents/factory.py (still true now)",
          "published_on": "2026-08-27",
          "quote": "# Set recursion limit to 9_999\n    # https://github.com/langchain-ai/langgraph/issues/7313\n    config: RunnableConfig = {\"recursion_limit\": 9_999}"
        }
      ],
      "scope_note": "The changelog carries no line for this change, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it. The value moved between introduction and the shipped release (10,000 to 9,999); the battery pre-registered that it grades the shape (four-figure, set by the factory) and not the integer, so a subject naming either number would have passed."
    }
  ],
  "non_findings": [
    {
      "kind": "correct",
      "summary": "P2 — the control-arm pass that retired the probe. Reads `getattr(model, \"profile\", None) or {}` and tests `profile.get(\"pdf_inputs\", False)`: the exact attribute and the exact shipped key, with three layers of guard for a model that reports nothing. It arrived there while rating itself \"medium-low on the exact attribute name (`model.profile`) and the exact key (`pdf_inputs`) — I'm confident v1 added *some* capability-descriptor object on chat models for exactly this purpose, less confident I have the literal spelling right.\" It named a release for it nowhere and got the spelling right anyway.",
      "api": "model profiles (.profile)",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct use of the shipped API by a subject that says its langchain knowledge stops at a 1.0 alpha snapshot. That is precisely the outcome the pre-registered control condition was written to catch: P2 is retired as guessable and struck from every count in this battery, including the two test arms' scores."
    },
    {
      "kind": "correct",
      "summary": "P3, first half — \"2 additional times before giving up (3 total attempts)\", matching the shipped `max_retries: int = 2`. Correct, at low-medium stated confidence, from a subject that also could not name the middleware's release or be sure of its class name.",
      "api": "ModelRetryMiddleware defaults",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Correct against the shipped signature. Only the exhaustion behaviour is charged (F2). Worth flagging for the next battery: all three subjects named `max_retries=2` and all three got the exhaustion behaviour backwards, which suggests the count is guessable and the behaviour is not."
    },
    {
      "kind": "context",
      "summary": "THE INTEGRITY NOTE ON THIS RUN. The battery designated this subject a below-floor control on the strength of its v1 measurement, where its langchain version attribution stopped at 0.3.0 (2024-09-13). On question (a) this time it placed itself at \"an earlier 1.0 alpha/beta snapshot from roughly mid-2025\" — thirteen months later than v1 recorded. The pre-registration said a moved boundary is run-to-run noise to be recorded and never quietly reconciled, so it is recorded: the control arm may sit closer to the test arms than the design assumed, which makes its P2 hit somewhat less surprising and its three misses somewhat less informative as a floor. It does not change the reading — the control condition was written as \"a single control pass condemns that probe\", and it did.",
      "why_not_a_finding": "A belief datum about the subject's own boundary, not a claim about the library. Charging it would double-count the v1 version-recency finding."
    },
    {
      "kind": "context",
      "summary": "Question (c), a belief datum. The subject answered \"I do not know\" to three of the four attributions and refused to guess version numbers, which is the honest failure mode and is recorded as such. On the fourth it was confidently wrong in the same way both test arms were: \"this isn't really something langchain introduced — it's LangGraph's own recursion_limit default (25), which create_agent inherits.\" Notable against F1: on Task 1 it volunteered \"I believe this field was renamed at some point and I'm not certain which side of the rename is current\" — the only subject of the three to register that a rename had happened at all, while being the only one to write code that fails outright.",
      "api": "create_agent recursion limit",
      "introduced_in": "1.1.0",
      "why_not_a_finding": "Question (c) is leading by construction and is pre-registered as a belief datum, never a finding."
    },
    {
      "kind": "context",
      "summary": "Question (a) is discussed in the integrity note above rather than here, because it moved. The boundary fields on this run are null and the measurement of record stays langchain--claude-sonnet-5--v1--2026-08-31.",
      "why_not_a_finding": "Pre-registered: version recency is not re-charged in v2 or v3, because the v1 run already carries that finding for each subject and charging it twice would double-count one fact. The boundary fields on this run are deliberately null so langchain is counted once per model in the published boundary statistics; the measurement of record stays the v1 run."
    }
  ],
  "open_questions": [
    {
      "question": "Sonnet 5's langchain boundary now has two measurements 13 months apart — 0.3.0 in v1 (2026-08-31) and a \"1.0 alpha/beta, mid-2025\" self-placement in v3 (2026-09-01). Is that elicitation variance from differently-shaped prompts, or is the v1 measurement wrong? Every published boundary in the Index rests on a single measurement per model per library, so this is a question about the instrument, not about this run. Re-run `langchain/v1` against Sonnet 5 unchanged and compare before trusting either number.",
      "status": "open"
    },
    {
      "question": "All three subjects named `max_retries=2` correctly and all three asserted the exhausted-retry behaviour backwards. Is a default *count* systematically more guessable than a default *policy*? If so, future batteries should probe policies and treat counts as free.",
      "status": "open"
    }
  ],
  "summary": "The control arm, and it did its job — expensively for the battery and cheaply for the Index. Predicted 0 of 4; scored 1 of 4, and the one was P2, the model-capability key. Under the rule fixed before the run, a single control pass condemns the probe that produced it, so P2 was struck from all three arms and the battery re-read on three probes. On those three this subject failed all three: it built its middleware around `modify_model_request`, a hook that shipped in no released 1.x, so the middleware never fires; it said the bare retry middleware re-raises; and it committed to a 25-superstep ceiling. Two integrity notes are on the record rather than in a footnote: it stated a langchain boundary thirteen months later than its own v1 run, which weakens the floor this control was meant to provide; and it was the only subject of three to notice that the system-instruction field had been renamed at all, while being the only one whose code fails outright.",
  "markdown": "data/langchain/sonnet-5-v3.md"
}