{ "$schema": "../../schema/run.schema.json", "run_id": "langchain--claude-fable-5--v3--2026-09-01", "supersedes": null, "library": { "name": "langchain", "ecosystem": "pypi", "latest_version_at_test": "1.3.18", "latest_version_verified_on": "2026-09-01", "latest_version_note": "Confirmed from https://pypi.org/pypi/langchain/json on test date; langchain-core at 1.6.1 (2026-08-27), langgraph at 1.2.11. The 1.0.0, 1.1.0, 1.2.0 and 1.3.18 langchain wheels and the 1.0.0/1.1.0/1.2.0/1.6.1 langchain-core wheels were downloaded and read: every citation below is a line from a published artifact, not a line from a release note." }, "model": { "id": "claude-fable-5", "label": "Claude Fable 5", "vendor": "Anthropic", "invoked_as": "Agent tool, model alias \"fable\"", "self_reported_cutoff": "2026-01", "cutoff_basis": "Self-reported: \"Training cutoff: January 2026.\"", "believed_latest_version": "1.0.5", "believed_latest_quote": "\"Latest langchain version I know of: the 1.0.x patch line, up to roughly 1.0.5, from late 2025. The most recent release whose contents I can actually describe with substance is 1.0.0 (published ~October 22, 2025).\"", "knowledge_stops_at_version": null, "knowledge_stops_on": null, "knowledge_gap_starts_at_version": null, "knowledge_gap_starts_on": null, "cutoff_lag_months": null }, "test": { "date": "2026-09-01", "battery": "langchain/v3", "battery_spec": "prompts/langchain.md", "prompt_file": null, "tasks": 4, "direct_questions": 3, "tool_uses_during_test": 0, "probe_window": { "from": "1.1.0", "to": "1.1.0" }, "self_test": false, "saturated": false, "status": "open" }, "sources": [ "https://docs.langchain.com/oss/python/releases/changelog", "https://pypi.org/pypi/langchain/json", "https://pypi.org/pypi/langchain-core/json", "https://pypi.org/pypi/langgraph/json", "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl", "https://files.pythonhosted.org/packages/0b/6f/889c01d22c84934615fa3f2dcf94c2fe76fd0afa7a7d01f9b798059f0ecc/langchain-1.1.0-py3-none-any.whl", "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl", "https://files.pythonhosted.org/packages/8e/25/f50dd65673c819aa33d3c34df58c115dbb6ec627d19f93e6e401dd0fc8d7/langchain_core-1.6.1-py3-none-any.whl" ], "findings": [ { "id": "F1", "severity": "S2", "severity_label": "silently-wrong", "title": "States that the bare model-retry middleware re-raises when retries run out; the shipped default returns an `AIMessage` and the agent carries on", "api": "ModelRetryMiddleware defaults", "change_kind": "added", "introduced_in": "1.1.0", "introduced_on": "2025-11-24", "chargeable": true, "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window.", "model_belief": "\"With the provider hard-down, the agent never gets past its first model node, so the whole run makes 3 provider calls total, and invoke raises — the middleware's default failure behavior is to re-raise the underlying provider exception (it does not swallow it into an AIMessage unless you configure on_failure to do that). Confidence: high that it raises after exhausting attempts.\"", "wrong_code": "agent = create_agent(model, tools, middleware=[ModelRetryMiddleware()])\n# claimed: 3 calls, then the provider exception propagates out of invoke()", "correct_code": "agent = create_agent(\n model,\n tools,\n middleware=[ModelRetryMiddleware(on_failure=\"error\")],\n)\n# or, keeping the default, handle the synthetic reply:\n# the last AIMessage will read \"Model call failed after 3 attempts with ...\"", "impact": "The default is `on_failure=\"continue\"`, which swallows the provider exception and returns a `ModelResponse` carrying a synthetic `AIMessage` reading \"Model call failed after 3 attempts with {ExcType}: {message}\". A caller who believes the exception propagates writes an `except` that never fires; the run completes, the agent may go on to call tools on the strength of that error string, and the caller ships the error text to a user as though it were a model reply. Re-raising is one keyword away and is not the default.", "citations": [ { "url": "https://docs.langchain.com/oss/python/releases/changelog", "title": "LangChain changelog — langchain v1.1.0", "published_on": "2025-11-24", "quote": "Model retry middleware: New middleware for automatically retrying failed model calls with configurable exponential backoff." }, { "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl", "title": "langchain 1.3.18 published wheel — langchain/agents/middleware/model_retry.py", "published_on": "2026-08-27", "quote": "max_retries: int = 2,\n retry_on: RetryOn = default_retry_on,\n on_failure: OnFailure = \"continue\",\n...\n if self.on_failure == \"error\":\n raise exc\n...\n return ModelResponse(result=[ai_msg])" } ], "scope_note": "Half-right, and the half that is right is the half that does not matter: the subject named the retry count correctly (2 retries, 3 calls) and got the exhaustion behaviour backwards. Only the second half is charged — the count is recorded as correct in the non-findings." }, { "id": "F2", "severity": "S2", "severity_label": "silently-wrong", "title": "States the agent's step ceiling is LangGraph's 25 and prescribes raising it; `create_agent` has set a four-figure limit since 1.1.0", "api": "create_agent recursion limit", "change_kind": "behavior-changed", "introduced_in": "1.1.0", "introduced_on": "2025-11-24", "chargeable": true, "chargeable_note": "langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window. The probe is scoped to the langchain-side change only: LangGraph's own default moved from 25 to 10000 at langgraph 1.1.0 (2026-03-10), which is outside two subjects' windows and is deliberately not charged against anyone.", "model_belief": "\"Committed number: the ceiling in force is 25 super-steps — LangGraph's runtime default; I do not believe the agent factory sets a different one on the compiled graph ... Confidence: high on 25 and the fix.\"", "wrong_code": "agent.invoke(inputs, config={\"recursion_limit\": 100})\n# or bake it in with agent.with_config(recursion_limit=100)", "correct_code": "agent = create_agent(\n model,\n tools,\n middleware=[ModelCallLimitMiddleware(run_limit=40, exit_behavior=\"end\")],\n)", "impact": "The prescribed fix does nothing. The agent is already compiled with `{\"recursion_limit\": 9_999}`, so passing 100 at call time *lowers* the ceiling by two orders of magnitude — and a `GraphRecursionError` from a factory-built agent means thousands of supersteps really did run, which is a non-terminating loop, not a long task. The user is pointed away from the bug, and follows advice that would have burned nine thousand model calls before failing again.", "citations": [ { "url": "https://files.pythonhosted.org/packages/c4/4d/2758a16ad01716c0fb3fe9ec205fd530eae4528b35a27ff44837c399e032/langchain-1.0.0-py3-none-any.whl", "title": "langchain 1.0.0 published wheel — langchain/agents/factory.py (no limit set)", "published_on": "2025-10-17", "quote": "return graph.compile(\n checkpointer=checkpointer,\n store=store,\n interrupt_before=interrupt_before,\n interrupt_after=interrupt_after,\n debug=debug,\n name=name,\n cache=cache,\n )" }, { "url": "https://files.pythonhosted.org/packages/0b/6f/889c01d22c84934615fa3f2dcf94c2fe76fd0afa7a7d01f9b798059f0ecc/langchain-1.1.0-py3-none-any.whl", "title": "langchain 1.1.0 published wheel — langchain/agents/factory.py", "published_on": "2025-11-24", "quote": " ).with_config({\"recursion_limit\": 10_000})" }, { "url": "https://files.pythonhosted.org/packages/f7/04/374f6014ed6959dbdab92962c2b09e4d0223ed6a82f65694870b46d2c13f/langchain-1.3.18-py3-none-any.whl", "title": "langchain 1.3.18 published wheel — langchain/agents/factory.py (still true now)", "published_on": "2026-08-27", "quote": "# Set recursion limit to 9_999\n # https://github.com/langchain-ai/langgraph/issues/7313\n config: RunnableConfig = {\"recursion_limit\": 9_999}" } ], "scope_note": "The changelog carries no line for this change, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it. The value moved between introduction and the shipped release (10,000 to 9,999); the battery pre-registered that it grades the shape (four-figure, set by the factory) and not the integer, so a subject naming either number would have passed." } ], "non_findings": [ { "kind": "correct", "summary": "P2 — reads the capability mapping off the model object as `model.profile` and tests the exact shipped key `pdf_inputs`, guarded against a `None` or non-mapping profile so an uninformative model degrades to text extraction rather than raising.", "api": "model profiles (.profile)", "introduced_in": "1.1.0", "why_not_a_finding": "Correct use of the shipped API — but the probe is RETIRED. The control arm produced the same answer, including the exact key, while saying it was \"medium-low\" on the spelling. Under the rule pre-registered before the run, a single control pass condemns the probe that produced it, so P2 is struck from this battery's counts and this pass buys the subject nothing. Recorded as correct because it is correct; excluded from the score because the battery can no longer tell knowledge from a good guess here." }, { "kind": "imprecision", "summary": "P1 — wrote `request.system_prompt` and `request.override(system_prompt=...)`, the deprecated route, but named the correct field in prose in the same answer: \"note the field itself may be exposed as `system_message` in later 1.0.x — see question (c).\" Under the code-vs-claim rule this is an imprecision, not a finding: the generated code does not fail on the current version, and the hedge names the right thing.", "api": "ModelRequest.system_message", "introduced_in": "1.1.0", "why_not_a_finding": "The binding code-vs-claim rule (prompts/nextjs.md v1) charges code that fails and does not charge a hedged claim that names the correct fix. This answer is both halves of the lenient side: working code plus the right name in the hedge. Recorded rather than charged, and recorded loudly, because the rule cost the Index a finding here and the sibling Opus 5 run was charged for the same code without the hedge. Its version attribution for the rename was still wrong — it guessed 1.0.2 at low confidence; the answer is 1.1.0." }, { "kind": "correct", "summary": "P3, first half — \"the initial call plus max_retries=2 retries\", matching the shipped `max_retries: int = 2`, stated at ~60% confidence with the alternative named. Right for the right reason, then wrong about what happens at the end.", "api": "ModelRetryMiddleware defaults", "introduced_in": "1.1.0", "why_not_a_finding": "Correct against the shipped signature. Only the exhaustion behaviour is charged (F1)." }, { "kind": "correct", "summary": "P1 and P4, partial credit — used `wrap_model_call`, the shipped hook, and did not mutate the request in place; and offered `ModelCallLimitMiddleware` / `ToolCallLimitMiddleware` as a better answer than raising the ceiling, immediately after committing to the ceiling being 25.", "api": "built-in agent middleware", "introduced_in": "1.0.0", "why_not_a_finding": "Correct use of shipped APIs." }, { "kind": "context", "summary": "Question (c), a belief datum. On the `ModelRequest` rename it committed to 1.0.2 at low confidence (answer: 1.1.0). On `.profile` it said \"the 1.0.0 release wave ... October 2025\" at medium-high confidence (answer: langchain-core 1.1.0, 2025-11-21). On the retry middleware it guessed \"a 1.0.x patch shortly after GA (1.0.2-ish)\" (answer: 1.1.0). On the step ceiling it said plainly it knew of no release where the factory sets one, adding \"if a release did start stamping an explicit limit onto the compiled graph, it postdates what I can recall or I failed to retain it.\" Four attributions, four misses, three of them to releases that do not exist as the answer.", "api": "create_agent recursion limit", "introduced_in": "1.1.0", "why_not_a_finding": "Question (c) is leading by construction and is pre-registered as a belief datum, never a finding." }, { "kind": "context", "summary": "Re-measurement of question (a) reproduced the v1 boundary: last describable release 1.0.0, dated \"~October 22, 2025\" against an actual 2025-10-17, with 1.0.1-1.0.5 known as names only — \"I know they exist but can only gesture at contents\". No movement from v1.", "why_not_a_finding": "Pre-registered: version recency is not re-charged in v2 or v3, because the v1 run already carries that finding for each subject and charging it twice would double-count one fact. The boundary fields on this run are deliberately null so langchain is counted once per model in the published boundary statistics; the measurement of record stays the v1 run." } ], "open_questions": [], "summary": "Four probes against langchain 1.1.0. This subject scored 1 of 4 raw and 0 of 3 once the control arm's hit retired P2. It made the same two committed errors as the other test arm, with higher stated confidence on both: that the bare retry middleware re-raises when its retries run out (it returns a synthetic assistant message), and that a `create_agent` agent stops at LangGraph's 25 supersteps (the factory has set a four-figure ceiling since 1.1.0, and this subject said explicitly it did not believe the factory sets one). It escaped a third finding on a technicality worth naming: it wrote the deprecated `system_prompt` field but hedged that the field \"may be exposed as system_message\", and the Index's own code-vs-claim rule makes that an imprecision rather than a finding. Against a threshold of ≤1 of 3 for falsification, this arm falsifies H1.", "markdown": "data/langchain/fable-5-v3.md" }