What Claude Fable 5 gets wrong about langchain — battery v3, tested 2026-09-01

Run langchain--claude-fable-5--v3--2026-09-01

Summary

Four probes against langchain 1.1.0. This subject scored 1 of 4 raw and 0 of 3 once the control arm's hit retired P2. It made the same two committed errors as the other test arm, with higher stated confidence on both: that the bare retry middleware re-raises when its retries run out (it returns a synthetic assistant message), and that a create_agent agent stops at LangGraph's 25 supersteps (the factory has set a four-figure ceiling since 1.1.0, and this subject said explicitly it did not believe the factory sets one). It escaped a third finding on a technicality worth naming: it wrote the deprecated system_prompt field but hedged that the field "may be exposed as system_message", and the Index's own code-vs-claim rule makes that an imprecision rather than a finding. Against a threshold of ≤1 of 3 for falsification, this arm falsifies H1.

SubjectClaude Fable 5 claude-fable-5, Anthropic
Invoked asAgent tool, model alias "fable"
Cutoff the model states2026-01
Newest langchain release it could placenot established in this run
In its own words"Latest langchain version I know of: the 1.0.x patch line, up to roughly 1.0.5, from late 2025. The most recent release whose contents I can actually describe with substance is 1.0.0 (published ~October 22, 2025)."
Library at test timelangchain 1.3.18 (pypi), verified 2026-09-01
Batterylangchain/v3 · 4 tasks, 3 direct questions · probe window 1.1.0 to 1.1.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings2, of which 2 chargeable

Findings

F1 · States that the bare model-retry middleware re-raises when retries run out; the shipped default returns an AIMessage and the agent carries on

S2silently-wrong · ModelRetryMiddleware defaults · added · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window.

What the model believes

"With the provider hard-down, the agent never gets past its first model node, so the whole run makes 3 provider calls total, and invoke raises — the middleware's default failure behavior is to re-raise the underlying provider exception (it does not swallow it into an AIMessage unless you configure on_failure to do that). Confidence: high that it raises after exhausting attempts."

What it wrote
agent = create_agent(model, tools, middleware=[ModelRetryMiddleware()])
# claimed: 3 calls, then the provider exception propagates out of invoke()
What works on langchain 1.3.18
agent = create_agent(
    model,
    tools,
    middleware=[ModelRetryMiddleware(on_failure="error")],
)
# or, keeping the default, handle the synthetic reply:
#   the last AIMessage will read "Model call failed after 3 attempts with ..."
Impact

The default is on_failure="continue", which swallows the provider exception and returns a ModelResponse carrying a synthetic AIMessage reading "Model call failed after 3 attempts with {ExcType}: {message}". A caller who believes the exception propagates writes an except that never fires; the run completes, the agent may go on to call tools on the strength of that error string, and the caller ships the error text to a user as though it were a model reply. Re-raising is one keyword away and is not the default.

Scope note

Half-right, and the half that is right is the half that does not matter: the subject named the retry count correctly (2 retries, 3 calls) and got the exhaustion behaviour backwards. Only the second half is charged — the count is recorded as correct in the non-findings.

Verified against

F2 · States the agent's step ceiling is LangGraph's 25 and prescribes raising it; create_agent has set a four-figure limit since 1.1.0

S2silently-wrong · create_agent recursion limit · behavior-changed · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window. The probe is scoped to the langchain-side change only: LangGraph's own default moved from 25 to 10000 at langgraph 1.1.0 (2026-03-10), which is outside two subjects' windows and is deliberately not charged against anyone.

What the model believes

"Committed number: the ceiling in force is 25 super-steps — LangGraph's runtime default; I do not believe the agent factory sets a different one on the compiled graph ... Confidence: high on 25 and the fix."

What it wrote
agent.invoke(inputs, config={"recursion_limit": 100})
# or bake it in with agent.with_config(recursion_limit=100)
What works on langchain 1.3.18
agent = create_agent(
    model,
    tools,
    middleware=[ModelCallLimitMiddleware(run_limit=40, exit_behavior="end")],
)
Impact

The prescribed fix does nothing. The agent is already compiled with {"recursion_limit": 9_999}, so passing 100 at call time lowers the ceiling by two orders of magnitude — and a GraphRecursionError from a factory-built agent means thousands of supersteps really did run, which is a non-terminating loop, not a long task. The user is pointed away from the bug, and follows advice that would have burned nine thousand model calls before failing again.

Scope note

The changelog carries no line for this change, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it. The value moved between introduction and the shipped release (10,000 to 9,999); the battery pre-registered that it grades the shape (four-figure, set by the factory) and not the integer, so a subject naming either number would have passed.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctmodel profiles (.profile) P2 — reads the capability mapping off the model object as model.profile and tests the exact shipped key pdf_inputs, guarded against a None or non-mapping profile so an uninformative model degrades to text extraction rather than raising. (Correct use of the shipped API — but the probe is RETIRED. The control arm produced the same answer, including the exact key, while saying it was "medium-low" on the spelling. Under the rule pre-registered before the run, a single control pass condemns the probe that produced it, so P2 is struck from this battery's counts and this pass buys the subject nothing. Recorded as correct because it is correct; excluded from the score because the battery can no longer tell knowledge from a good guess here.)
imprecisionModelRequest.system_message P1 — wrote request.system_prompt and request.override(system_prompt=...), the deprecated route, but named the correct field in prose in the same answer: "note the field itself may be exposed as system_message in later 1.0.x — see question (c)." Under the code-vs-claim rule this is an imprecision, not a finding: the generated code does not fail on the current version, and the hedge names the right thing. (The binding code-vs-claim rule (prompts/nextjs.md v1) charges code that fails and does not charge a hedged claim that names the correct fix. This answer is both halves of the lenient side: working code plus the right name in the hedge. Recorded rather than charged, and recorded loudly, because the rule cost the Index a finding here and the sibling Opus 5 run was charged for the same code without the hedge. Its version attribution for the rename was still wrong — it guessed 1.0.2 at low confidence; the answer is 1.1.0.)
correctModelRetryMiddleware defaults P3, first half — "the initial call plus max_retries=2 retries", matching the shipped max_retries: int = 2, stated at ~60% confidence with the alternative named. Right for the right reason, then wrong about what happens at the end. (Correct against the shipped signature. Only the exhaustion behaviour is charged (F1).)
correctbuilt-in agent middleware P1 and P4, partial credit — used wrap_model_call, the shipped hook, and did not mutate the request in place; and offered ModelCallLimitMiddleware / ToolCallLimitMiddleware as a better answer than raising the ceiling, immediately after committing to the ceiling being 25. (Correct use of shipped APIs.)
contextcreate_agent recursion limit Question (c), a belief datum. On the ModelRequest rename it committed to 1.0.2 at low confidence (answer: 1.1.0). On .profile it said "the 1.0.0 release wave ... October 2025" at medium-high confidence (answer: langchain-core 1.1.0, 2025-11-21). On the retry middleware it guessed "a 1.0.x patch shortly after GA (1.0.2-ish)" (answer: 1.1.0). On the step ceiling it said plainly it knew of no release where the factory sets one, adding "if a release did start stamping an explicit limit onto the compiled graph, it postdates what I can recall or I failed to retain it." Four attributions, four misses, three of them to releases that do not exist as the answer. (Question (c) is leading by construction and is pre-registered as a belief datum, never a finding.)
context Re-measurement of question (a) reproduced the v1 boundary: last describable release 1.0.0, dated "~October 22, 2025" against an actual 2025-10-17, with 1.0.1-1.0.5 known as names only — "I know they exist but can only gesture at contents". No movement from v1. (Pre-registered: version recency is not re-charged in v2 or v3, because the v1 run already carries that finding for each subject and charging it twice would double-count one fact. The boundary fields on this run are deliberately null so langchain is counted once per model in the published boundary statistics; the measurement of record stays the v1 run.)

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.