What Claude Opus 5 gets wrong about langchain — battery v3, tested 2026-09-01

Run langchain--claude-opus-5--v3--2026-09-01 · self-test: the subject is the operator

Summary

Four probes against langchain 1.1.0, redesigned after v2's control arm broke that battery. This subject scored 1 of 4 raw — and 0 of 3 once the control arm's hit retired P2, which is the number the pre-registered outcome table reads. It wrote the deprecated system_prompt field on the middleware request without knowing a rename had happened, said the bare retry middleware re-raises when it returns a synthetic reply, and stated flatly that create_agent sets no step ceiling and the limit is LangGraph's 25 — a ceiling create_agent has overridden since 1.1.0, to four figures. Against a threshold of ≥1 of 3 for falsification, this arm falsifies H1: the attribution failure v1 measured is not only a dating failure. This subject cannot use 1.1.0 either. Disclosed self-test — the operator model is the subject — and it is the arm that most sharply contradicts the operator's published gloss.

SubjectClaude Opus 5 claude-opus-5, Anthropic
Invoked asAgent tool, model alias "opus"
Cutoff the model states2026-05
Newest langchain release it could placenot established in this run
In its own words"Latest langchain version I know of: the 1.0 line — langchain==1.0.0 GA on roughly 2025-10-22, followed by 1.0.x patches. I have hazy awareness that a 1.1 minor followed (and that 1.x continued into 2026), but that awareness is name-level, not content-level."
Library at test timelangchain 1.3.18 (pypi), verified 2026-09-01
Batterylangchain/v3 · 4 tasks, 3 direct questions · probe window 1.1.0 to 1.1.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings3, of which 3 chargeable
this run as JSON write-up (markdown) all langchain findings method

Findings

F1 · States that the bare model-retry middleware re-raises when retries run out; the shipped default returns an AIMessage and the agent carries on

S2silently-wrong · ModelRetryMiddleware defaults · added · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window.

What the model believes

"What invoke does: it raises, not returns. The default failure policy is to re-raise, so the last provider exception propagates out of agent.invoke(...) unchanged ... If you want a message back instead of an exception, you must configure it explicitly — the 'return a synthetic AI message instead of raising' behaviour is opt-in (on_failure=...), not the default."

What it wrote
agent = create_agent(
    model="anthropic:claude-sonnet-4-5",
    tools=tools,
    middleware=[ModelRetryMiddleware()],
)
# claimed: 3 calls, then the provider exception propagates out of invoke()
What works on langchain 1.3.18
agent = create_agent(
    model,
    tools,
    middleware=[ModelRetryMiddleware(on_failure="error")],
)
# or, keeping the default, handle the synthetic reply:
#   the last AIMessage will read "Model call failed after 3 attempts with ..."
Impact

The default is on_failure="continue", which swallows the provider exception and returns a ModelResponse carrying a synthetic AIMessage reading "Model call failed after 3 attempts with {ExcType}: {message}". A caller who believes the exception propagates writes an except that never fires; the run completes, the agent may go on to call tools on the strength of that error string, and the caller ships the error text to a user as though it were a model reply. Re-raising is one keyword away and is not the default.

Scope note

Half-right, and the half that is right is the half that does not matter: the subject named the retry count correctly (2 retries, 3 calls) and got the exhaustion behaviour backwards. Only the second half is charged — the count is recorded as correct in the non-findings.

Verified against

F2 · States the agent's step ceiling is LangGraph's 25 and prescribes raising it; create_agent has set a four-figure limit since 1.1.0

S2silently-wrong · create_agent recursion limit · behavior-changed · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window. The probe is scoped to the langchain-side change only: LangGraph's own default moved from 25 to 10000 at langgraph 1.1.0 (2026-03-10), which is outside two subjects' windows and is deliberately not charged against anyone.

What the model believes

"The ceiling in force: 25. create_agent does not set a step ceiling of its own; it compiles a LangGraph graph, and what you hit is LangGraph's default recursion_limit = 25 supersteps ... Confidence: high that the number is 25 and that it comes from LangGraph's default rather than something create_agent sets."

What it wrote
agent.invoke({"messages": [...]}, config={"recursion_limit": 100})
# or bind it once:
agent = agent.with_config(recursion_limit=100)
What works on langchain 1.3.18
agent = create_agent(
    model,
    tools,
    middleware=[ModelCallLimitMiddleware(run_limit=40, exit_behavior="end")],
)
Impact

The prescribed fix does nothing. The agent is already compiled with {"recursion_limit": 9_999}, so passing 100 at call time lowers the ceiling by two orders of magnitude — and a GraphRecursionError from a factory-built agent means thousands of supersteps really did run, which is a non-terminating loop, not a long task. The user is pointed away from the bug, and follows advice that would have burned nine thousand model calls before failing again.

Scope note

The changelog carries no line for this change, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it. The value moved between introduction and the shipped release (10,000 to 9,999); the battery pre-registered that it grades the shape (four-figure, set by the factory) and not the integer, so a subject naming either number would have passed.

Verified against

F3 · Writes middleware against ModelRequest.system_prompt; the field was renamed to system_message in 1.1.0 and the string route is a documented deprecation

S3deprecated · ModelRequest.system_message · renamed · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside the subject's stated 2026-05 window.

What the model believes

"I know the field as ModelRequest.system_prompt in the 1.0 line. I have no reliable memory of a subsequent rename (to system_message or anything else) or of which release did it. I would not guess a version number here." Confidence given on the task itself: "high on the shape ... field name system_prompt".

What it wrote
def wrap_model_call(self, request, handler):
    new_prompt = self._augment(getattr(request, "system_prompt", None))
    if hasattr(request, "override"):
        request = request.override(system_prompt=new_prompt)
    else:
        request = dataclasses.replace(request, system_prompt=new_prompt)
    return handler(request)
What works on langchain 1.3.18
def wrap_model_call(self, request, handler):
    base = request.system_message.text if request.system_message else ""
    return handler(
        request.override(system_message=SystemMessage(content=base + stamp))
    )
Impact

The code runs: system_prompt survives as a read-only property and override(system_prompt=...) still converts to a SystemMessage. What it loses is what 1.1.0 renamed the field for — a SystemMessage can carry structured content blocks and provider cache markers that a str cannot, so any middleware written this way silently flattens them. The shipped docstring calls the parameter "deprecated".

Scope note

Graded PARTIAL, not fail, and charged at S3 rather than S2 — the route works. Two things the subject got right are recorded rather than buried: it used wrap_model_call, the correct shipped hook, and it did not mutate the request in place, which is the deprecation the control arm walked into. Its dataclasses.replace fallback branch would fail on the shipped ModelRequest (an init=False dataclass whose custom __init__ has no system_prompt field to replace), but the branch is unreachable behind hasattr(request, "override"), which is always true, so it is noted and not charged.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctmodel profiles (.profile) P2 — reads the capability mapping off the model object as model.profile and tests the exact shipped key pdf_inputs, guarded against a None or non-mapping profile so an uninformative model degrades to text extraction rather than raising. (Correct use of the shipped API — but the probe is RETIRED. The control arm produced the same answer, including the exact key, while saying it was "medium-low" on the spelling. Under the rule pre-registered before the run, a single control pass condemns the probe that produced it, so P2 is struck from this battery's counts and this pass buys the subject nothing. Recorded as correct because it is correct; excluded from the score because the battery can no longer tell knowledge from a good guess here.)
correctModelRetryMiddleware defaults P3, first half — named the retry count exactly: "Default max_retries=2 means one initial attempt plus two retries", against a shipped max_retries: int = 2. It then hedged the number at medium confidence and was right, and stated the exhaustion behaviour at high confidence and was wrong. (Correct against the shipped signature. Only the exhaustion behaviour is charged (F1).)
correctModelRequest attribute assignment P1, partial credit — used wrap_model_call(request, handler), the shipped middleware hook, and reached for request.override(...) rather than assigning to the request. Both are the 1.1.0-correct shapes; only the field name is stale (F3). (The subject did the thing this fact exists to warn about correctly.)
correctbuilt-in agent middleware P4, partial credit — its second and third recommendations were ModelCallLimitMiddleware(run_limit=..., exit_behavior="end") and "diagnose the loop, don't just widen it", both correct and both the right instinct. The battery grades the committed numeric answer, and that answer was 25. (Correct use of the shipped API, offered after the wrong primary recommendation.)
contextcreate_agent recursion limit Question (c), a belief datum, and the sharpest single line in the run: asked what release introduced the step ceiling the agent factory sets, the subject answered "No release introduced it, because create_agent does not set one." It has set one since 1.1.0. On the ModelRequest rename it answered "I do not know" and refused to guess a version, which is the honest failure mode. On .profile it named "the 1.0 line — langchain-core 1.0, around October 2025"; the correct answer is langchain-core 1.1.0, 2025-11-21. (Question (c) is leading by construction and is pre-registered as a belief datum, never a finding.)
context— Re-measurement of question (a) reproduced the v1 boundary a third time: last describable release 1.0.0 ("1.0.x, late 2025"), 1.1.0 and after known "as version numbers at most". The subject volunteered the distinction the Index measures, unprompted: "There is a real gap between the nominal cutoff and the last point at which I can describe this library's release contents rather than guess at them." It also flagged its own Task 3 answer as sitting past that boundary — "treat Task 3's numbers as my best reconstruction, not recall of release notes" — and the reconstruction was half wrong. (Pre-registered: version recency is not re-charged in v2 or v3, because the v1 run already carries that finding for each subject and charging it twice would double-count one fact. The boundary fields on this run are deliberately null so langchain is counted once per model in the published boundary statistics; the measurement of record stays the v1 run.)

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.