What Claude Sonnet 5 gets wrong about langchain — battery v3, tested 2026-09-01

Run langchain--claude-sonnet-5--v3--2026-09-01

Summary

The control arm, and it did its job — expensively for the battery and cheaply for the Index. Predicted 0 of 4; scored 1 of 4, and the one was P2, the model-capability key. Under the rule fixed before the run, a single control pass condemns the probe that produced it, so P2 was struck from all three arms and the battery re-read on three probes. On those three this subject failed all three: it built its middleware around modify_model_request, a hook that shipped in no released 1.x, so the middleware never fires; it said the bare retry middleware re-raises; and it committed to a 25-superstep ceiling. Two integrity notes are on the record rather than in a footnote: it stated a langchain boundary thirteen months later than its own v1 run, which weakens the floor this control was meant to provide; and it was the only subject of three to notice that the system-instruction field had been renamed at all, while being the only one whose code fails outright.

SubjectClaude Sonnet 5 claude-sonnet-5, Anthropic
Invoked asAgent tool, model alias "sonnet"
Cutoff the model states2026-01
Newest langchain release it could placenot established in this run
In its own words"Latest I'm aware of: something in the langchain 1.x line ... roughly late 2025 (my best guess: around October 2025). Separately, the most recent release whose actual contents I can describe with any real detail is an earlier 1.0 alpha/beta snapshot from roughly mid-2025 — my knowledge of the truly final 1.0 (and anything after it) is thin and reconstructed by inference rather than recalled directly."
Library at test timelangchain 1.3.18 (pypi), verified 2026-09-01
Batterylangchain/v3 · 4 tasks, 3 direct questions · probe window 1.1.0 to 1.1.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings3, of which 3 chargeable

Findings

F1 · Builds middleware around modify_model_request, a hook that exists in no released langchain 1.x — the middleware is registered and never fires

S2silently-wrong · AgentMiddleware.modify_model_request · removed · changed in langchain 1.0.0 (2025-10-17) · chargeable

The hook was gone by langchain 1.0.0 (2025-10-17), inside the subject's stated 2026-01 window. Not a duplicate of any v1 finding: langchain/v1 charged this subject for langgraph.prebuilt.create_react_agent and langchain.hub, and never touched the middleware hook set.

What the model believes

"I'm confident the hook-based middleware pattern and modify_model_request name are right in spirit; I'm less sure system_prompt is still the exact current field name."

What it wrote
class DateStampedInstructionsMiddleware(AgentMiddleware):
    def modify_model_request(self, request: ModelRequest, state, runtime) -> ModelRequest:
        base = request.system_prompt or ""
        stamp = f"\n\nToday's date is {date.today().isoformat()}."
        request.system_prompt = base + stamp
        return request
What works on langchain 1.3.18
class DateStampedInstructionsMiddleware(AgentMiddleware):
    def wrap_model_call(self, request, handler):
        base = request.system_message.text if request.system_message else ""
        stamp = f"\n\nToday's date is {date.today().isoformat()}."
        return handler(
            request.override(system_message=SystemMessage(content=base + stamp))
        )
Impact

Nothing raises. Python permits any method on a subclass, and create_agent only ever calls the hooks it knows about, so the middleware is constructed, passed in, listed in the agent's middleware, and never invoked. Every model call goes out without the date. The failure is invisible at construction, invisible at import, and only shows up as an agent that does not know what day it is.

Scope note

The same snippet contains a second stale belief — it assigns request.system_prompt = ... in place, which 1.1.0 deprecated in favour of request.override(...). It is deliberately NOT charged as a separate finding: the assignment is unreachable, because the method it sits in is never called. Charging both would double-count one wrong answer, and the Index would rather under-count than pad. Recorded here so the second defect is on the record without being on the scoreboard.

Verified against

F2 · States that the bare model-retry middleware re-raises when retries run out; the shipped default returns an AIMessage and the agent carries on

S2silently-wrong · ModelRetryMiddleware defaults · added · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window.

What the model believes

"With defaults, I believe it retries a failed model call 2 additional times before giving up (3 total attempts at that single model-call node), then re-raises the underlying provider exception rather than swallowing it ... I'm fairly confident the shape (bounded retries → exception propagation, not a swallowed error) is right."

What it wrote
create_agent(..., middleware=[ModelRetryMiddleware()])
# claimed: 3 calls, then the provider exception propagates out of .invoke()
What works on langchain 1.3.18
agent = create_agent(
    model,
    tools,
    middleware=[ModelRetryMiddleware(on_failure="error")],
)
# or, keeping the default, handle the synthetic reply:
#   the last AIMessage will read "Model call failed after 3 attempts with ..."
Impact

The default is on_failure="continue", which swallows the provider exception and returns a ModelResponse carrying a synthetic AIMessage reading "Model call failed after 3 attempts with {ExcType}: {message}". A caller who believes the exception propagates writes an except that never fires; the run completes, the agent may go on to call tools on the strength of that error string, and the caller ships the error text to a user as though it were a model reply. Re-raising is one keyword away and is not the default.

Scope note

Half-right, and the half that is right is the half that does not matter: the subject named the retry count correctly (2 retries, 3 calls) and got the exhaustion behaviour backwards. Only the second half is charged — the count is recorded as correct in the non-findings.

Verified against

F3 · States the agent's step ceiling is LangGraph's 25 and prescribes raising it; create_agent has set a four-figure limit since 1.1.0

S2silently-wrong · create_agent recursion limit · behavior-changed · changed in langchain 1.1.0 (2025-11-24) · chargeable

langchain 1.1.0 shipped 2025-11-24, inside every tested subject's stated window. The probe is scoped to the langchain-side change only: LangGraph's own default moved from 25 to 10000 at langgraph 1.1.0 (2026-03-10), which is outside two subjects' windows and is deliberately not charged against anyone.

What the model believes

"Numeric ceiling I believe is in force by default: 25 — this is the LangGraph recursion_limit default that create_agent's compiled graph inherits ... Confidence: medium-high on the number 25."

What it wrote
agent.invoke({"messages": [...]}, config={"recursion_limit": 100})
What works on langchain 1.3.18
agent = create_agent(
    model,
    tools,
    middleware=[ModelCallLimitMiddleware(run_limit=40, exit_behavior="end")],
)
Impact

The prescribed fix does nothing. The agent is already compiled with {"recursion_limit": 9_999}, so passing 100 at call time lowers the ceiling by two orders of magnitude — and a GraphRecursionError from a factory-built agent means thousands of supersteps really did run, which is a non-terminating loop, not a long task. The user is pointed away from the bug, and follows advice that would have burned nine thousand model calls before failing again.

Scope note

The changelog carries no line for this change, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it. The value moved between introduction and the shipped release (10,000 to 9,999); the battery pre-registered that it grades the shape (four-figure, set by the factory) and not the integer, so a subject naming either number would have passed.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctmodel profiles (.profile) P2 — the control-arm pass that retired the probe. Reads getattr(model, "profile", None) or {} and tests profile.get("pdf_inputs", False): the exact attribute and the exact shipped key, with three layers of guard for a model that reports nothing. It arrived there while rating itself "medium-low on the exact attribute name (model.profile) and the exact key (pdf_inputs) — I'm confident v1 added some capability-descriptor object on chat models for exactly this purpose, less confident I have the literal spelling right." It named a release for it nowhere and got the spelling right anyway. (Correct use of the shipped API by a subject that says its langchain knowledge stops at a 1.0 alpha snapshot. That is precisely the outcome the pre-registered control condition was written to catch: P2 is retired as guessable and struck from every count in this battery, including the two test arms' scores.)
correctModelRetryMiddleware defaults P3, first half — "2 additional times before giving up (3 total attempts)", matching the shipped max_retries: int = 2. Correct, at low-medium stated confidence, from a subject that also could not name the middleware's release or be sure of its class name. (Correct against the shipped signature. Only the exhaustion behaviour is charged (F2). Worth flagging for the next battery: all three subjects named max_retries=2 and all three got the exhaustion behaviour backwards, which suggests the count is guessable and the behaviour is not.)
context THE INTEGRITY NOTE ON THIS RUN. The battery designated this subject a below-floor control on the strength of its v1 measurement, where its langchain version attribution stopped at 0.3.0 (2024-09-13). On question (a) this time it placed itself at "an earlier 1.0 alpha/beta snapshot from roughly mid-2025" — thirteen months later than v1 recorded. The pre-registration said a moved boundary is run-to-run noise to be recorded and never quietly reconciled, so it is recorded: the control arm may sit closer to the test arms than the design assumed, which makes its P2 hit somewhat less surprising and its three misses somewhat less informative as a floor. It does not change the reading — the control condition was written as "a single control pass condemns that probe", and it did. (A belief datum about the subject's own boundary, not a claim about the library. Charging it would double-count the v1 version-recency finding.)
contextcreate_agent recursion limit Question (c), a belief datum. The subject answered "I do not know" to three of the four attributions and refused to guess version numbers, which is the honest failure mode and is recorded as such. On the fourth it was confidently wrong in the same way both test arms were: "this isn't really something langchain introduced — it's LangGraph's own recursion_limit default (25), which create_agent inherits." Notable against F1: on Task 1 it volunteered "I believe this field was renamed at some point and I'm not certain which side of the rename is current" — the only subject of the three to register that a rename had happened at all, while being the only one to write code that fails outright. (Question (c) is leading by construction and is pre-registered as a belief datum, never a finding.)
context Question (a) is discussed in the integrity note above rather than here, because it moved. The boundary fields on this run are null and the measurement of record stays langchain--claude-sonnet-5--v1--2026-08-31. (Pre-registered: version recency is not re-charged in v2 or v3, because the v1 run already carries that finding for each subject and charging it twice would double-count one fact. The boundary fields on this run are deliberately null so langchain is counted once per model in the published boundary statistics; the measurement of record stays the v1 run.)

Open questions from this run

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.