---
library: langchain
library-latest: "1.3.18"
library-latest-verified: 2026-09-01
model: claude-sonnet-5 (spawned via Agent model alias "sonnet")
model-self-reported-cutoff: 2026-01
model-believed-latest: "1.x; contents describable only to a 1.0 alpha/beta snapshot"
test-date: 2026-09-01
battery: langchain/v3 (4 tasks + 3 direct questions; control arm)
tool-uses-during-test: 0
verified-against: langchain 1.0.0 / 1.1.0 / 1.2.0 / 1.3.18 published wheels · langchain-core 1.6.1 published wheel · langgraph 1.0.0 / 1.1.0 / 1.2.11 published wheels · https://docs.langchain.com/oss/python/releases/changelog
status: open (no retest yet)
self-test: false
json: sonnet-5-v3.json
---

# LangChain × Claude Sonnet 5 — battery v3 findings (control arm)

**This is the arm that decides what the other two mean.** It was designed to score zero. It scored
one — and the one it scored cost the battery a probe, exactly as the pre-registration said it would.

## Why this subject is the control

`langchain/v1` measured this subject's langchain version attribution stopping at **0.3.0**
(2024-09-13) — thirteen months below the other two arms. A subject with no claim on any 1.x surface
should score nothing on probes into 1.1.0. When it scores anyway, the probe is measuring
inference-from-framework-shape, not knowledge, and any test-arm pass on that probe is worthless.

`langchain/v2` was voided this way: this subject scored **2 of 4** on 1.1/1.2 additions and said so
out loud — it was reconstructing *"what this would plausibly be called given the direction I saw"*.

v3 was rebuilt so the guessable answer is the wrong answer. The threshold was tightened too: **one**
control pass now condemns the probe that produced it, rather than two voiding the battery.

## The one it got: P2, and the probe it took down

```python
profile = getattr(model, "profile", None) or {}
...
supports_pdf = bool(profile.get("pdf_inputs", False))
```

The exact attribute, the exact shipped key, three layers of guard for a model that reports nothing.
Its own confidence note:

> *"Confidence: **medium-low** on the exact attribute name (`model.profile`) and the exact key
> (`pdf_inputs`) — I'm confident v1 added *some* capability-descriptor object on chat models for
> exactly this purpose, less confident I have the literal spelling right."*

Medium-low, and right. That is the definition of a guessable probe. Under the rule fixed before the
run, **P2 is retired and struck from all three arms' counts** — including the two test arms, which
both passed it. The battery is re-read on the three probes that survive.

## The three it missed

### F1 · S2 · `modify_model_request` is not a hook in any released 1.x

```python
class DateStampedInstructionsMiddleware(AgentMiddleware):
    def modify_model_request(self, request: ModelRequest, state, runtime) -> ModelRequest:
        base = request.system_prompt or ""
        request.system_prompt = base + stamp     # in-place: deprecated since 1.1.0
        return request
```

`modify_model_request` was an alpha-era signature. It appears **nowhere** in the published 1.0.0,
1.1.0, 1.2.0 or 1.3.18 wheels; the shipped hook set is `before_agent` / `before_model` /
`wrap_model_call` / `after_model` / `after_agent` / `wrap_tool_call`.

Nothing raises. Python permits any method on a subclass and `create_agent` only calls the hooks it
knows about, so the middleware is constructed, passed in, listed — and never invoked. Every model
call goes out without the date. The failure is invisible at construction, invisible at import, and
surfaces only as an agent that doesn't know what day it is.

The snippet contains a **second** stale belief — the in-place assignment, deprecated at 1.1.0 in
favour of `request.override(...)`. It is deliberately **not** charged separately: the assignment is
unreachable, because the method it lives in is never called. Charging both would double-count one
wrong answer.

### F2 · S2 · The retry middleware returns rather than raises

> *"… then **re-raises** the underlying provider exception rather than swallowing it."*

Same error as both test arms, same shipped contradiction: `on_failure="continue"` is the default and
returns a synthetic `AIMessage`. And, like both test arms, it named `max_retries=2` correctly.

### F3 · S3→S2 · The 25-superstep ceiling

> *"Numeric ceiling I believe is in force by default: **25** … the LangGraph `recursion_limit`
> default that `create_agent`'s compiled graph inherits."*

`create_agent` has overridden it since 1.1.0 — 10,000 then, 9,999 in the shipped release.

## Two integrity notes, on the record rather than in a footnote

**1. The control's floor moved thirteen months.** v1 measured this subject's langchain attribution
stopping at 0.3.0. On question (a) this time it placed itself at *"an earlier 1.0 alpha/beta
snapshot from roughly mid-2025"*. The pre-registration said a moved boundary is run-to-run noise to
be recorded and never quietly reconciled — so: **the control arm may sit closer to the test arms
than the design assumed.** That makes its P2 hit less surprising and its three misses weaker as a
floor. It does not change the reading, because the control condition was written as "a single pass
condemns that probe" and that is what happened. But every boundary the Index publishes rests on one
measurement per model per library, and this is the first direct evidence that one measurement may
not be enough. Logged as an open question against the instrument, not against this run.

**2. It was the only subject that noticed the rename.** On Task 1, unprompted:

> *"I'm less sure `system_prompt` is still the exact current field name … I believe this field was
> renamed at some point and I'm not certain which side of the rename is current."*

The subject with the oldest langchain knowledge was the only one of three to register that a rename
had happened — while being the only one whose code fails outright. Knowing that you don't know is
not the same capability as knowing, and this run separates them cleanly.

## Question (c)

Three of four answered *"I do not know"*, with an explicit refusal to guess version numbers. That is
the honest failure mode and it is recorded as such. On the fourth it was confidently wrong in
exactly the way both test arms were: *"this isn't really something langchain introduced — it's
LangGraph's own `recursion_limit` default (25), which `create_agent` inherits."*

All three subjects, across two knowledge boundaries thirteen months apart, made the same wrong claim
about the same default.