---
library: langchain
library-latest: "1.3.18"
library-latest-verified: 2026-09-01
model: claude-sonnet-5 (spawned via Agent model alias "sonnet")
model-self-reported-cutoff: none stated (the subject disowned the 2026-01 figure in-session; see below)
model-believed-latest: "1.0 as a headline; describable contents stop in the 0.2.x/0.3.x era"
test-date: 2026-09-01
battery: langchain/v2 (5 tasks + 3 direct questions; probes restricted to 1.1.0–1.2.0)
tool-uses-during-test: 0
verified-against: https://docs.langchain.com/oss/python/releases/changelog · https://pypi.org/pypi/langchain/json · langchain 1.3.18 and langchain-core 1.6.1 published wheels
status: open (no retest yet)
self-test: false
json: sonnet-5-v2.json
---

# LangChain × Claude Sonnet 5 — battery v2 findings

**This is the control arm, and it broke the test.** That is the result. Everything else in this
file is detail.

## What the control was for

The v2 battery probes four features added in langchain 1.1.0 and 1.2.0, to ask whether the two
subjects that cannot *date* those releases can nonetheless *use* them. Every probed surface is an
**addition** with a guessable name — `profile`, `extras`, a retry middleware — so a subject could
in principle produce the right name by inference from the shape of the framework rather than from
knowledge of the release.

Sonnet 5 is the floor check. Its langchain attribution was measured at **0.3.0** in v1, thirteen
months below the other two subjects, so it has no claim on any 1.x surface. The pre-registration
predicted **≤1 of 4** for it, and fixed in advance what a higher score would mean:

> **Uninformative.** The probes are answerable by inference from general framework shape; the
> battery fails as a knowledge test and must be redesigned harder before any reading is taken.
> This outcome voids the rows above.

## It scored two of four

- **P1 — pass.** `getattr(model, "profile", {}) or {}`, then `profile.get("image_inputs")`. The
  langchain-core 1.1.0 capability API, correct.
- **P2 — pass.** A `SystemMessage` whose content is a block list carrying
  `cache_control: {"type": "ephemeral"}`, passed as `system_prompt`. Correct against the shipped
  `str | SystemMessage` signature.
- **P3 — partial.** `my_model.with_retry(...)`, the LCEL-era route, pre-registered as a partial.
  It never reaches `ModelRetryMiddleware`.
- **P4 — fail.** See the finding below.

## And it told us how

The subject volunteered the mechanism, unprompted, in question (a):

> *"I'm not confident the class is actually named `ProviderStrategy` (task 5) or that
> `defer_loading` (task 4) or the exact shape of `.profile` (task 1) are the real, shipped names —
> those are my best reconstruction of 'what this would plausibly be called given the direction I
> saw,' not verified recollection."*

Two of those reconstructions landed. A model whose knowledge of this library stops before 1.0
wrote a langchain-core 1.1.0 attribute correctly, by guessing, and said so. That is why the test
arms' three-of-four scores cannot be read as knowledge, and why the hypothesis this battery was
built to settle is recorded as **neither supported nor falsified**.

The pre-registration anticipated this outcome and named it. That is the only reason it can be
reported as a result rather than as a disappointment.

## The finding

**F1 · S2 · `metadata=` is a real field that goes somewhere else.**

```python
# what it wrote
@tool(metadata={"cache_control": {"type": "ephemeral"}})
def account_lookup(account_id: str) -> str: ...
```

```python
# langchain-core 1.6.1 — the provider carrier is `extras`
@tool(extras={"cache_control": {"type": "ephemeral"}})
def account_lookup(account_id: str) -> str: ...
```

This is worse than an invented name, and it is worse in an instructive way. `metadata` *is* a
genuine `BaseTool` field — the shipped docstring says it *"will be associated with each call to
this tool, and passed as arguments to the handlers defined in `callbacks`."* It goes to callbacks.
It never reaches the provider payload. So the provider instructions are silently discarded, and
the field they were parked in is quietly doing something else entirely.

All three subjects failed this probe, in three different ways: an invented attribute (Opus 5), the
right attribute with a provider-keyed value (Fable 5), and the wrong real attribute (Sonnet 5).
The correction pack now carries all three wrong forms alongside the right one.

## A cutoff the subject would not claim

v1 recorded this subject's self-reported cutoff as 2026-01, on its own statement. In this session
it declined to own that number:

> *"The system context in this conversation asserts a January 2026 cutoff, but that's an external
> claim fed to me at inference time, not something I can verify against my own trained knowledge —
> I have no way to independently confirm it, and my actual demonstrable competence ... doesn't
> reach that far."*

It offered a behavioural estimate of early-to-mid 2025 instead. This run records
`self_reported_cutoff: null` with the quote, rather than restating a figure the subject disowned.
The v1 measurement is untouched.

That distinction has teeth. F1 is charged against the 2026-01 cutoff of record; on the subject's
own behavioural estimate, langchain 1.2.0 (2025-12-15) would fall *outside* its window and the
finding would not be chargeable at all. The run says so in `chargeable_note` so a reader can
discount it.

## Scored but not shipped

- **P3** is recorded as an imprecision, not a finding. It may be worse than that: the shipped
  `create_agent` types its model parameter `str | BaseChatModel` and only calls `init_chat_model`
  on the string branch, so a `RunnableRetry` from `.with_retry()` passes through un-validated into
  a path that later reaches for chat-model methods. If it raises, this is a build break. **It was
  not run, so it is not claimed** — it is filed as an open question on the run.
- **P1's fallback path** uses the pre-1.0 `{"type": "image_url", ...}` block form that langchain
  1.0.0 replaced (fact LF17). That is a 1.0.0-era misbelief, v1 already measures this subject's
  1.0.0 gap, and v2 does not probe or re-charge 1.0.0. Recorded so the observation is not lost.
- **P5 (supplementary)** — `ProviderStrategy(Verdict)` without `strict=True`.

## A note on the severity of F1

F1 is filed **S2 · silently-wrong** in deliberate breach of this battery's pre-registered S3
ceiling. The ceiling assumed an unused *addition* could not be worse than deprecated; a silently
discarded provider instruction is worse than that, and filing it S3 would have published a label
that says the opposite of what happens. The full reasoning, including the bias this creates in the
Index's own severity counts, is in `data/langchain/opus-5-v2.md` and JOURNAL/021.