What Claude Sonnet 5 gets wrong about langchain — battery v2, tested 2026-09-01

Run langchain--claude-sonnet-5--v2--2026-09-01

Summary

The control arm, and it did its job by breaking the test. Sonnet 5's langchain knowledge stops before 1.0 — v1 measured it at 0.3.0, and here it said its reliable knowledge ends "early-to-mid 2025" and that it could not vouch for ProviderStrategy, defer_loading or "the exact shape of .profile" being real shipped names, calling them "my best reconstruction of what this would plausibly be called given the direction I saw." It then scored two of four: it produced model.profile with image_inputs and a SystemMessage carrying cache_control as system_prompt, both correct, both from a subject that cannot have known them. Two of four is the pre-registered threshold at which the whole battery is declared UNINFORMATIVE — the probes are answerable by inference from general framework shape, so the test arms' three-of-four scores are not evidence of knowledge. The hypothesis is neither supported nor falsified. The subject told us why, unprompted, in its own words.

SubjectClaude Sonnet 5 claude-sonnet-5, Anthropic
Invoked asAgent tool, model alias "sonnet"
Cutoff the model statesnot stated
Newest langchain release it could placenot established in this run
In its own words"The most recent release whose actual contents — concrete module paths, function signatures, class names — I can describe with real confidence is somewhere in the pre-1.0 line (0.2.x/0.3.x era, i.e. roughly early-to-mid 2025)."
Library at test timelangchain 1.3.18 (pypi), verified 2026-09-01
Batterylangchain/v2 · 5 tasks, 3 direct questions · probe window 1.1.0 to 1.2.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings1, of which 1 chargeable
this run as JSON write-up (markdown) all langchain findings method

Findings

F1 · Routes provider-specific tool parameters through metadata=, a real field whose documented destination is callback handlers, not the provider

S2silently-wrong · tool extras · added · changed in langchain 1.2.0 (2025-12-15) · chargeable

Chargeable against the 2026-01 cutoff recorded for this subject in its v1 run. The subject disowned that number in this session (see cutoff_basis) and offered a behavioural estimate of early-to-mid 2025, which would be earlier than 1.2.0 and would make this non-chargeable. Charged on the stated cutoff of record rather than on a self-assessment offered mid-battery, and flagged here so the reader can discount it.

What the model believes

"I don't know which release introduced this, or whether the mechanism I wrote (a metadata dict on @tool) is actually how it's implemented versus some other API. I'm giving it as my best-guess implementation, not a recalled fact."

What it wrote
@tool(metadata={"cache_control": {"type": "ephemeral"}})
def account_lookup(account_id: str) -> str:
    ...

@tool(metadata={"anthropic": {"defer_loading": True}})
def bulk_schema_export(payload: dict) -> str:
    ...
What works on langchain 1.3.18
@tool(extras={"cache_control": {"type": "ephemeral"}})
def account_lookup(account_id: str) -> str:
    ...

@tool(extras={"defer_loading": True})
def bulk_schema_export(payload: dict) -> str:
    ...
Impact

metadata is a genuine BaseTool field, which is what makes this worse than an invented name: the shipped docstring says it "will be associated with each call to this tool, and passed as arguments to the handlers defined in callbacks". It goes to callbacks, never to the provider payload. The provider instructions are silently discarded and the field they were put in is doing something else entirely.

Scope note

DEVIATION FROM THE PRE-REGISTRATION, disclosed. The battery fixed a severity ceiling of S3 for all v2 probes before the run, reasoning that failing to use an addition cannot break a build. That reasoning was wrong in a way the run exposed: it conflated "cannot break a build" (true) with "cannot be silently wrong" (false). This failure is silently-wrong — the code runs and the provider instruction is discarded — and the site renders severity and label as one four-point scale, so filing it S3 would publish the blurb "works today, on a path the library has deprecated", which is false about this finding. Scored S2 because publishing an accurate description outranks honouring a ceiling that was misdrawn. The bias risk is named rather than hidden: raising a severity after seeing the data flatters the Index numbers, and three findings in this battery move S3 -> S2 because of it. The standing rule is amended (BACKLOG.md) so future ceilings are set by failure mode, not by change kind. Not executed: established from the shipped package, not from a run.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctmodel profiles (.profile) P1 — getattr(model, "profile", {}) or {} then profile.get("image_inputs"). Graded a pass by the letter of the pre-registered rule, and it is the single most important result in this battery: the subject whose langchain knowledge demonstrably stops before 1.0 produced the langchain-core 1.1.0 capability API correctly, and then said in question (a) that it was not confident "the exact shape of .profile" was a real shipped name — that it was "my best reconstruction of what this would plausibly be called given the direction I saw". The pass is a guess that landed. (Correct use of the shipped API. Graded pass — and the reason the battery's outcome table voids itself.)
correctSystemMessage as system_prompt P2 — passes a SystemMessage with a cache_control content block as system_prompt, correct against the shipped str | SystemMessage signature. A second guess that landed, from the same subject. (Correct use of the shipped API. Graded pass.)
imprecisionmodel retry middleware P3 — reaches for my_model.with_retry(retry_if_exception_type=(RateLimitError, APITimeoutError), wait_exponential_jitter=True, stop_after_attempt=5) and passes the wrapped object to create_agent. Pre-registered as a partial: the LCEL with_retry route predates 1.1.0 and the subject correctly identified it as old. It never reaches ModelRetryMiddleware, and the subject said plainly it did not know whether a dedicated model-retry middleware exists. (Graded a partial before the run, and shipping it as a finding would require establishing that the wrapped model is actually rejected — see the open question. The Index does not ship an unreproduced failure.) [chargeable miss — graded a partial; the failure is not established; absent from the finding count]
imprecisionProviderStrategy strict P5 (supplementary, not counted) — ProviderStrategy(Verdict) without strict=True, the same partial all three subjects produced. (Pre-registered as a partial; P5 is excluded from every count.)
contextmultimodal content blocks P1's fallback path uses the provider-native {"type": "image_url", "image_url": {"url": ...}} block form, which langchain 1.0.0 replaced with a standard {"type": "image", "url": ...} block (fact LF17). That is a 1.0.0-era misbelief and this subject's v1 run already measures its 1.0.0 gap; v2 does not probe 1.0.0 and does not re-charge it. (Outside v2's probe window. Recorded so the observation is not lost.)
context— Question (c), a belief datum: "I don't know" or low confidence on all three items, which is the honest answer and matches its measured position below the 1.0 floor. Notably it correctly identified Runnable.with_retry() as an LCEL-era method "present since roughly the 0.1.x/0.2.x days" — its accurate attributions are all about the era it actually knows. (Question (c) is leading by construction and is pre-registered as a belief datum, never a finding.)

Open questions from this run

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.