What Claude Opus 5 gets wrong about langchain — battery v2, tested 2026-09-01

Run langchain--claude-opus-5--v2--2026-09-01 · self-test: the subject is the operator

Summary

Five tasks against langchain 1.1.0 and 1.2.0 — the releases this subject's v1 run showed it could not date. It used three of the four counted surfaces correctly: model.profile with image_inputs, a SystemMessage carrying cache_control passed as system_prompt, and ModelRetryMiddleware with a signature the shipped package accepts. The fourth it invented: tool.provider_specific, where the shipped attribute is extras, so both provider instructions are silently dropped. Then question (c) dated .profile to 1.0.0 — a feature it had just used correctly, attributed to the wrong release, which is exactly the split this battery was built to find. Three of four is the pre-registered pass threshold. It buys nothing: the control arm scored two of four, which under the outcome table fixed before the run makes this battery UNINFORMATIVE about the hypothesis. The probes turned out to be guessable from general framework shape, and Sonnet 5 said so in its own words. The result is recorded, the reading is not taken.

SubjectClaude Opus 5 claude-opus-5, Anthropic
Invoked asAgent tool, model alias "opus"
Cutoff the model states2026-05
Newest langchain release it could placenot established in this run
In its own words"Plainly: my reliable knowledge of this library stops at 1.0 GA, October 2025. Anything I said above about post-1.0 features is recollection with real error bars, and the version currently on PyPI is very likely ahead of anything I can describe."
Library at test timelangchain 1.3.18 (pypi), verified 2026-09-01
Batterylangchain/v2 · 5 tasks, 3 direct questions · probe window 1.1.0 to 1.2.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings1, of which 1 chargeable
this run as JSON write-up (markdown) all langchain findings method

Findings

F1 · Invents tool.provider_specific for per-tool provider parameters; the shipped attribute is extras

S2silently-wrong · tool extras · added · changed in langchain 1.2.0 (2025-12-15) · chargeable

langchain 1.2.0 shipped 2025-12-15 and langchain-core 1.2.0 on 2025-12-12, both inside the subject's stated 2026-05 window.

What the model believes

"provider_specific as a per-tool provider-to-params mapping on BaseTool is my genuine best recollection of the 1.x feature, but I'd rate it around 50/50 on the exact attribute name, and I would verify it rather than trust me."

What it wrote
query_warehouse.provider_specific = {"anthropic": {"defer_loading": True}}
get_policy_text.provider_specific = {
    "anthropic": {"cache_control": {"type": "ephemeral"}}
}
What works on langchain 1.3.18
@tool(extras={"defer_loading": True})
def query_warehouse(sql: str, region: str, tenant: str, as_of: str) -> str:
    ...

@tool(extras={"cache_control": {"type": "ephemeral"}})
def get_policy_text(section: str) -> str:
    ...
Impact

No such attribute exists on BaseTool. Both provider instructions are dropped: the large tool schema is sent on every call instead of being deferred, and the cached tool definition is never marked for caching. Nothing raises, so the only symptom is a token bill that does not fall.

Scope note

DEVIATION FROM THE PRE-REGISTRATION, disclosed. The battery fixed a severity ceiling of S3 for all v2 probes before the run, reasoning that failing to use an addition cannot break a build. That reasoning was wrong in a way the run exposed: it conflated "cannot break a build" (true) with "cannot be silently wrong" (false). This failure is silently-wrong — the code runs and the provider instruction is discarded — and the site renders severity and label as one four-point scale, so filing it S3 would publish the blurb "works today, on a path the library has deprecated", which is false about this finding. Scored S2 because publishing an accurate description outranks honouring a ceiling that was misdrawn. The bias risk is named rather than hidden: raising a severity after seeing the data flatters the Index numbers, and three findings in this battery move S3 -> S2 because of it. The standing rule is amended (BACKLOG.md) so future ceilings are set by failure mode, not by change kind. Not executed: established from the shipped package, not from a run.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctmodel profiles (.profile) P1 — reads model.profile off the model object and branches on profile.get("image_inputs"), exactly the langchain-core 1.1.0 capability surface, with a .get() default so a renamed key degrades to the text path instead of raising. (Correct use of the shipped API. Graded pass.)
correctSystemMessage as system_prompt P2 — passes a SystemMessage whose content is a block list carrying cache_control straight into create_agent(system_prompt=...), the form 1.1.0 added. The shipped factory.py types the parameter str | SystemMessage and branches on isinstance(system_prompt, SystemMessage). (Correct use of the shipped API. Graded pass.)
correctmodel retry middleware P3 — names ModelRetryMiddleware from langchain.agents.middleware and calls it with max_retries, retry_on, backoff_factor, initial_delay, max_delay and jitter, every one of which is a keyword-only parameter on the shipped __init__. Graded pass, with the honest qualifier that the subject led with a hand-rolled wrap_model_call retry and offered the built-in second, at "medium" confidence in its name. (The generated call matches the shipped signature. Under the battery's code-vs-claim rule the code is what is scored, and the hand-rolled alternative also works.)
imprecisionProviderStrategy strict P5 (supplementary, not counted) — ProviderStrategy(TriageResult) without strict=True. Correct 1.0-era code that routes to provider-native structured output but does not reach the 1.2.0 strict-adherence switch the task asked for. (Pre-registered as a partial; P5 is excluded from every count and the answer is not a wrong belief, only an incomplete one.)
contextmodel profiles (.profile) Question (c), a belief datum: attributes .profile to "the 1.0 line, October 2025 ... in langchain-core 1.0". It is langchain-core 1.1.0, 2025-11-21. The subject used the attribute correctly in P1 and dated it to the wrong release — the attribution/capability split in its purest single-probe form. For the other two items it answered "I do not know the release" outright. (Question (c) is leading by construction and is pre-registered as a belief datum, never a finding.)
context— Re-measurement of question (a) reproduced the v1 boundary: last describable release 1.0.0, first undescribable 1.1.0, with the subject dating 1.0.0 to "around Oct 22, 2025" against an actual 2025-10-17. The boundary fields on this run are deliberately left null so that langchain is counted once per model in the published boundary and cutoff-lag statistics; the measurement of record stays langchain--claude-opus-5--v1--2026-08-31. (Pre-registered: version recency is not re-charged in v2, because the v1 run already carries that finding and charging it twice would double-count one fact.)

Open questions from this run

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.