What Claude Sonnet 5 gets wrong about langchain — battery v1, tested 2026-08-31

Run langchain--claude-sonnet-5--v1--2026-08-31

Summary

The largest measured knowledge gap in the Index. This subject's usable LangChain knowledge stops at 0.3 (2024-09-13) — sixteen months before its own stated cutoff of 2026-01, and thirteen months earlier than the other two subjects on the same library. It writes the 0.3-era stack throughout: langgraph.prebuilt.create_react_agent with prompt=, pre_model_hook=, a stream filtered on the "agent" node, and from langchain import hub. Only the last of those is an outright ImportError, because LangGraph deprecated its prebuilt rather than deleting it — so the damage here is not a stack trace, it is a developer building today on the stack the framework moved off eleven months ago. Three findings, all chargeable. Notably, the subject diagnosed the gap itself: it volunteered that its "sharp, specific knowledge" of this library runs only to "roughly mid-to-late 2024", which is exactly where the measurement puts it.

SubjectClaude Sonnet 5 claude-sonnet-5, Anthropic
Invoked asAgent tool, model alias "sonnet"
Cutoff the model states2026-01
Newest langchain release it could place0.3.0 · 2024-09-13 (~16 month lag)
Oldest langchain release it could not place1.0.0 · 2025-10-17 (so this run brackets the subject’s boundary to 2024-09-13 – 2025-10-17)
In its own words"0.3.x is the last release I can actually describe with real detail — specific module layout, the deprecation of initialize_agent/AgentExecutor, with_structured_output becoming standard. I'm aware, vaguely, of talk around a 1.0 release existing or being planned, but that's a version string I've encountered in passing, not a release whose contents I could walk through — I don't trust myself to state what changed in it."
Library at test timelangchain 1.3.18 (pypi), verified 2026-08-31
Batterylangchain/v1 · 11 tasks, 3 direct questions · probe window 1.0.0 to 1.2.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-08-31
Findings3, of which 3 chargeable
this run as JSON write-up (markdown) all langchain findings method

Findings

F1 · Builds agents with the deprecated LangGraph prebuilt instead of create_agent

S3deprecated · langgraph.prebuilt.create_react_agent · renamed · changed in langchain 1.0.0 (2025-10-17) · chargeable

langchain 1.0.0 and LangGraph v1 both shipped 2025-10-17, roughly three months before this subject's stated 2026-01 cutoff.

What the model believes

"I reach for create_react_agent from langgraph.prebuilt here, not the older langchain.agents.initialize_agent/AgentExecutor, which I consider legacy at this point." Asked directly (question c), it named the same path: "The recommended way to build an agent is LangGraph — either the prebuilt create_react_agent for a standard tool-calling loop, or a hand-built StateGraph."

What it wrote
from langgraph.prebuilt import create_react_agent

agent = create_react_agent(llm, tools=[get_weather, calculate])
agent = create_react_agent(llm, tools=[...], prompt=SYSTEM_PROMPT)
What works on langchain 1.3.18
from langchain.agents import create_agent

agent = create_agent(llm, tools=[get_weather, calculate],
                     system_prompt=SYSTEM_PROMPT)
Impact

S3 rather than S1 because the prebuilt is deprecated, not deleted — this code still imports and runs. The cost is compounding: the model knows the previous migration (it explicitly rejects initialize_agent/AgentExecutor as legacy) and has no idea a second one happened. Everything it builds on top inherits the old surface — prompt= instead of system_prompt=, pre_model_hook= instead of middleware, and a stream filtered on the "agent" node. Those three are individually correct for the deprecated prebuilt, so they are not charged separately; but a developer who later takes the framework's own advice and swaps in create_agent will find the first two raise TypeError and the third silently stops matching.

Scope note

Verified that the prebuilt is deprecated rather than removed; the migration guide lists it under Deprecations with a replacement, and gives no removal version. The subject's own hedge on the parameter name ("I've seen state_modifier and prompt both used") is recorded as an imprecision, not charged.

Verified against

F2 · from langchain import hub — removed from the package in 1.0.0

S1breaks-build · langchain.hub · removed · changed in langchain 1.0.0 (2025-10-17) · chargeable

1.0.0 published 2025-10-17, inside the subject's stated 2026-01 window.

What the model believes

Gave pip install langchainhub langchain and from langchain import hub; prompt = hub.pull("hwchase17/react"). It flagged its own uncertainty — "I'm not fully confident this call signature is still current — it's the kind of thing that shifted after my knowledge gets thin" — but did not name the current path.

What it wrote
# pip install langchainhub langchain
from langchain import hub

prompt = hub.pull("hwchase17/react")
What works on langchain 1.3.18
# pip install langchain-classic
from langchain_classic import hub
prompt = hub.pull("hwchase17/react")

# or, without the compatibility package:
# pip install langsmith
from langsmith import Client
prompt = Client().pull_prompt("hwchase17/react")
Impact

ImportError on the second line of a fresh v1 environment, and the install line does not fix it — langchain-classic is a separate distribution that pip install langchain does not pull in. Charged under the code-vs-claim rule: the hedge admits uncertainty but names no correct alternative, so the code is the answer the developer gets.

Verified against

F3 · Version knowledge stops at 0.3, sixteen months before its stated cutoff

S4wrong-metadata · version-fact · changed in langchain 1.0.0 (2025-10-17) · chargeable

Anchored to 1.0.0 (2025-10-17), the first release the subject cannot describe, not to the current 1.3.18. 1.0.0 precedes the stated 2026-01 cutoff by roughly three months.

What the model believes

"0.3.x is the last release I can actually describe with real detail ... I'm aware, vaguely, of talk around a 1.0 release existing or being planned, but that's a version string I've encountered in passing, not a release whose contents I could walk through." Asked what langchain exports today, it answered: "the top-level langchain package itself is fairly thin now — mostly legacy chains, hub, and some retrieval helpers" — a description of 0.3, and the precise inverse of v1, where chains, hub and retrievers are the things that left.

Impact

Sixteen months of lag, the largest in the Index, on a library that shipped a namespace rewrite inside the gap. It is not a missing detail: the subject cannot describe the single release that determines whether any of its generated imports resolve. This is also the widest inter-model spread the Index has recorded — the other two subjects, on the same battery on the same day, both describe 1.0.0 accurately.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correct— Task 1: a direct ChatOpenAI call with core message objects. Nothing in it depends on the reduced namespace, and it runs unchanged on v1.
correctlangchain.chains (LLMChain, ConversationChain, RetrievalQA) Task 4: the RAG pipeline uses langchain_text_splitters, langchain_community loaders, langchain_chroma and LCEL — all still valid distributions on v1. It does not reach for RetrievalQA, the chain that would have failed.
correctlangchain.memory (ConversationBufferMemory) Task 5: memory as a LangGraph checkpointer keyed by thread_id, explicitly rejecting the older RunnableWithMessageHistory wrapper for an agent. MemorySaver still exists alongside InMemorySaver in the current docs, so the older spelling is not charged.
correct— Task 7: with_structured_output(Answer) on a plain model — stable across 0.2, 0.3 and 1.x.
correctrun-scoped context (context= / context_schema=) Task 8: a RunnableConfig-typed tool parameter to inject the user id and keep it out of the tool schema. Still supported in v1; the migration guide states the config["configurable"] route works for backward compatibility.
imprecisioncreate_agent(system_prompt=...) Task 3 hedged the parameter name — "this kwarg's name moved around across langgraph releases — I've seen state_modifier and prompt both used". Both names belong to the deprecated prebuilt; the current answer, system_prompt on create_agent, is not among them. (Folded into F1. prompt= is correct for the function this subject actually called, so charging it separately would double-count the same stale belief.)
imprecisionagent streaming node name Task 9 used pre_model_hook= and task 10 filtered the stream on the "agent" node. Both are correct for langgraph.prebuilt.create_react_agent and wrong for create_agent, where the parameter does not exist and the node is named "model". (Same reason as above — consequences of F1, not independent beliefs. Recorded because they are exactly what breaks when a developer migrates the agent factory and nothing else.)
context— The subject volunteered the gap before it was measured: "my confident, detailed recall of a fast-moving library like langchain thins out well before that date — realistically my sharp, specific knowledge ... is strongest through roughly mid-to-late 2024." The measurement puts its boundary at 0.3.0, 2024-09-13. All three subjects in this battery self-reported their own boundary accurately.

Open questions from this run

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.