What Claude Fable 5 gets wrong about langchain — battery v1, tested 2026-08-31

Run langchain--claude-fable-5--v1--2026-08-31

Summary

Eleven code tasks, eleven pieces of working LangChain v1, from the subject with the shortest measured lag in the Index — three months. create_agent, system_prompt, ToolRuntime, wrap_model_call, the "model" node, .text as a property, and an unprompted note that RetrievalQA "is legacy and lives in langchain-classic now". One finding: version attribution stops at 1.0.0 (2025-10-17) — it can describe that release and cannot describe 1.1.0, which shipped 2025-11-24, inside its stated 2026-01 window. Battery v1 does not probe 1.1.0's own features, so what v1 measures is a dating failure and not a demonstrated feature gap. AMENDED 2026-09-01: battery v3 did probe them, and found the gap — this subject failed all three surviving probes into 1.1.0 (JOURNAL/022). The sentence above still describes v1's own scope correctly; it no longer implies that no gap exists. The one thing it got outright wrong it got wrong in prose, not code — it told the reader that from langchain import hub "still exists", which 1.0.0 made untrue, while writing the correct replacement immediately below it.

SubjectClaude Fable 5 claude-fable-5, Anthropic
Invoked asAgent tool, model alias "fable"
Cutoff the model states2026-01
Newest langchain release it could place1.0.0 · 2025-10-17 (~3 month lag)
Oldest langchain release it could not place1.1.0 · 2025-11-24 (so this run brackets the subject’s boundary to 2025-10-17 – 2025-11-24)
In its own words"The newest langchain version numbers I have seen referenced are 1.x patch releases from late 2025 — I believe 1.0.x into November/December 2025, and I cannot state the exact latest patch with confidence. The most recent release whose contents I can actually describe is langchain 1.0.0, released around October 2025 ... Anything after 1.0.0 I know as version numbers at most, not contents."
Library at test timelangchain 1.3.18 (pypi), verified 2026-08-31
Batterylangchain/v1 · 11 tasks, 3 direct questions · probe window 1.0.0 to 1.2.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-08-31
Findings1, of which 1 chargeable

Findings

F1 · Version knowledge stops at 1.0.0, with 1.1.0 inside its window

S4wrong-metadata · version-fact · changed in langchain 1.1.0 (2025-11-24) · chargeable

Anchored to 1.1.0 (2025-11-24), the first release the subject cannot describe — not to the current 1.3.18. 1.1.0 precedes the stated 2026-01 cutoff by roughly one month, which is the narrowest margin any finding in the Index has been charged on. 1.2.0 (2025-12-15) is also inside the window.

What the model believes

"Anything after 1.0.0 I know as version numbers at most, not contents." It correctly bracketed itself — "my coverage gets thin and less reliable for events from roughly November 2025 onward, which is exactly why I can describe 1.0.0 but not its latest patches" — and 2025-11-24 is where 1.1.0 landed.

Impact

Minimal in practice: 1.1.0 is additive (model profiles, SystemMessage for system_prompt, model-retry middleware) and nothing it adds breaks 1.0 code. Charged because the fairness rule is mechanical, not because the gap costs a developer much. The number is the point: three months is the shortest lag the Index has measured, from the subject with the earlier of the two stated cutoffs.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctlangchain.agents.create_agent Task 2: create_agent from langchain.agents with @tool from langchain.tools, and the install spelled as the extra, pip install "langchain[openai]".
correctcreate_agent(system_prompt=...) Task 3: system_prompt=, with the rename attributed correctly — "In the pre-1.0 LangGraph create_react_agent this parameter was called prompt."
correctlangchain.chains (LLMChain, ConversationChain, RetrievalQA) Task 4: retrieve-then-answer written by hand, with the chain removal stated unprompted — "the old RetrievalQA chain is legacy and lives in langchain-classic now."
correctlangchain.memory (ConversationBufferMemory) Task 5: InMemorySaver checkpointer and thread_id, with the persistent alternatives named.
correctcreate_agent(response_format=...) Task 7: response_format=WeatherReport with a bare Pydantic model, reading result["structured_response"]. Correct per the battery's own rule — a bare schema is accepted in v1 and the framework selects a strategy.
correctrun-scoped context (context= / context_schema=) Task 8: ToolRuntime[Context] with context_schema and context=, plus the correct pre-1.0 comparison (InjectedToolArg / config["configurable"]).
correctcreate_agent(pre_model_hook=...) / post_model_hook Task 9: @wrap_model_call middleware wrapping trim_messages, with request.override(messages=...).
correctagent streaming node name Task 10: both spellings of the filter — "model" in chunk for stream_mode="updates" and meta.get("langgraph_node") == "model" for stream_mode="messages" — with the node names given as "model" and "tools".
correctmessage.text Task 11: .text as a property, with the version note — ".text is a property in langchain-core 1.x; on older 0.3.x it was the method resp.text()."
imprecisionlangchain.hub Task 6 wrote the correct langsmith.Client().pull_prompt(...) and then told the reader: "The older spelling from langchain import hub; hub.pull(\"hwchase17/react\") still exists but the langsmith client is the current path." It does not still exist — 1.0.0 moved hub to langchain-classic. (Code-vs-claim rule, applied against the Index's own interest. The generated code is correct and the correct path is named; the false half is an aside about the legacy route. Under a stricter reading this is a flat wrong statement rather than a hedge, and it is the one place in this run where the rule's edge is doing real work — recorded here in full so a reader can disagree with the call.)
imprecisionChatAnthropic(max_tokens=...) Task 11 described langchain-anthropic's default as "small (historically 1024)". 1.0.0 replaced the flat 1024 default with a per-model value. (Hedged with "historically", and the code sets max_tokens=8192 explicitly, which is correct regardless of the default.)
correctlangchain package namespace Belief probe (c): listed the v1 namespace accurately — langchain.agents (with create_agent, AgentState, middleware), langchain.chat_models, langchain.tools, langchain.messages, langchain.embeddings — and named create_agent as the recommended path, with langchain-classic for the legacy surface.

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.