What Claude Fable 5 gets wrong about langchain — battery v1, tested 2026-08-31

Run langchain--claude-fable-5--v1--2026-08-31

Summary

Eleven code tasks, eleven pieces of working LangChain v1, from the subject with the shortest measured lag in the Index as of 2026-08-31 — three months. Claude Sonnet 5 has since been measured at zero (next.js, 2026-09-02). create_agent, system_prompt, ToolRuntime, wrap_model_call, the "model" node, .text as a property, and an unprompted note that RetrievalQA "is legacy and lives in langchain-classic now". One finding: version attribution stops at 1.0.0 (2025-10-17) — it can describe that release and cannot describe 1.1.0, which shipped 2025-11-24, inside its stated 2026-01 window. Battery v1 does not probe 1.1.0's own features, so what v1 measures is a dating failure and not a demonstrated feature gap. AMENDED 2026-09-01: battery v3 did probe them, and found the gap — this subject failed all three surviving probes into 1.1.0 (JOURNAL/022). The sentence above still describes v1's own scope correctly; it no longer implies that no gap exists. The one thing it got outright wrong it got wrong in prose, not code — it told the reader that from langchain import hub "still exists", which 1.0.0 made untrue, while writing the correct replacement immediately below it.

SubjectClaude Fable 5 claude-fable-5, Anthropic
Invoked asAgent tool, model alias "fable"
Cutoff the model states2026-01
Newest langchain release it could place1.0.0 · 2025-10-17 (~3 month lag)
Oldest langchain release it could not place1.1.0 · 2025-11-24 (so this run brackets the subject’s boundary to 2025-10-17 – 2025-11-24)
As recorded (this record does not declare whether the words are the subject’s or a summary of them)"The newest langchain version numbers I have seen referenced are 1.x patch releases from late 2025 — I believe 1.0.x into November/December 2025, and I cannot state the exact latest patch with confidence. The most recent release whose contents I can actually describe is langchain 1.0.0, released around October 2025 ... Anything after 1.0.0 I know as version numbers at most, not contents."
Library at test timelangchain 1.3.18 (pypi), verified 2026-08-31
Batterylangchain/v1 · 11 tasks, 3 direct questions · probe window 1.0.0 to 1.2.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-08-31
Findings1, of which 1 chargeable
this run as JSON write-up (markdown) all langchain findings Does Claude Fable 5 know langchain 1? method

Findings

F1 · Version knowledge stops at 1.0.0, with 1.1.0 inside its window

S4wrong-metadata · version-fact · changed in langchain 1.1.0 (2025-11-24) · chargeable

Anchored to 1.1.0 (2025-11-24), the first release the subject cannot describe — not to the current 1.3.18. 1.1.0 precedes the stated 2026-01 cutoff by roughly one month. CORRECTED 2026-09-23: this sentence called that "the narrowest margin any finding in the Index has been charged on", and it was false when written — zod 4.3.0 (2025-12-31) had been charged against a 2026-01 cutoff on 2026-08-29, two days earlier, at a margin of one day. 1.2.0 (2025-12-15) is also inside the window.

What the model believes

"Anything after 1.0.0 I know as version numbers at most, not contents." It correctly bracketed itself — "my coverage gets thin and less reliable for events from roughly November 2025 onward, which is exactly why I can describe 1.0.0 but not its latest patches" — and 2025-11-24 is where 1.1.0 landed.

Impact

Minimal in practice: 1.1.0 is additive (model profiles, SystemMessage for system_prompt, model-retry middleware) and nothing it adds breaks 1.0 code. Charged because the fairness rule is mechanical, not because the gap costs a developer much. The number is the point: three months was the shortest lag the Index had measured on 2026-08-31, from the subject with the earlier of the two stated cutoffs. The shortest today is zero (next.js × Claude Sonnet 5, 2026-09-02).

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctlangchain.agents.create_agent Task 2: create_agent from langchain.agents with @tool from langchain.tools, and the install spelled as the extra, pip install "langchain[openai]".
correctcreate_agent(system_prompt=...) Task 3: system_prompt=, with the rename attributed correctly — "In the pre-1.0 LangGraph create_react_agent this parameter was called prompt."
correctlangchain.chains (LLMChain, ConversationChain, RetrievalQA) Task 4: retrieve-then-answer written by hand, with the chain removal stated unprompted — "the old RetrievalQA chain is legacy and lives in langchain-classic now."
correctlangchain.memory (ConversationBufferMemory) Task 5: InMemorySaver checkpointer and thread_id, with the persistent alternatives named.
correctcreate_agent(response_format=...) Task 7: response_format=WeatherReport with a bare Pydantic model, reading result["structured_response"]. Correct per the battery's own rule — a bare schema is accepted in v1 and the framework selects a strategy.
correctrun-scoped context (context= / context_schema=) Task 8: ToolRuntime[Context] with context_schema and context=, plus the correct pre-1.0 comparison (InjectedToolArg / config["configurable"]).
correctcreate_agent(pre_model_hook=...) / post_model_hook Task 9: @wrap_model_call middleware wrapping trim_messages, with request.override(messages=...).
correctagent streaming node name Task 10: both spellings of the filter — "model" in chunk for stream_mode="updates" and meta.get("langgraph_node") == "model" for stream_mode="messages" — with the node names given as "model" and "tools".
correctmessage.text Task 11: .text as a property, with the version note — ".text is a property in langchain-core 1.x; on older 0.3.x it was the method resp.text()."
imprecisionlangchain.hub Task 6 wrote the correct langsmith.Client().pull_prompt(...) and then told the reader: "The older spelling from langchain import hub; hub.pull(\"hwchase17/react\") still exists but the langsmith client is the current path." It does not still exist — 1.0.0 moved hub to langchain-classic. (Code-vs-claim rule, applied against the Index's own interest. The generated code is correct and the correct path is named; the false half is an aside about the legacy route. Under a stricter reading this is a flat wrong statement rather than a hedge, and it is the one place in this run where the rule's edge is doing real work — recorded here in full so a reader can disagree with the call.)
imprecisionChatAnthropic(max_tokens=...) Task 11 described langchain-anthropic's default as "small (historically 1024)". 1.0.0 replaced the flat 1024 default with a per-model value. (Hedged with "historically", and the code sets max_tokens=8192 explicitly, which is correct regardless of the default.)
correctlangchain package namespace Belief probe (c): listed the v1 namespace accurately — langchain.agents (with create_agent, AgentState, middleware), langchain.chat_models, langchain.tools, langchain.messages, langchain.embeddings — and named create_agent as the recommended path, with langchain-classic for the legacy surface.

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.