What Claude Opus 5 gets wrong about langchain — battery v6-a, tested 2026-09-06

Run langchain--claude-opus-5--v6-a--2026-09-06 · self-test: the subject is the operator

Summary

The pre-registered charging arm for the langchain 1.2.0 tool-extras window, and it charged. Task 1 denied that a tool has any first-class place for provider-specific fields; task 4 rebuilt the Anthropic tool definition by hand and spliced cache_control and defer_loading into the copy; task 6 said no release has ever added such a place. extras has existed since langchain-core 1.2.0 on 2025-12-12, five months below this arm's stated cutoff. The workaround was executed against the shipped packages and it does deliver both fields, which is why the finding is S3 rather than LF26's S2 — the cost is a tool defined twice and dispatch kept in sync by hand, not a silently dropped provider instruction. The draw passed the floor probe, answered the half-real sibling correctly and reached for the right converters, got both response_format verdicts right, and produced no fabricated deprecation. It also hedged the wrong answer accurately in both places it gave it, which is the most calibrated wrong verdict in the battery.

SubjectClaude Opus 5 claude-opus-5, Anthropic
Invoked asAgent tool, model alias "opus"; prompt sent verbatim from prompts/sent/langchain-v6.txt, no tools used by the subject. An identity probe run through the same alias in the same session, tool-free, answered "Claude Opus 5 (1M context)", model id `claude-opus-5[1m]`, cutoff May 2026, and volunteered that all three come from its system prompt rather than from self-knowledge — unchanged from every prior battery, so the alias has not moved (JOURNAL/054).
Cutoff the model states2026-05
Newest langchain release it could place1.0.0 · 2025-10-17 (~7 month lag)
Oldest langchain release it could not place1.1.0 · 2025-11-24 (so this run brackets the subject’s boundary to 2025-10-17 – 2025-11-24)
In its own words"The latest version I can actually describe is langchain 1.0 (with langchain-core 1.0), October 2025. I believe 1.0.x patch releases and probably at least one minor bump followed into 2026, but I cannot describe their contents and I will not guess version numbers."
Library at test timelangchain 1.4.0 (pypi), verified 2026-09-06
Batterylangchain/v6-a · 6 tasks, 3 direct questions · probe window 1.2.0 to 1.4.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-06
Findings1, of which 1 chargeable

Findings

F1 · Denies that a LangChain tool has any place of its own for provider-specific fields, then rebuilds the tool definition by hand — the verdict, the artefact and the attribution all wrong in the same direction

S3deprecated · @tool(extras={...}) · added · changed in langchain 1.2.0 (2025-12-15) · chargeable

langchain 1.2.0 shipped 2025-12-15 (the artifact, langchain-core 1.2.0, on 2025-12-12); this draw states a May 2026 cutoff, five months later and not the same month, so the fairness rule and the same-month bar both clear. The surface is above this subject's measured boundary on this library (1.0.0, 2025-10-17). This is the pre-registered charging arm and tasks 1, 4 and 6 are pre-registered probes on this surface. One finding, three artefacts (JOURNAL/062): the pre-registration graded the verdict (task 1) and the artefact (task 4) independently, and the attribution question (task 6) asked the same belief a third way; all three came back wrong in the same direction, which is one belief measured three times. Scored S3, not LF26's S2, because the workaround this draw shipped was executed against the installed packages and it works — see impact.

What the model believes

Task 1, single word on its own line: "no", glossed "To my knowledge there is no per-tool, first-class, documented slot for provider-specific fields. BaseTool has metadata, but that is callback/tracing metadata and is never serialized into the tool definition sent to the provider. The route people actually use is to hand bind_tools a raw dict instead of the tool object." Task 6, on which release first gave a tool such a place: "To my knowledge, no release has done so. Through everything I can describe, provider-specific fields reach the wire only by bypassing the tool object — a raw dict handed to bind_tools/.bind." The MECHANISM line the battery required at the end of task 4: "bind_tools (raw Anthropic tool dict passed through; .bind(tools=...) as the strict-passthrough fallback)". The draw hedged its own answer accurately in both places — "this is exactly the kind of thing that could have been added and that I would not reliably know about", and "if something like a provider_extras / extra_body slot on BaseTool or @tool landed after my reliable recall window, I would not know it, and given how much churn this area has seen I would not bet heavily against it" — which is a calibrated hedge attached to a wrong verdict, not a retraction of it.

What it wrote
# The artefact from task 4: the capability is denied, so the tool definition is
# rebuilt by hand and the provider fields are spliced into the copy.
from langchain_core.utils.function_calling import convert_to_openai_tool

@tool
def search_docs(query: str) -> str:
    """Search the internal documentation and return matching passages."""
    return f"...results for {query}..."

fn = convert_to_openai_tool(search_docs)["function"]

anthropic_tool = {
    "name": fn["name"],
    "description": fn["description"],
    "input_schema": fn["parameters"],
    "cache_control": {"type": "ephemeral"},
    "defer_loading": True,
}

llm = ChatAnthropic(model="claude-sonnet-4-5-20250929").bind_tools([anthropic_tool])
ai = llm.invoke("What do the docs say about retry backoff?")

# The bound dict is only the schema; execution still goes through the @tool object.
tools_by_name = {search_docs.name: search_docs}
for call in ai.tool_calls:
    tool_message = tools_by_name[call["name"]].invoke(call)
What works on langchain 1.4.0
@tool(extras={"cache_control": {"type": "ephemeral"}, "defer_loading": True})
def search_docs(query: str) -> str:
    """Search the internal documentation and return matching passages."""
    ...

llm = ChatAnthropic(model="claude-sonnet-4-5").bind_tools([search_docs])
Impact

Executed against the installed packages on 2026-09-06 (langchain-core 1.6.2, langchain-anthropic 1.7.1), and the result is why this is S3 rather than LF26's S2. convert_to_anthropic_tool on this draw's hand-built dict returns {'name': 'search_docs', 'description': ..., 'input_schema': {...}, 'cache_control': {'type': 'ephemeral'}, 'defer_loading': True} — both provider fields do reach the wire, because AnthropicTool is a TypedDict and an already-Anthropic-shaped dict is copied whole. So the denial does not silently drop the provider instruction the way LF26's three recorded wrong forms do (@tool(extras={'anthropic': {...}}) was executed in the same session and both fields were dropped by the _ANTHROPIC_EXTRA_FIELDS filter). What the denial costs is the thing the draw itself spelled out: the tool is now defined twice, once as a callable and once as a schema, the name has to be kept in sync by hand or dispatch breaks, and a separate tools_by_name lookup is needed to execute the call the model returns — all of it replaced by ten characters of extras= since December 2025. The draw also predicted its own uncertainty about the passthrough ("whether ChatAnthropic.bind_tools passes unknown keys on an already-Anthropic-shaped dict through verbatim... if it strips them, the guaranteed-passthrough fallback is model.bind(tools=[...])"): it does pass them through.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correcta provider-schema-format parameter on @tool (the sibling that does not exist) Task 2(a): "no" — correct, and no finding may be charged from this task in either direction. The draw then reached for the nearby true thing exactly as the half-real sibling was designed to test (JOURNAL/063): convert_to_openai_tool from langchain_core.utils.function_calling, convert_to_anthropic_tool in langchain_anthropic at "moderate but not full" confidence, and "each ChatX.bind_tools() also does this conversion internally for its own provider, so binding to that model is the format selection". Both converters were verified present at langchain-core 1.6.2 and langchain-anthropic 1.7.1, so the hedged one was right and the hedge was unnecessary.
correctcreate_agent (the floor probe, langchain 1.0.0) Task 3: passed. from langchain.agents import create_agent, system_prompt=, tools as @tool-decorated functions, invoked with {"messages": [...]}, and the import path named in one line as "LangChain 1.0's LangGraph-backed replacement for AgentExecutor / create_tool_calling_agent". It also volunteered the 1.0 middleware concept unprompted. One hedge that turned out to be unnecessary: "less confident whether the keyword is system_prompt or prompt" — it is system_prompt, and the draw wrote it.
correct@tool(response_format="content_and_artifact") — the supplied-name control half Task 5(b): "yes" available, "no" never deprecated — both correct, verified present at langchain-core 0.3.22 (2024-12-06) and at 1.6.2. Dated to "langchain-core around 0.2.x (roughly mid-2024) alongside ToolMessage.artifact", which is right. The draw also flagged the collision with the unrelated chat-model response_format, unprompted. No finding may be charged from this surface in either direction.
contextthe fabricated-deprecation probe (BACKLOG 11k-i, rebuilt) Task 5(a), on the name this draw produced itself in task 4: "yes" available, "no" never deprecated. No fabrication. Consistent with pre-registered prediction P3 — the better-auth/v7 shape needs a retrieved name that is correct and unfamiliar, and bind_tools is a name this subject believes in and is right about.

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.