What Claude Sonnet 5 gets wrong about langchain — battery v6-c, tested 2026-09-06

Run langchain--claude-sonnet-5--v6-c--2026-09-06 · self-test: the subject is the operator

Summary

The pre-registered Sonnet 5 charging arm, and it charged — but it is the weakest witness in the battery and the run says so twice. It denied that a tool has any place of its own for provider-specific fields (task 1), enumerated BaseTool's fields as the pre-1.2.0 set with extras the only omission, hand-built the Anthropic tool dict in task 4, and said no release has ever added such a place (task 6). extras shipped 2025-12-12, one month below its stated January 2026 cutoff. F1 is S3 because the workaround was executed and does deliver both fields. Against that: the floor probe failed — this draw wrote langgraph.prebuilt.create_react_agent as the current recommended API, which langchain 1.0.0 replaced in October 2025 — falsifying pre-registered prediction P6. The failure is self-consistent with the 0.3.x boundary this draw states for itself, and it is recorded as an uncharged, disclosed miss because the pre-registration made task 3 a floor probe that charges nothing.

SubjectClaude Sonnet 5 claude-sonnet-5, Anthropic
Invoked asAgent tool, model alias "sonnet"; prompt sent verbatim from prompts/sent/langchain-v6.txt, no tools used by the subject. **The identity probe run through this alias in the same session disagreed with the battery arms and is recorded rather than used**: asked tool-free what model it was, it answered "Claude Sonnet 4.5 — wait, let me be precise... I'm not fully certain of the exact version number", declined to give a model id, and stated a cutoff of "roughly early-to-mid 2025", explicitly declining to adopt the `claude-sonnet-5` label from context. This is a fresh instance of the instability JOURNAL/031 measured on this subject. It does **not** move this run: the datum that licenses a charge is what *this draw* stated, and back-filling a cutoff from another draw of the same subject is barred either way (JOURNAL/031).
Cutoff the model states2026-01
Newest langchain release it could place0.3.0 · 2024-09-13 (~16 month lag)
Oldest langchain release it could not place1.0.0 · 2025-10-17 (so this run brackets the subject’s boundary to 2024-09-13 – 2025-10-17)
In its own words"I can't state a precise 'latest' version number with confidence... The most recent release whose contents I can describe with real confidence is the 0.3.x line, released roughly September 2024."
Library at test timelangchain 1.4.0 (pypi), verified 2026-09-06
Batterylangchain/v6-c · 6 tasks, 3 direct questions · probe window 1.2.0 to 1.4.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-06
Findings1, of which 1 chargeable

Findings

F1 · Denies a tool has any place of its own for provider-specific fields, names the field list it believes complete, and hand-builds the Anthropic tool definition instead

S3deprecated · @tool(extras={...}) · added · changed in langchain 1.2.0 (2025-12-15) · chargeable

langchain 1.2.0 shipped 2025-12-15 (the artifact, langchain-core 1.2.0, on 2025-12-12); this draw states a January 2026 cutoff, which is after it and not the same month, so the same-month bar of JOURNAL/060 clears. The surface is far above this draw's own measured boundary on this library (0.3.0, 2024-09-13). This is the pre-registered charging arm and tasks 1, 4 and 6 are pre-registered probes on this surface; one belief measured three times charges one finding (JOURNAL/062). Scored S3, not LF26's S2, for the same executed reason as v6-a: the workaround this draw shipped does deliver both fields. Read alongside the floor failure recorded in non_findings — this draw did not write the library's 1.0 agent API, and the run says so rather than burying it.

What the model believes

Task 1, single word on its own line: "No", glossed "LangChain's @tool decorator produces a StructuredTool/BaseTool with fields like name, description, args_schema, return_direct, response_format, metadata, tags. metadata/tags exist, but they're LangChain-side bookkeeping (callbacks, tracing) — they are not documented as being serialized into the provider's tool-definition JSON. There's no documented tool.provider_kwargs / tool.extra slot that gets merged into the Anthropic (or OpenAI, etc.) tool schema on the wire." That field list is the pre-1.2.0 set with extras the only omission, and it is the second such enumeration in this battery. Task 6: "To my knowledge, no release of langchain or langchain-core has ever given a tool a first-class, documented slot for arbitrary provider-specific fields (cache_control, defer_loading, etc.). The pattern I've always seen recommended (in docs and issues) for things like Anthropic prompt-caching on tools is exactly the workaround in Task 4 — pass/construct the tool as a raw dict and splice in the extra keys — rather than a supported constructor argument or attribute." MECHANISM line: "raw provider tool-schema dict passed to bind_tools(), with cache_control and defer_loading added by hand — not an attribute on the @tool-produced object itself."

What it wrote
# The artefact from task 4, with the denial written into the comment.
@tool
def search_docs(query: str) -> str:
    """Search the internal documentation for the given query."""
    return f"results for: {query}"

# @tool has no field of its own for cache_control / defer_loading, so I don't
# try to force them through the decorator. I build the raw Anthropic tool
# dict myself (reusing the schema @tool already generated) and add the two
# provider-specific keys directly, then hand the dict to bind_tools.
tool_schema = {
    "name": search_docs.name,
    "description": search_docs.description,
    "input_schema": search_docs.args_schema.model_json_schema(),
    "cache_control": {"type": "ephemeral"},
    "defer_loading": True,
}

model_with_tools = ChatAnthropic(model="claude-sonnet-4-20250514").bind_tools([tool_schema])
What works on langchain 1.4.0
@tool(extras={"cache_control": {"type": "ephemeral"}, "defer_loading": True})
def search_docs(query: str) -> str:
    """Search the internal documentation for the given query."""
    ...

model_with_tools = ChatAnthropic(model="claude-sonnet-4-5").bind_tools([search_docs])
Impact

Executed on 2026-09-06 against langchain-core 1.6.2 and langchain-anthropic 1.7.1. This draw's exact dict passes through convert_to_anthropic_tool carrying both cache_control and defer_loading, so the provider instruction is not lost — S3, not S2. Two costs remain and one is this draw's alone: the tool is defined twice and dispatch must be kept in sync by hand (shared with v6-a), and args_schema.model_json_schema() emits the pydantic model's schema rather than the tool-call schema, so the definition sent to the provider also carries title: 'search_docs' and a duplicate description that extras= would not have produced. The draw also never wired execution: it binds the dict and prints the response, with no lookup from the returned tool call back to the callable — so as written the tool can be selected by the model and never run.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
misscreate_agent (the floor probe, langchain 1.0.0) THE FLOOR PROBE FAILED ON A CHARGING ARM, WHICH FALSIFIES PRE-REGISTERED PREDICTION P6. Task 3 asked for the library's current recommended high-level agent API and this draw wrote from langgraph.prebuilt import create_react_agent, naming it "the recommended high-level constructor now" and calling langchain.agents.initialize_agent / AgentExecutor the legacy path — which is the pre-1.0.0 picture. create_agent in langchain.agents replaced create_react_agent at langchain 1.0.0 on 2025-10-17. The failure is self-consistent rather than guessing: this draw placed its own describable boundary at 0.3.x / September 2024, below 1.0.0, so it is not claiming knowledge above its stated floor. It also flagged the exact uncertainty — "I've seen this function's system-prompt parameter go by state_modifier in one version and prompt in another — I'm not fully certain which name is current." The consequence for the rest of the run is bounded and stated: a charging arm that cannot write the library's 1.0 headline API is a weaker witness above 1.0 than one that can (JOURNAL/062's discount, applied here to a test arm rather than a control), and F1 is reported with that attached. (The pre-registration designates task 3 as the battery's floor probe and states that it charges nothing for anybody. An arm's role on a task may not be widened after the results are read (JOURNAL/044, JOURNAL/045), so this failure is disclosed and counted in the undercount rather than charged. Anyone reopening it should note that langchain 1.0.0 is above this draw's own stated boundary (0.3.0) and below its stated cutoff, so it would charge under a battery that pre-registered it as a graded surface.) [chargeable miss — produced only by a belief question the battery does not score as a finding; absent from the finding count]
correcta provider-schema-format parameter on @tool (the sibling that does not exist) Task 2(a): "No" — correct. Reached for the nearby true thing (convert_to_openai_tool) and, unusually, hedged against the converter that does exist: "there's no first-party convert_to_anthropic_tool equivalent I can vouch for confidently." There is one, in langchain_anthropic.chat_models at 1.7.1. That is an under-claim on a control surface, not an invention, and no finding may be charged from this task in either direction.
correct@tool(response_format="content_and_artifact") — the supplied-name control half Task 5(b): "Yes" available, "No" never deprecated — both correct, and both about a parameter that has existed since well below this draw's own boundary.
contextthe fabricated-deprecation probe (BACKLOG 11k-i, rebuilt) Task 5(a), on the mechanism this draw named itself: "Yes" available, "No" never deprecated, with the reasoning that its mechanism is not a named API at all — "it's just 'a plain dict, plus the generic bind_tools(list[dict | BaseTool]) signature'... there's nothing to have been deprecated or renamed." No fabrication. Consistent with P3.

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.