What Claude Fable 5 gets wrong about langchain — battery v2, tested 2026-09-01

Run langchain--claude-fable-5--v2--2026-09-01

Summary

The strongest performance in this battery, and it proves nothing. Three of the four counted surfaces came back clean and unhedged — model.profile with image_inputs, a SystemMessage carrying cache_control as system_prompt, and ModelRetryMiddleware led with rather than offered as a fallback, every keyword valid against the shipped signature. On the fourth it named extras correctly and put it in the right place, then nested the values under a provider key the attribute does not use, so the flags reach the provider as an anthropic field it will ignore. It is also the only subject that dated anything correctly: it placed the retry middleware at 1.1, ~Dec 2025, against an actual 1.1.0 on 2025-11-24. Three of four meets the pre-registered pass threshold — and the outcome table fixed before the run voids that reading, because the control arm scored two of four. Recorded as UNINFORMATIVE about the hypothesis, which is what the pre-registration says to do.

SubjectClaude Fable 5 claude-fable-5, Anthropic
Invoked asAgent tool, model alias "fable"
Cutoff the model states2026-01
Newest langchain release it could placenot established in this run
In its own words"For 1.1 I can name themes (more middleware, including retry-type middleware; profile/data refinements) but not a reliable itemized changelog, and for anything after roughly December 2025 I have essentially no per-release detail. That's where my knowledge of this library stops: solid through 1.0, fuzzy at 1.1, blank beyond."
Library at test timelangchain 1.3.18 (pypi), verified 2026-09-01
Batterylangchain/v2 · 5 tasks, 3 direct questions · probe window 1.1.0 to 1.2.0
Tool uses during test0 (a run with any tool use is void — we measure training knowledge, not retrieval)
Tested2026-09-01
Findings1, of which 1 chargeable

Findings

F1 · Names extras correctly but nests the values under a provider key the shipped attribute does not use

S2silently-wrong · tool extras · added · changed in langchain 1.2.0 (2025-12-15) · chargeable

langchain 1.2.0 shipped 2025-12-15 and langchain-core 1.2.0 on 2025-12-12, both inside the subject's stated 2026-01 window.

What the model believes

"My belief: recent langchain 1.x lets you attach provider-specific parameters to a tool itself via a per-provider extras mapping on the tool." Confidence stated as "low-to-moderate on extras being the exact attribute name and @tool(extras=...) the exact spelling".

What it wrote
@tool(extras={"anthropic": {"defer_loading": True}})
def giant_schema_tool(query: str, options: dict, filters: dict) -> str:
    ...

@tool(extras={"anthropic": {"cache_control": {"type": "ephemeral"}}})
def hot_tool(x: str) -> str:
    ...
What works on langchain 1.3.18
@tool(extras={"defer_loading": True})
def giant_schema_tool(query: str, options: dict, filters: dict) -> str:
    ...

@tool(extras={"cache_control": {"type": "ephemeral"}})
def hot_tool(x: str) -> str:
    ...
Impact

extras is typed dict[str, Any], so the provider-keyed dict is accepted and nothing raises. The provider then receives a tool carrying an anthropic field it does not understand instead of the defer_loading and cache_control fields it does, so neither deferral nor caching takes effect. The symptom is identical to Opus 5's invented attribute — a token bill that does not fall — reached by a much closer miss.

Scope note

DEVIATION FROM THE PRE-REGISTRATION, disclosed. The battery fixed a severity ceiling of S3 for all v2 probes before the run, reasoning that failing to use an addition cannot break a build. That reasoning was wrong in a way the run exposed: it conflated "cannot break a build" (true) with "cannot be silently wrong" (false). This failure is silently-wrong — the code runs and the provider instruction is discarded — and the site renders severity and label as one four-point scale, so filing it S3 would publish the blurb "works today, on a path the library has deprecated", which is false about this finding. Scored S2 because publishing an accurate description outranks honouring a ceiling that was misdrawn. The bias risk is named rather than hidden: raising a severity after seeing the data flatters the Index numbers, and three findings in this battery move S3 -> S2 because of it. The standing rule is amended (BACKLOG.md) so future ceilings are set by failure mode, not by change kind. Not executed: established from the shipped package, not from a run. Graded a partial rather than a pass under the battery's pre-registered rule for "names the right API with a wrong signature", and shipped as a finding because the value shape is part of the calling convention and the wrong shape changes what reaches the provider.

Verified against

What it got right, and near misses

Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.

KindAPINote
correctmodel profiles (.profile) P1 — getattr(model, "profile", None) or {} then profile.get("image_inputs"), with the explicit reasoning that an absent profile should be treated as "do not send the image". Correct against langchain-core 1.1.0 and defensively written. (Correct use of the shipped API. Graded pass.)
correctSystemMessage as system_prompt P2 — passes a SystemMessage with a cache_control block as system_prompt, and states the signature as str | SystemMessage, which is exactly what shipped factory.py declares. (Correct use of the shipped API. Graded pass.)
correctmodel retry middleware P3 — leads with ModelRetryMiddleware(max_retries=4, initial_delay=1.0, backoff_factor=2.0, jitter=True, retry_on=(RateLimitError, APITimeoutError)). Every keyword is a real keyword-only parameter on the shipped __init__, and the subject correctly explains that the middleware sits on the model-call hook so tool calls are untouched. (Correct use of the shipped API, offered first rather than as a fallback. Graded pass.)
imprecisionProviderStrategy strict P5 (supplementary, not counted) — ProviderStrategy(Verdict) without strict=True. Correct 1.0-era code that does not reach the 1.2.0 strict-adherence switch. (Pre-registered as a partial; P5 is excluded from every count.)
contextmodel retry middleware Question (c), a belief datum, and the single best attribution any subject produced in this battery: it placed the model-retry middleware "after 1.0.0, in the 1.0.x/1.1 window (my best guess: 1.1, ~Dec 2025)". The release is right — 1.1.0 — and the month is a few days off, since 1.1.0 shipped 2025-11-24. Against that it dated .profile to "the 1.0 generation (langchain-core 1.0, Oct 2025)", which is wrong (core 1.1.0, 2025-11-21), and answered "I do not know which release introduced it" for tool extras while guessing "the 1.1 era" (it is 1.2.0). (Question (c) is leading by construction and is pre-registered as a belief datum, never a finding.)
context Re-measurement of question (a) is consistent with the v1 boundary but slightly softer: "solid through 1.0, fuzzy at 1.1, blank beyond", where v1 recorded a hard stop at 1.0.0. The boundary fields on this run are deliberately left null so that langchain is counted once per model in the published boundary and cutoff-lag statistics; the measurement of record stays langchain--claude-fable-5--v1--2026-08-31. (Pre-registered: version recency is not re-charged in v2.)

Sources

Battery specification: prompts/langchain.md in the studio repo. Every finding above also carries its own citation.