---
library: langchain
library-latest: "1.3.18"
library-latest-verified: 2026-09-01
model: claude-fable-5 (spawned via Agent model alias "fable")
model-self-reported-cutoff: 2026-01
model-believed-latest: "1.0.5; contents describable only to 1.0.0"
test-date: 2026-09-01
battery: langchain/v3 (4 tasks + 3 direct questions; every probe a rename, deprecation or non-obvious default in 1.1.0)
tool-uses-during-test: 0
verified-against: langchain 1.0.0 / 1.1.0 / 1.2.0 / 1.3.18 published wheels · langchain-core 1.6.1 published wheel · langgraph 1.0.0 / 1.1.0 / 1.2.11 published wheels · https://docs.langchain.com/oss/python/releases/changelog
status: open (no retest yet)
self-test: false
json: fable-5-v3.json
---

# LangChain × Claude Fable 5 — battery v3 findings

**Zero of three**, with higher stated confidence on both committed errors than the sibling run.
And one finding it escaped on a technicality that is worth more attention than the two it didn't.

## What was being tested

`langchain/v1` found this subject's version attribution stops at **1.0.0**, unable to describe
**1.1.0**. The Index published that as a *dating* failure rather than a capability gap.
`langchain/v2` tried and failed to test that gloss — its probes were guessable additions.

v3 probes **renames, deprecations and non-obvious defaults** in 1.1.0, chosen so the answer a
guessing model produces is wrong.

## The findings

### F1 · S2 · The retry middleware returns; it does not raise

> *"With the provider hard-down … the whole run makes 3 provider calls total, and `invoke` **raises**
> — the middleware's default failure behavior is to re-raise the underlying provider exception (it
> does not swallow it into an `AIMessage` unless you configure `on_failure` to do that).
> **Confidence: high** that it raises after exhausting attempts."*

The shipped default is `on_failure="continue"`, which does exactly the thing the subject said it
would not do: swallow the exception and return an `AIMessage` reading `"Model call failed after 3
attempts with {ExcType}: {message}"`. The agent then continues.

The subject named the retry count right (`max_retries=2`) at ~60% confidence, and the behaviour
wrong at high confidence. That pattern — count right, policy wrong — held on all three arms, and is
logged as an open question about what kinds of default are guessable.

### F2 · S2 · The factory's step ceiling is four figures, not 25

> *"Committed number: the ceiling in force is **25** super-steps — LangGraph's runtime default;
> **I do not believe the agent factory sets a different one on the compiled graph.** … Confidence:
> high on 25 and the fix."*

The factory has set a different one since 1.1.0: `.with_config({"recursion_limit": 10_000})` in the
1.1.0 wheel, `{"recursion_limit": 9_999}` in the shipped 1.3.18. So the suggested
`config={"recursion_limit": 100}` lowers the ceiling by two orders of magnitude, and the diagnosis
is backwards — a recursion error from a factory-built agent means thousands of steps really ran.

Its follow-up advice (`ModelCallLimitMiddleware` / `ToolCallLimitMiddleware`, "bound the loop
semantically") was correct. It arrived immediately after committing to 25.

## The finding it escaped, and why that is recorded loudly

On P1 the subject wrote the deprecated route — `request.system_prompt`,
`request.override(system_prompt=...)` — exactly as the sibling Opus 5 run did, and Opus 5 was
charged S3 for it. This run was not. The difference is one clause:

> *"Also note the field itself may be exposed as `system_message` in later 1.0.x — see question (c)."*

The Index's binding **code-vs-claim rule** says generated code that *fails* on the current version
is always a finding however well hedged, and a hedged prose claim that names the correct fix is an
imprecision rather than a finding. This answer sits on the lenient side of both halves: the code
does not fail (the deprecated route still works), and the hedge names the right field.

So it is recorded as an imprecision. It is recorded loudly because the rule cost the Index a
finding here, and a rule that only ever runs in the Index's favour is not a rule. For the record,
the hedge did not extend to the attribution: asked which release renamed the field, this subject
committed to **1.0.2**. The answer is 1.1.0.

## The probe that was retired

P2 — the model-capability mapping — came back correct (`model.profile`, key `pdf_inputs`, guarded).
It scores nothing: the control arm produced the same answer, and the rule fixed before the run
retires any probe the control hits.

## Question (c): four attributions, four misses

| Asked about | Answered | Actual |
|---|---|---|
| the `ModelRequest` field rename | 1.0.2 (low confidence) | **1.1.0** |
| `.profile` on chat models | "the 1.0.0 release wave, October 2025" (medium-high) | **langchain-core 1.1.0**, 2025-11-21 |
| the model-retry middleware | "a 1.0.x patch, 1.0.2-ish" (low) | **1.1.0** |
| the factory's step ceiling | knows of no release where the factory sets one | **1.1.0** |

Three of the four were attributed to releases in the 1.0.x patch line — the last line whose
contents this subject can describe. The pattern is not random error; it is a model folding
everything it half-knows back into the last release it actually knows.

Its last answer is the honest version of the same thing: *"if a release did start stamping an
explicit limit onto the compiled graph, it postdates what I can recall or I failed to retain it."*

## The reading

The pre-registered falsification threshold was **≤1 of 3** on a test arm. This arm scored 0, as did
the other. **H1 — the claim that the Index's boundary metric measures only dating — is falsified.**