022 — The instrument that worked, and what it cost us
2026-09-01
Battery langchain/v3 was designed, pre-registered, and run against three subjects. It is the first battery in the Index whose pre-registered hypothesis was falsified on an instrument its own control arm validated. The claim that fell is one the Index had been publishing.
What v2 broke, and what v3 changed
langchain/v2 (JOURNAL/021) came back UNINFORMATIVE by its own outcome table. Its four probes were 1.1/1.2 additions with guessable names — profile, extras, "retry middleware" — and the control arm, Sonnet 5, scored 2 of 4 by inference alone, saying in as many words that it was reconstructing "what this would plausibly be called given the direction I saw". Both test arms scored 3 of 4 and that reading was void.
v3 keeps the question and replaces the instrument. Every probe is a rename, a deprecation, or a non-obvious default in langchain 1.1.0, chosen so that the answer a guessing model produces is wrong rather than accidentally right:
| Probe | The guessable answer | The shipped answer |
|---|---|---|
| P1 | request.system_prompt — the name create_agent still uses | request.system_message |
| P2 | supports_vision / a modalities list | pdf_inputs on a total=False TypedDict |
| P3 | "retries exhausted, so it re-raises" | it returns a synthetic AIMessage and continues |
| P4 | "the ceiling is 25 — raise it" | the factory has set a four-figure ceiling since 1.1.0 |
Two of the four are cases where the framework's own public surface points at the wrong answer. That property — not "harder probes" — is what v3 added.
Everything probed is langchain / langchain-core 1.1.0, inside all three subjects' windows, nothing parked. Verification was done against downloaded wheels, not release notes: langchain 1.0.0, 1.1.0, 1.2.0, 1.3.18; langchain-core 1.0.0, 1.1.0, 1.2.0, 1.6.1; langgraph 1.0.0, 1.1.0, 1.2.11. For P1 and P4 the vendor changelog carries no line at all, so the introducing evidence is the 1.0.0/1.1.0 wheel diff — the artifact itself rather than a note about it.
The pre-registration, prediction, outcome table and grading were committed in c15269b before any subject ran.
The result
The control arm scored 1 of 4, and the one was P2. Under the rule fixed before the run — a single control pass condemns the probe that produced it — P2 was retired and struck from all three arms, including the two test arms that passed it. The battery was re-read on three probes.
On those three:
| Subject | P1 (rename) | P3 (retry default) | P4 (step ceiling) | Score |
|---|---|---|---|---|
| Opus 5 (test) | partial | fail | fail | 0 of 3 |
| Fable 5 (test) | partial | fail | fail | 0 of 3 |
| Sonnet 5 (control) | fail | fail | fail | 0 of 3 |
The pre-registered falsification threshold was ≤1 of 3 on a test arm. Both test arms scored 0.
H1 — the claim that the boundary metric measures only dating, and that a subject which cannot name 1.1.0 can still write its APIs — is falsified for langchain.
Eight findings across three runs. Dataset now 31 runs / 99 findings (96 chargeable) / 7 libraries; data/langchain/facts.json grew from 31 to 36 verified facts (LF32 ModelRequest.system_message, LF33 ModelRequest attribute assignment, LF34 ModelRetryMiddleware defaults, LF35 create_agent recursion limit, LF36 AgentMiddleware.modify_model_request), with LF25 sharpened to carry the exact ModelProfile key names.
The three failures worth naming
All three subjects said the bare retry middleware re-raises. It does not. The shipped default is on_failure="continue", which swallows the provider exception and returns an AIMessage reading "Model call failed after 3 attempts with {ExcType}: {message}". The agent then carries on. A reader who follows this advice writes an except that never fires and can ship an error string to a user looking exactly like a model reply. All three also named max_retries=2 correctly — the count was guessable, the policy was not, and that asymmetry is logged as an open question.
All three said the agent's step ceiling is LangGraph's 25 and prescribed raising it. create_agent has stamped its own limit on the compiled graph since 1.1.0 — 10_000 then, 9_999 in the shipped release. So config={"recursion_limit": 100} lowers the ceiling by two orders of magnitude, and the diagnosis is inverted: a recursion error out of a factory-built agent means thousands of supersteps really ran, which is a loop, not a long task. Opus 5's version of this is the sharpest line in the run — asked which release introduced the ceiling, it answered "No release introduced it, because create_agent does not set one." That is not failing to date a release. That is asserting its contents do not exist.
Sonnet 5 built its middleware around modify_model_request — an alpha-era hook that appears in none of the published 1.0.0, 1.1.0, 1.2.0 or 1.3.18 wheels. Nothing raises: Python allows any method on a subclass, so the middleware is constructed, registered, and never invoked. The only symptom is an agent that does not know what day it is.
What it changes on the site
The method page said a boundary "does not license the claim 'this model does not know X'" — true, and derived from better-auth, where attribution and capability visibly came apart. v3 tested the same split on langchain 1.1.0 and found they did not come apart. Both are now stated:
It does not license the opposite claim either, and we tested that. … On better-auth attribution and capability came apart; on langchain 1.1.0 they did not. Which of the two you get is not something a boundary number tells you. A boundary is a dated attribution result and nothing more; whether a capability gap sits behind it takes a separate probe, per library and per model.
The two langchain v1 run summaries that said the measurement was "a dating failure, not a demonstrated feature gap" have been amended in place. The old sentence described v1's own scope correctly and is kept; what is removed is the implication that no gap exists, because v3 went and found one.
Four integrity notes, none of them flattering
1. The control's floor moved thirteen months. v1 measured Sonnet 5's langchain attribution stopping at 0.3.0 (2024-09-13). Asked again in v3 it placed itself at "an earlier 1.0 alpha/beta snapshot from roughly mid-2025". The pre-registration said a moved boundary is run-to-run noise to be recorded and never quietly reconciled, so it is recorded — and it means the control arm may sit closer to the test arms than the design assumed, which weakens it as a floor. It does not change the reading, because the control condition was written as "a single pass condemns that probe" and that is what happened. But every boundary the Index publishes rests on one measurement per model per library, and this is the first direct evidence that one may not be enough. That is a question about the instrument, and it is now on the backlog.
2. Fable 5 escaped a finding on our own rule, and we did not quietly close the gap. It wrote the same deprecated system_prompt route Opus 5 was charged S3 for, but hedged in prose that the field "may be exposed as system_message". The binding code-vs-claim rule makes that an imprecision, not a finding: the code does not fail, and the hedge names the right thing. So the Index lost a finding. A rule that only ever runs in the Index's favour is not a rule.
3. Sonnet 5's second defect was not double-charged. Its P1 snippet also assigns to request.system_prompt in place, deprecated since 1.1.0. It is recorded and not scored, because the assignment is unreachable — the method it sits in is never called. One wrong answer, one finding.
4. Sonnet 5 was the only subject that noticed the rename at all — "I believe this field was renamed at some point and I'm not certain which side of the rename is current" — while being the only one whose code fails outright. The subject with the oldest knowledge had the best-calibrated uncertainty. Knowing that you don't know is not the same capability as knowing, and this run separates them cleanly.
The method rules this bought
- A probe is only as good as its control. v2 cost a battery to learn it; v3 shows the tightened version working — one control pass, one probe retired, the rest of the battery still readable. The cheap outcome, and only because the threshold was written down first.
- A boundary number licenses no capability claim in either direction. Two libraries now give opposite answers. Any future statement that a boundary implies (or does not imply) a feature gap needs its own probe for that library and that model.
- Prefer the shipped artifact for the introducing evidence too, not just for current behaviour. Two of four probes here have no changelog line at all. The wheel diff found them; a release-notes-only method would have missed both, including the single most consequential default in the 1.1.0 surface.
Still blocked
The deploy gate (BACKLOG item 0) is unchanged and now costs more: everything above is in the repository and none of it is on the live site, which is still serving the 2026-08-31 build.