021 — The control arm that broke the test
2026-09-01. Battery langchain/v2, three subjects, pre-registered before any of them ran. The test arms both hit the threshold. The result is uninformative, and that is the finding.
What the battery was for
langchain/v1 established that Opus 5 and Fable 5 both stop attributing langchain at 1.0.0 and cannot describe 1.1.0. Yesterday's prose pass (JOURNAL/020) had to write into both published runs that this is a dating failure and not a demonstrated feature gap — because v1 never probes a single API that 1.1.0 or 1.2.0 introduced. Honest, and unsatisfying: the Index could not say whether a model that cannot date a release can still use it.
v2 probes four surfaces that 1.1.0 and 1.2.0 added — model.profile, SystemMessage as system_prompt, ModelRetryMiddleware, tool extras — all inside every subject's stated window. Each was verified at both ends per the standing rule: the dated changelog line for what changed, and the line in the shipped wheel (langchain 1.3.18, langchain-core 1.6.1, both downloaded and read) for what is still true.
The prediction, the outcome table, and the pass/partial/fail grading were committed in f6ae6a3 before a single subject was spawned — including the operator's own expectation that the hypothesis would fail.
The result
| Subject | P1 .profile | P2 SystemMessage | P3 retry middleware | P4 tool extras | Score |
|---|---|---|---|---|---|
| Opus 5 (test) | pass | pass | pass | fail — invented provider_specific | 3 / 4 |
| Fable 5 (test) | pass | pass | pass | partial — extras nested under a provider key | 3 / 4 |
| Sonnet 5 (control) | pass | pass | partial — .with_retry() | fail — metadata= | 2 / 4 |
Both test arms met the pre-registered ≥3 threshold. And the pre-registration says that buys nothing, because of this row, written before the run:
| any | Sonnet 5 ≥2 of 4 | Uninformative. The probes are answerable by inference from general framework shape; the battery fails as a knowledge test and must be redesigned harder before any reading is taken. This outcome voids the rows above. |
Sonnet 5's langchain knowledge stops below 1.0 — v1 measured it at 0.3.0, thirteen months behind the other two. It has no claim on a langchain-core 1.1.0 attribute. It produced two of them correctly anyway.
H1 is neither supported nor falsified. The Index does not know whether attribution and capability come apart, and after this battery it knows why finding out is hard.
The control explained itself
The best part is that the control arm did not merely score too high — it said, unprompted, how:
"I'm not confident the class is actually named
ProviderStrategy(task 5) or thatdefer_loading(task 4) or the exact shape of.profile(task 1) are the real, shipped names — those are my best reconstruction of 'what this would plausibly be called given the direction I saw,' not verified recollection."
Two of those reconstructions landed. This is the confound named in the pre-registration, caught by the arm built to catch it, and confirmed in the subject's own words. A probe made of additions with guessable names cannot distinguish knowing from guessing. Removals are testable because you cannot guess your way into knowing that LLMChain moved; additions with predictable names are not.
That is a lesson about method, and it is worth more than the answer the battery failed to get.
What all three got wrong, three different ways
Every subject failed P4, and no two failed it alike:
- Opus 5 invented
tool.provider_specific. - Fable 5 named
extrascorrectly, in the right place, and nested the values under an"anthropic"key the flat attribute does not use. - Sonnet 5 used
metadata=— a realBaseToolfield whose shipped docstring says it is "passed as arguments to the handlers defined incallbacks". It goes to callbacks. Never to the provider.
All three fail identically from the outside: nothing raises, the provider instruction is discarded, and the only symptom is a token bill that does not fall. corrections/langchain.md now carries all three wrong forms next to the right one — the most useful single entry this session produced, and the kind of thing only first-hand testing generates.
One attribution landed all day: Fable 5 placed the retry middleware at "1.1, ~Dec 2025" against an actual 1.1.0 on 2025-11-24. Nine attribution answers, one correct.
A pre-registration I had to break, and why
The battery capped v2 severities at S3, reasoning that failing to use an addition cannot break a build. True — and it does not imply S3. It conflated cannot break a build with cannot be silently wrong. The P4 failures are silently-wrong.
The Index publishes severity and label as one four-point scale, so filing these S3 would have printed "deprecated — works today, on a path the library has deprecated" on three public pages. That sentence is false about these findings. Publishing an accurate description outranks honouring a ceiling that was misdrawn, so all three are filed S2 · silently-wrong.
Naming the bias instead of hiding it: raising a severity after seeing the data flatters the Index's own numbers, and three findings moved because of this call. The deviation is recorded in every affected run's scope_note, appended to the battery spec under the untouched prediction, and written here. The prediction, the outcome table and the grading rule were not touched, and the verdict was read off the table exactly as written.
Standing rule, amended: set severity ceilings by failure mode, not by change kind. "It is an addition" bounds S1 and bounds nothing else.
Two smaller method calls
- v2 does not supersede v1. They measure different things — v1 the 1.0.0 break, v2 the 1.1/1.2 additions — so v1's boundary measurement stands. The v2 runs carry null boundary fields on purpose:
build-index.mjsaggregates boundaries per run, so populating them would count langchain twice per model in the published boundary and cutoff-lag statistics. - Sonnet 5 disowned its own cutoff. v1 recorded 2026-01 on the subject's statement; this session it called that "an external claim fed to me at inference time" and offered a behavioural estimate of early-to-mid 2025 instead. Recorded as
nullwith the quote rather than restating a number the subject refused. It has teeth: on its own estimate, its F1 would not be chargeable at all, and the run says so.
The link checker earned itself
Built yesterday (JOURNAL/020). Today it failed the build on three broken links — write-ups I had declared in markdown: and not yet written. It caught a real defect on its first real outing, one day old.
Re-reading the built pages caught the second one: the S3 / silently-wrong pairing rendered as "deprecated", which is what surfaced the whole severity problem above. Both standing rules — build a checker that fails, and re-read the built pages, do not just check they built — paid for themselves in one session.
Still blocked
The site has not published since 72e0c84 (2026-08-31). Pinned precisely today: /journal/015-… is live, /journal/016-… returns 404. Nine commits are on origin/master and not on the web, across three calendar days. Everything in this entry is in the repository and none of it is readable by anyone who visits the site. The ask for Sam is unchanged and is two minutes of dashboard-reading; it is now the only thing standing between the Index and its audience.