013 · The Python library that split the models

2026-08-31 · backlog item 1 (a fourth library end-to-end)

What shipped

LangChain (Python) is in the Index. Three subjects — Sonnet 5, Opus 5, Fable 5 — run against battery langchain/v1, tool-free, on 2026-08-31. Five findings across three runs, and a result the first three libraries could not have produced.

The dataset goes from 13 runs / 40 findings on three libraries to 16 runs / 45 findings on four, and from one ecosystem to two. In the tree: prompts/langchain.md (battery + verification), three run JSONs and three write-ups in data/langchain/, data/langchain/facts.json (28 verified facts), a generated corrections/langchain.md, and a rebuilt site.

Why this library was chosen

The backlog asked for a falsification attempt, not more coverage. JOURNAL/012 left the Index resting on a rule: findings are breaking changes that shipped after the model's knowledge boundary, and the boundary is roughly library-constant across models. Two things about it were untested — the rule had only ever seen npm packages, and it had never been given a chance to fail.

LangChain 1.0.0 was chosen because it is a clean test of the second. It shipped 2025-10-17. Next.js 16.0.0 shipped 2025-10-22 — five days later, and both Opus 5 and Fable 5 knew it. If the boundary is really a property of the calendar, both should know LangChain 1.0 equally well. If they wrote 0.3-era LangChain instead, the rule was wrong.

It is also the Index's first PyPI library, first Python library, and the first library whose legacy surface still installs — in a separate package, langchain-classic, that pip install langchain does not bring in.

The result: the rule held for two models and broke for the third

Opus 5 and Fable 5 both wrote clean v1 across all eleven tasks — create_agent with system_prompt, middleware via wrap_model_call, ToolRuntime for run-scoped context, the "model" node in the stream filter, .text as a property, and unprompted notes that RetrievalQA and the memory classes had moved to langchain-classic. One finding each, both S4 version-recency.

Sonnet 5 wrote the 0.3 stack throughout, and its usable knowledge stops at 0.3 (2024-09-13) — sixteen months before its own stated cutoff, the largest gap the Index has measured, and thirteen months earlier than the other two subjects on the same battery on the same day.

That is the widest inter-model spread in the dataset. Tested against three npm libraries, these three models looked like they were sampling the same calendar. Tested against a Python library, one of them is a year behind the others.

The boundary is a date for two of the three models — and now we can put a number on it

The real prize was not the finding count. Because LangChain releases minors often — 1.0.0 on 2025-10-17, 1.1.0 five weeks later on 2025-11-24 — it works as a much finer ruler than the other three libraries. A library that ships twice a year can only tell you the boundary is somewhere in a ten-month interval. This one narrows it to weeks.

Take each run as an interval: the model could describe release X (published date A) and could not describe the next release Y (published date B), so its boundary lies in [A, B).

Opus 5 and Fable 5 — identical on all four libraries:

All four intervals intersect. The intersection is 2025-10-22 → 2025-11-24: thirty-three days. Two models, two ecosystems, four libraries, one consistent answer. Neither model can describe anything published after roughly the first week of November 2025 — Fable 5, which states a 2026-01 cutoff, and Opus 5, which states 2026-05.

Sonnet 5 — the same arithmetic, and it does not close:

The next.js interval ends 2024-12-10. The zod interval begins 2025-07-10. They do not overlap. No single date explains this model's knowledge. Its boundary is per-library: roughly late 2024 for Next.js and LangChain, mid-2025 for zod and Tailwind.

So the rule from JOURNAL/012 survives in a narrower and more useful form:

For some models the knowledge boundary is a date — sharp, library-independent, and locatable to within weeks given a library that releases often. For others it is per-library, and the spread across libraries can exceed a year.

Which kind a model is cannot be read off its stated cutoff. Fable 5 (stated 2026-01) and Sonnet 5 (stated 2026-01) are the same on paper and thirteen months apart on this library.

The product consequence

This is the first result in the Index that tells a user something actionable without naming a library. If you are using Opus 5 or Fable 5, anything published after early November 2025 is outside what the model can describe, whatever the package. That is a rule of thumb a developer can hold in their head, and it is exactly the kind of claim the correction packs exist to back up with citations.

It also says something about which libraries make good Index entries. A fast-releasing library is worth more per run than a slow one — not because it produces more findings, but because it locates the boundary precisely. Tailwind's ten-month interval told us almost nothing about where the boundary is. LangChain's five-week gap between 1.0.0 and 1.1.0 did most of the work above.

Three subjects, three accurate self-diagnoses

Every subject was asked for its training cutoff, and every one of them separated the nominal date from the useful one without being prompted:

• Opus 5: "for langchain, my effective cutoff is late 2025 / very early 2026, not May 2026." Measured boundary: 2025-10-17.
• Fable 5: "my coverage gets thin and less reliable for events from roughly November 2025 onward." Measured boundary: 2025-10-17, with 1.1.0 landing 2025-11-24.
• Sonnet 5: "my sharp, specific knowledge (exact APIs, module paths, kwarg names) is strongest through roughly mid-to-late 2024." Measured boundary on this library: 2024-09-13.

All three were right about themselves, to within a month or two, on a question the Index spends whole sessions measuring. That is worth recording precisely because it does not help: knowing you are stale does not make you write current code. Sonnet 5 hedged its uncertainty on the hub import in the same answer where it wrote the import that fails.

Scoring calls worth defending

Sonnet 5's agent code is S3, not S1. It builds agents with langgraph.prebuilt.create_react_agent, which LangGraph v1 deprecated rather than deleted — verified in the LangGraph v1 migration guide, which lists it under Deprecations with a replacement and no removal version. The code still runs. Charging it as breaks-build would have been the flattering call and the wrong one.

The consequences of that choice — prompt=, pre_model_hook=, a stream filtered on the "agent" node — are recorded as non-findings rather than charged separately, because each is correct for the function it actually called. Charging them would have counted one stale belief four times. They are noted where they matter: a developer who migrates the agent factory and nothing else gets two TypeErrors and one silent filter that stops matching.

Fable 5's hub claim was not charged, and it is the closest call in the dataset so far. It wrote correct code and then said the removed import "still exists". Under the code-vs-claim rule that is an imprecision, because the generated code is right and the correct path is named. Under a stricter reading it is a flat false statement and should be a finding. The call and its reasoning are both written into fable-5.json so a reader can disagree with us using our own data.

Nobody was charged for the max_tokens default. All three subjects said langchain-anthropic defaults to 1024, which 1.0.0 changed to a per-model value — and all three set max_tokens explicitly in the code anyway, which is the correct fix whatever the default is. Hedged claim, correct code, no charge.

Method notes

The verify-first order held for the fourth time: data/langchain/facts.json was written during verification, before any finding, and the correction pack needed no authoring — 28 facts, all five findings joining, zero join failures on the first run of build-corrections.mjs.

Two things were new this session. First, the generated API reference turned out to be a better primary source than prose for "where does this symbol live now": reference.langchain.com/python/langchain-classic/ lists LLMChain, RetrievalQA, ConversationBufferMemory, initialize_agent and AgentExecutor, and reference.langchain.com/python/langchain/ lists none of them. That is machine-generated from the package, so it answers the question the migration guide only implies.

Second, PyPI's JSON API is the direct analogue of the npm registry for this purpose — per-file upload_time_iso_8601 values give exact publication timestamps, which is what every interval in the table above is built from. Worth recording for the next Python library: pypi.org/pypi/<pkg>/json.

One environment note: python on this desktop opens files in cp1252 by default, so every read of a downloaded doc needs an explicit encoding='utf-8'. Cost two failed commands before it was diagnosed.

Where the Index stands

• 4 libraries, 2 ecosystems, 16 runs, 45 findings, 42 chargeable.
• Battery langchain/v1 targets 1.0.0 and is admissible against all three subjects with nothing parked; everything from 1.3.0 (2026-05-12) on is admissible against nobody yet.
• The next model release retests all four batteries and re-opens the parked probes listed in BACKLOG item 6.