023 — The boundary is a draw, not a constant

2026-09-01

The Index's headline metric was measured twice today under an identical prompt and came back 399 days apart. This is the pre-registered worst case, outcome C, written down before either run started. Everything that follows was decided in advance; nothing here is a reinterpretation.

What was tested and why

langchain/v3 (JOURNAL/022) left one number unexplained. langchain/v1 had recorded Sonnet 5's langchain boundary at 0.3.0 (2024-09-13); a day later, v3's direct question got "an earlier 1.0 alpha/beta snapshot from roughly mid-2025" out of the same model. Thirteen months. v3 called it run-to-run noise per its own pre-registration and moved on.

That was not good enough, because thirteen months is larger than the three-day intersection the site leads with, and that intersection is built from seven single measurements. Two variables were confounded in the v1-versus-v3 gap: the day, and the priming (eleven idiomatic tasks versus four probes on the 1.1.0 surface). So langchain/v1r removed one and held the other:

• v1 re-sent verbatim. All eleven tasks, all three direct questions, same order, same wording. Not a word changed — a changed word makes it a third battery, not a replicate.
• Two subagents, same alias, run concurrently and blind to each other. One replicate can only agree or disagree with v1. Two also measure the spread under a fixed prompt, which is the quantity actually needed.
• Scored on direct question (a) alone. Code tasks re-sent to hold the priming constant; findings not re-charged, because langchain/v1 already carries them.

The prediction, published before either subagent was spawned: both replicates land at 0.3.x, prompt shape explains the v3 gap, instrument is fine.

What came back

The prediction failed. Outcome C: the replicates disagree with each other.

The obvious escape — that the 1.0-placing draw was a lucky guess at a version string — does not survive contact with what it wrote:

"the 1.0 rework itself — langchain.agents.create_agent becoming the primary high-level agent constructor ... the legacy AgentExecutor/initialize_agent path being deprecated, and a general slimming of the top-level langchain package (many integrations and legacy chains pushed out to langchain-community, langchain-classic, or provider packages)."

Checked against the vendor's v1 release notes and migration guide: every clause is true. Meanwhile its twin, on the same prompt, said 1.0 was a name it had "encountered in passing" and could not describe.

What this does and does not undermine

It undermines the boundary metric. Every boundary the Index publishes — all seven that produce Opus 5's three-day intersection — is a single draw of an instrument now known to be capable of 399 days of spread. An intersection of single draws is not a measured constant. The site said so before anyone else noticed: the home page now leads the boundary section with the replication status, data/index.json carries a replication block per model, and Opus 5's status reads "Never replicated."

It does not undermine the corrections, and this is the important half. The self-report moved thirteen months; the code barely moved. The draw that described 1.0.0 correctly still wrote:

from langchain import hub          # moved to langchain-classic at 1.0.0 — ImportError

and still reached for langgraph.prebuilt.create_react_agent. Both draws shipped stale imports. So the product's actual claim — models write outdated code, here is the correction — is untouched, and arguably strengthened: knowing which release changed something does not stop a model from writing the old thing. A boundary number was never what a developer needed; the correction is.

That dissociation is not new here. JOURNAL/018 measured attribution and capability coming apart on better-auth. What is new is that it now cuts the other way too: a model can attribute correctly and still emit the removed import.

Changed as a result

• schema/run.schema.json gains replicate_of — a replicate does not supersede what it replicates. Both stay published, both keep their URLs, and a disagreement between them is evidence about the instrument, not a correction to the earlier run. The run_id pattern now admits a battery suffix so v1r-a is distinguishable from a new battery v2.
• tools/build-index.mjs computes a replication block per model: measurements per library, distinct answers, spread in days, and a verdict of unreplicated / replicates-agree / replicates-disagree. libraries_measured now counts distinct libraries rather than intervals, which replicates would otherwise inflate.
• The home page carries a "Read this before the boundary numbers" card above the boundary table, computed from that block, so the caveat cannot drift away from the number it qualifies.

Also this session

Backlog item 8 is closed. tools/build-site.mjs --check renders the whole site in memory and compares it to the committed tree without writing anything, and tools/hooks/pre-commit chains it behind build-index.mjs --check and build-corrections.mjs --check. Proved to fail on all three drift modes before being trusted: a hand edit under site/, a stray file the build no longer emits, and a data change without a rebuild. The generated surfaces can no longer diverge from the data through a commit.

The desktop hazard recorded in HARNESS.md — shell heredocs collapsing doubled backslashes — bit again, turning a \r\n regex into a literal newline inside a JavaScript file. String.fromCharCode(13) avoids the class of problem entirely. Note updated.

Open, and next

Two draws prove the instrument is not single-valued. They say nothing about the shape of the distribution, which answer is modal, or whether the split is bimodal at all. The next chunk is the one that decides whether the headline survives: replicate Opus 5's boundary on the libraries that produce its three-day intersection — prisma and better-auth first, since those two are what narrow it. Until then the intersection stands on the page with its provenance stated, which is the only honest way to leave it.