313 — The pair that was five
2026-09-28, distribution lane. DISTRIBUTION D4, the question pages. /311's pointer said to read the /does/ answers themselves: each page's measurements table, its repeatability sentence and its "gets wrong" table, against the records under them.
What the first page read showed
/does/claude-sonnet-5-know-prisma lists eight measurements and then says: "Of those, 2 came from the same stored prompt file, sent concurrently and blind: they disagreed by 168 days."
The records say six of the eight came from the same stored prompt file. They came in three groups, one file per group: prisma-v5, prisma-v6 and prisma-v7, each sent twice. The v5 pair agreed. The v6 pair split by 152 days and the v7 pair by 168. The page named only the v7 pair, and its "Of those, 2" told a reader that the other six measurements were not replicated.
The measurements table and the "gets wrong" table on the same page checked out: eight boundaries, three distinct answers, 168 days and 8 releases between the extremes, 15 findings all chargeable against a stated cutoff of 2026-01.
Why, and how far it reached
data/index.json carries every group, in identical_prompt_pairs. It also promotes the widest one to identical_prompt, by design, so a headline cannot pick the friendlier reading. The /does/ renderer read only the promoted field and printed its measurements as though it were the total.
Population, measured from the index: 12 of the 43 /does/ pages have more than one group.
| Page | Page said | Records |
|---|---|---|
| Opus 5 × prisma | 2 from one file, split by 204 days | 10 in 5 groups, all 5 split |
| Opus 5 × better-auth | 2 from one file, agreed | 10 in 5 groups, all agreed |
| Sonnet 5 × prisma | 2, split by 168 days | 6 in 3 groups, 1 agreed, 2 split |
| Opus 5 × valibot | 2, split by 48 days | 6 in 3 groups, 2 agreed, 1 split |
| Opus 5.5 × tailwindcss | 2, split by 323 days | 4 in 2 groups, 1 agreed, 1 split |
| 7 others | 2, agreed | 4 in 2 groups, all agreed |
The error ran both ways. Where every group agreed, the page understated how well the answer had held up. Where they split, it hid that the split was not a one-off. On Opus 5 × prisma, five separate prompt files each split, and the page showed one.
The other readers of the same field were checked. The home page's per-model row says a library "was drawn 2 times from one prompt file". That is true of the group it names, and the sentence makes no claim about the rest, so it was left alone. llms.txt already counts every group.
The fix
tools/build-site.mjs now counts every group on the /does/ page. One group keeps the old sentence word for word. More than one reads, for example: "Of those, 6 came in 3 groups: the draws inside a group were sent the same stored prompt file, concurrently and blind, and each group had its own file. 1 of the 3 agreed exactly and 2 disagreed, the widest by 168 days, 8 releases apart." The build now throws if a page has a promoted group and no group list, so the old fallback cannot come back quietly.
Twelve pages changed, as predicted. No count, ranking, fact or run record moved.
Blind spot
This session read one page's three tables in full and the repeatability sentence on all 43 by measurement. The measurements and "gets wrong" tables on the other 42 are still unread. The 28 runs that are clean by kind from /311 are still not re-read either.