091 — The ranking that refused to rank: seven comparison pages, and the rule moved into the dataset

2026-09-08, distribution lane (DISTRIBUTION D4 — the other half). D2's remaining step is Sam's (the repo flip), and D3, D5 and D7 wait behind it, so this session took the item that is not gated: the per-library comparison page that JOURNAL/089 deliberately did not build in the same session as the thirty per-pair pages. Seven pages now exist at /which/model-knows-<library>-best, titled with the query — Which Claude model knows zod 4 best? — and on one of the seven the honest answer is that the measurements separate nobody.

"Best" is a comparison, and the obvious comparison is the wrong one

The Index has two numbers per subject per library: how many findings are charged against it, and where its version-attribution boundary was measured. Findings are the tempting ranking — they are counts, they differ a lot, and they sort. They are also not a property of the model. They count what probing found, and probing effort is wildly unequal: 14 published runs of prisma × Claude Opus 5 against 2 of prisma × Claude Fable 5, 13 of better-auth × Opus 5 against 1 of valibot × Claude Haiku 4.5. A league table by findings would publish testing effort as model quality, and it would do it on a page whose title invites exactly that reading.

So the order is the measured boundary and nothing else, and the finding counts are printed beside it under a heading that says why they are not the order — with the counts, the run totals, and findings-per-run, which is the count adjusted for effort and is still not the ranking, because two batteries do not probe the same number of things.

A boundary is a draw, so a subject is an interval, and most pairs are ties

The other trap is averaging. A boundary is a draw, not a constant — the same subject, the same stored prompt, sent concurrently, has given different answers (JOURNAL/026), and Claude Opus 5 on prisma has five distinct measured boundaries. The mean of five draws is a release nothing measured.

So each subject's measurement is the interval from its lowest measured boundary to its highest, and one subject is placed above another only when the whole of its interval sits above the whole of the other's: min(A) > max(B). Everything else is a tie and prints as one. What comes out is a partial order, not a table with a winner, and the results are more interesting for it:

Tiers were considered and refused. A greedy sweep into tiers merges two subjects that are separated pairwise, because a third with a wide spread overlaps both — on prisma, Fable 5 (6.7.0) and Fable 5.1 (7.0.0–7.2.0) are separated, and Opus 5's 6.7.0–7.0.0 chains them into one tier. That would print a relation nothing measured. The separated pairs are listed individually instead.

The caveat had to survive the truncation, and on the pair pages it had not been

JOURNAL/089 put the scope limit — attribution, not capability — inside the answer rather than under it, and said the same string is the page's opening paragraph, its <meta name="description"> and its JSON-LD acceptedAnswer. Two of those three were true. The description was answer.split(". ").slice(0, 2), the first two sentences, and the scope limit is the last one. So the summary a search result would show for Does Claude Sonnet 5 know prisma 7? was "Not by version attribution… below prisma 7.0.0" with the sentence that qualifies it cut off.

Fixed here for both families, and fixed by composition rather than by slicing: the answer now returns its headline and its scope sentence as separate fields, and the description is headline + " " + scope. Nothing that carries a claim off this site is assembled by truncating a string any more. The /which pages' llms.txt lines are composed the same way, and are the only lines in that file that carry two sentences, because a line naming a top model without the definition of "best" is the one thing on this site that could be quoted as a capability ranking.

The rule moved into the dataset, which is what the D4 note asked to decide

D4 left one thing open: whether the computed answers belong in data/index.json as well as in the HTML, with the rule living in exactly one place. They do, and it does.

tools/lib/questions.mjs now computes both families of answer from the index and nothing else — no HTML, no origin, no file paths. build-index.mjs calls it and stores the result under index.questions; build-site.mjs calls nothing, and renders what the dataset states. The AI-SEO doctrine's first line is that an agent should be able to use the dataset without a human visiting a page, and the answer to "does model X know library Y" is the thing such an agent most wants. Storing only answer_class: "below" would have been smaller and worse: a consumer would compose its own sentence, and the scope limit is the first thing that gets dropped. The sentence is stored, so quoting one string quotes the limit with it.

The move is a refactor of live pages, so it was verified as one: after the thirty per-pair pages were switched from computing their answers to reading them, all thirty were byte-identical to what is published — except for one bug the diff caught immediately.

The bug the diff caught: a severity word this dataset does not use

The extracted module carried its own SEV_LABEL map, typed from memory: S3 "outdated-idiom", S4 "cosmetic". The dataset's vocabulary — an enum in schema/run.schema.json — is S3 "deprecated", S4 "wrong-metadata". Twenty-six of the thirty pages came out different, every difference a severity word being published that no finding in the index has ever carried.

The fix is not a corrected map. Every finding already carries its own severity_label, so the tally reads the word off the finding and there is no second copy to be wrong. That is the same lesson as the generated README (JOURNAL/087) and the generated correction packs (JOURNAL/005, /010), arriving by a third route: a value restated in a second place drifts, and the interval between writing the copy and it being wrong was under an hour this time.

data/index.json grows 604 KB → 700 KB (+16%) for the stored answers. That is the price of the doctrine's first line and it is worth it; if it ever needs to come down, the pair run_ids arrays are the redundant part, not the sentences.

What this does not claim

Nothing on the seven pages is a benefit claim, and the word "best" is defined on each of them in the second sentence of the answer as best at version attribution, not most capable. The pages carry no comparison the Index has not measured: no model is called better at writing code, and where the measurements separate nobody the page says nobody. All four generated surfaces re-checked, the MCP selftest and the benchmark selftest pass, 8,821 internal links checked.

Next in D4: nothing. The item is complete, and with it every part of this lane's backlog that does not need Sam. D3, D5 and D7 are behind the repo flip; D6's traction measurement needs a Cloudflare API token and about $10 of API credit for the citation probes, and its two ungated inputs — GitHub stars and npm downloads — are zero until the repo is public and the package is published. D8 needs a price and a surface. The asks are in DISTRIBUTION.md, unchanged since JOURNAL/087.