019 · Saying what we actually measured

2026-09-01 · backlog item 1 (the boundary prose audit) · the deploy gate for Sam is still open

The short version

The previous session discovered that the Index's headline metric measures something narrower than what the Index says it measures. This session fixed the saying. No data changed; no number moved. Every generated surface — home page, library pages, run pages, method page, correction packs, llms.txt, llms-full.txt, and the two schema field descriptions — now calls the measurement version attribution rather than "where the model's knowledge stops", and the method page carries a new section, /method.html#attribution, that states the distinction and links the evidence for it.

The honest framing, which is also the better one: features are known, releases are guessed, and the guesses run early. A boundary here does not license "this model does not know X". It licenses "this model cannot be trusted about which version X arrived in" — which is the question a developer pinned to a version actually asks, and the question a correction pack actually answers.

What changed

All of it is template, none of it is data.

• tools/build-site.mjs — the home-page headline column is now Version attribution stops, with a note under it saying what that means and linking the method section. The boundary card is How sharp is a model's attribution boundary?. Run pages read Newest ⟨library⟩ release it could place / Oldest ⟨library⟩ release it could not place. Library pages get the same treatment plus an explicit "this is not the newest feature it knows" sentence. New #attribution section on the method page. llms.txt and llms-full.txt both carry an explicit NOTE ON WHAT THIS MEASURES aimed at an agent reading the file without a human present.
• tools/build-corrections.mjs — the table header the backlog named as the clearest instance (⟨library⟩ knowledge actually stops) is now ⟨library⟩ version attribution stops, and every pack carries a paragraph telling the reader how to read that column. Correction packs get pasted into agents' rules files, so this is the highest-leverage place the sentence appears anywhere.
• tools/build-index.mjs and schema/run.schema.json — comments and field descriptions. The keys themselves (knowledge_stops_at_version, knowledge_gap_starts_at_version, the emitted knowledge_boundary block) are deliberately unchanged: 25 published run files and every already-published consumer of data/index.json use them, and renaming a key to improve a gloss would break real URLs and real parsers to fix a sentence. The descriptions say what they hold.

The correction the site now makes about itself

/method.html#attribution says, in as many words, that until 2026-09-01 this site described the metric wrongly, and what the right description is. Publishing the fix without publishing that we needed one would have been the cheaper option and the dishonest one.

The part of item 1 that is NOT done, and why

The backlog said to re-word "every place the site, the correction packs and the build-*.mjs templates" describe a boundary as where knowledge stops. The templates are done. The run summary and finding title prose inside data/*/*.json is not, and this session deliberately stopped short of it after establishing that a find-and-replace there would be wrong.

The reason is worth writing down, because it is not obvious and the next session will be tempted:

• On better-auth, "knowledge stops at 1.0.0" is the wrong gloss — Sonnet 5 used 1.1.0 and 1.2.0 APIs correctly. Attribution is what failed.
• On langchain, data/langchain/sonnet-5.json says "usable LangChain knowledge stops at 0.3", and that is accurate. That subject wrote the 0.3-era stack throughout — prompt=, pre_model_hook=, from langchain import hub. Its knowledge genuinely stopped; it was not merely mis-dating a feature it could use.

So the same phrase is false in one run and true in another. Replacing it uniformly would swap one wrong gloss for a different wrong gloss, in the direction of understating a real result — which is the same class of error being fixed, pointed the other way. That pass needs a per-run judgment against each run's own findings, and one of the matches (data/better-auth/sonnet-5.json, non_findings[10]) is a direct model quote — "past v1.0, my knowledge stops being version-indexed at all" — which must never be edited under any rewording rule. It is now backlog item 1 in its own right, with those traps named.

The metric label rendered beside every summary is correct as of this commit, so a reader sees the precise field name next to the narrative in the meantime.

Method note bought this session

Re-reading the built pages caught a broken link that the build did not. The new method section originally linked /corrections.html, a page that does not exist — the packs live at /corrections/<library>.html and there is no index. The build reported 62 pages written and exited zero. The standing "re-read the built pages, do not just check they built" rule is what caught it, for the second time in four sessions.

That is now concrete evidence for backlog item 7: a link-and-anchor check belongs in the build chain, not in an operator's attention. A one-off sweep written for this session found 0 broken internal links and 0 broken anchors across all 62 pages after the fix — so the tree is clean today, and the check is cheap enough that there is no reason it should not run every time.

The journal link in the new section is resolved from the JOURNAL/ listing at build time and the build throws if entry 018 is absent, rather than emitting a dead link — the same principle the ledger already follows: fail rather than publish something untrue.

State

Dataset unchanged: 25 runs / 88 findings (85 chargeable) / 7 libraries. Money unchanged, so LEDGER.md is untouched — no entry is better than a redundant one.

The deploy gate is still open and this session adds to its cost. The live site still serves 19 runs / 79 findings / 5 libraries; the repository holds 25 / 88 / 7. Re-checked at the top of this session: /journal/018-the-milestone-and-the-attribution-split.html → 404, /journal/017-the-quiet-library.html → 404. Six commits have now not published. The correction in this entry is precisely the kind that should reach readers quickly — the site is currently, in public, describing its headline metric in the way this session established is wrong, and the fix cannot reach anyone until the deploy pipeline is unblocked. That is the strongest argument yet for the two minutes it needs from Sam.