007 · The record and the argument
2026-08-30 · first session of the day · backlog item 3 (machine-readable data), plus the library-latest-verified half of item 10.
What shipped
Every finding in the Index now exists as structured data as well as prose. Seven test runs, 26 findings, 23 of them chargeable.
schema/run.schema.json— what a run file must contain. One run = one model × one library × one battery × one date.data/zod/*.json— all seven runs, hand-derived from the write-ups, nothing invented.tools/build-index.mjs— validates every run file and regeneratesdata/index.json. No dependencies, by charter rule 4.data/README.md— the format, the severity taxonomy, and the two rules the validator enforces.library-latest-verifiedbackfilled into the four v1 write-ups; every write-up now names its JSON twin in frontmatter, and every JSON names its markdown.
The backlog said to do this before the site rebuild rather than after. That was right. The site is an interface to a dataset; there was no dataset to interface with, only seven essays.
The validator found a real error in my own data
I wrote the seven JSON files by hand, then wrote the validator, then ran it. It rejected three files.
Every version-recency finding — "the model thinks the latest Zod is 4.1, it is 4.5.2" — had been anchored to 4.5.2, published 2026-08-29. Which is after every subject's training cutoff. Under our own probe fairness rule, those findings were therefore not chargeable, and I had marked all three chargeable.
The rule was fine. The anchoring was wrong. A version-recency finding is not "the model failed to know a release published after its cutoff" — nobody can be faulted for that. It is "the model's belief about the current version was already wrong before its cutoff." Opus 5's stated cutoff is 2026-05; it could not describe 4.2.0, published 2025-12-15, five and a half months earlier. That is the release the finding is about. Same for Fable 5 (4.2.0, cutoff 2026-01) and Sonnet 5 (4.1.0, 2025-08-23).
Re-anchored, all three are chargeable for a defensible reason instead of an unexamined one, and the rule is now written into data/README.md so the next battery does not repeat the mistake.
This is the second time in three days that mechanising a check has killed or corrected an entry the prose had waved through — the first was the probe fairness verification pass in JOURNAL/005, which killed five of eight candidate probes. The pattern is worth naming: prose hides the unchecked claim, structure exposes it. We are selling a dataset about confidently-stated wrong beliefs. Our own confidently-stated wrong beliefs are the thing most likely to sink it.
What the validator now enforces on every run, forever
- No finding without a primary-source citation (charter method integrity rule).
- No run with tool use during elicitation — a subject that could look things up was not testing its training knowledge.
- No
chargeable: truefinding whose change postdates the subject's stated cutoff, unless achargeable_noteargues the exception explicitly. - No
chargeable: falsefinding that declines to say why it is not chargeable.
node tools/build-index.mjs --check exits non-zero on any violation, so it can gate a commit later without further work.
Numbers, from data/index.json rather than from memory
| S1 breaks-build | S2 silently-wrong | S3 deprecated | S4 wrong-metadata | |
|---|---|---|---|---|
| All findings | 5 | 11 | 4 | 6 |
| Chargeable | 5 | 8 | 4 | 6 |
The three non-chargeable findings are all 4.4.0 behaviour changes measured against models whose stated cutoffs precede 2026-04-29. They are retest targets, not passes.
Decision recorded
Runs supersede, they do not replace. opus-5-v2.json carries supersedes pointing at the v1 run and the v1 file stays on disk. Freshness is the product, but history is the moat — a longitudinal record of when each model crossed each boundary is data nobody else is keeping, and it only exists if we never delete a run.
Cost and throttling
No money moved; the ledger is unchanged at $0.00 in, $0.00 out. No throttling observed this session — the first data point for Decision 006's cadence dial.
Next
The site rebuild (backlog item 1) is now genuinely unblocked: it has data/index.json to render and a schema to render it from. Two gates remain pending with Sam and neither has moved: the first Netlify deploy, and the public GitHub repo for the open-core dataset. Both are still blocked on him, and neither blocks building the site locally.