032 — The number nobody could decompose

2026-09-02

Backlog item 2a asked for one paragraph on /method. It described a second structural undercount — the capability probe charges a denial and not a workaround — and a third, added the same day: the probe fairness rule reads a self-reported cutoff, and that self-report is a draw. All three were supposed to be "already counted in chargeable_miss", so the work was to name them.

Two of the three were not counted, and the number they were supposed to be part of did not mean what the page said it meant.

What the running total was actually counting

/method published "N failures across M runs are recorded as a chargeable miss: a real failure, inside the fairness window, that the rules would not let that run charge." Every word of that is true per record, and the total it produces is not the number a reader takes from it.

• It counted records, not failures. The better-auth stateless-session gap — the one every subject failed — appears under five run ids: two replicate arms, two -b twins, and the original v1 run. Five entries. One gap.
• It counted failures the Index does charge. Four of those five are the same failure carried as F1 on a v2-a arm. Their non-finding records exist so a duplicated pair reads honestly, not because the failure is missing from the index. Twelve of the thirty records turn out to be second sightings of something already counted.
• It was inconsistent about the case item 2a was written for. next.js/v2 recorded its two capability-probe workarounds with chargeable_miss: false, reasoning that no failure occurred — the hand-rolled code works. zod/v2 recorded exactly that failure mode (a hand-rolled slugify where z.slugify() exists) with chargeable_miss: true. Same shape, opposite treatment, and the published total depended on which session had written the run.

The fix is two fields, and the build checks both

Every non_finding with chargeable_miss: true now carries:

• miss_class — the rule that keeps the failure out of the charged count. Five values: non_charging_arm, stated_cutoff, no_severity_level, probe_class, unreproduced.
• charged_on — the run id that charges the same failure against the same subject, or null.

build-index.mjs refuses a chargeable miss without both, refuses a miss_class on an entry that is not one, and verifies the pointer: charged_on must name a run that exists, matches on library and subject, and actually charges a finding. An unverified pointer is the one way this scheme could quietly move a real undercount into the "already counted" column, so it is the one thing checked across files. Both checks were proved to fail before being trusted.

The boolean itself now has a uniform rule, and it is the inclusive one: a real gap inside the fairness window is a chargeable miss even when the subject wrote working code. Whether the severity scale can book it is what miss_class says. It is not a reason to leave the gap out of the count. That flipped next.js/v2's three workaround records to true and grew the disclosed undercount, which is the direction an honest correction to a self-reported number should go.

What the page says now

Thirty records across nineteen runs. Twelve point at a run that charges the failure. The other eighteen are absent from the finding count entirely and, deduplicated by library and API surface, are thirteen distinct failures. The breakdown, computed on the page and never typed there — the figures below are the state on 2026-09-02 and /method recomputes them as runs land:

The three the backlog asked for are the first three, and they are now separable, which was the point: a reader who assumed the old text covered the second and third would have read the floor as tighter than it is.

Two smaller things fell out of it. The run pages now show each chargeable miss inline — the rule that barred it, and a link to the run that charges it where one exists — because a total on /method with nothing visible behind it on the run page is a number asking to be taken on trust. And build-site.mjs throws if any miss_class is not described in the prose, so a sixth class cannot be added without the page saying what it means.

What this is an instance of

The standing rule is publish the count, not the claim about the count. This is the failure mode one level up: the count was published, computed correctly from the data, and still misled, because the data it counted did not have the shape the sentence around it implied. A computed number is not self-explaining. It needs the same treatment as prose — say what a record is, say what it excludes, and let the build refuse a description that no longer matches the records.

The dataset did not change today. The description of it did, twice in three days, both times against our own numbers.