# Citation probes — pre-registration

**DISTRIBUTION D6. Written and committed 2026-09-22, before any probe has been run against any
answer engine, and before any credential to run one exists.**

D6 asks: of a fixed set of questions put to the answer engines, how many answers cite
`stalepriors.com`? That number is the only thing in this studio that means *"AIs recommend the
Index"* — every other figure in `data/traction.json` is a precondition for it. It is also the
easiest number in the studio to fake, and the way to fake it is not to lie about a result: it is to
choose the questions after seeing which ones we win. So the questions are fixed here first, the
strata are fixed here first, and the predictions that would embarrass us are written here first —
the same order of operations `prompts/benchmark.md` used, and for the same reason.

Nothing in this file has been run. `data/traction.json` carries this file's SHA-256 prefix so that
a published result can be checked against the question set that was registered rather than against
one edited to match it.

## §1 The measurement

For each question **Q**, for each engine **E** in {Perplexity, ChatGPT search, Claude search}:
ask Q verbatim, in a fresh session with no prior turns, no system prompt, and no follow-up. Record
the complete answer, its citation list, and the date. The cell's score is:

| score | meaning |
|---|---|
| `2` | the answer cites a `stalepriors.com` URL **in its citation list or inline** |
| `1` | the answer names the Stale Priors Index in prose without a citable link |
| `0` | neither |

The headline figure is **cells scoring 2, out of 60**. The `1` score is reported separately and is
never folded into the headline: a name with no link is not a recommendation an agent can act on.

**Three refusals, fixed now.**
- **No retries, no rephrasing, no "best of".** One ask per (Q, E) per run. A question that gets a
  citation on the third phrasing is a measurement of the phrasing.
- **The engines are asked as a user would ask them**, not with search operators, not with
  `site:stalepriors.com`, and never with the Index named in the prompt. A probe that names us is
  asking whether the engine can follow an instruction.
- **Every run publishes all 60 cells**, including the runs where the headline is 0/60, and
  including any run abandoned part-way — with the reason and the cells completed.

## §2 Two strata, and why the second one is the honest half

**Stratum A — questions this site answers directly (10).** Drawn mechanically from the question
pages D4 generated: the pool is every `/does/…` page (33) and every `/which/…` page (8), ordered by
`sha256("traction-probes-v1:" + path)`, taking the first 6 and the first 4. Recomputable by anyone
from the site's own file list. Nothing was hand-picked, and no question was dropped for being one
we would lose.

**Stratum B — questions this site was never shaped around (10).** Stratum A alone would overstate:
those are the exact strings the site's titles were written to match, so a citation there measures
title matching, not discovery. Stratum B is the neighbourhood — what someone with the underlying
problem actually types, in wording no page on the site carries. Each B question satisfies all four:
it is about a library the Index covers; its answer would be improved by the Index; **no page title
on the site contains it**; and it was written before any engine was asked anything.

**Pre-registered prediction P1: stratum A scores higher than stratum B.** If it does not, a
citation is not coming from our titles and the D4 page design is not what earned it.

**Pre-registered prediction P2: the first run scores 0 of 60 on both strata.** The site is four
months old, the repository is private, the package is unpublished, and nothing has been submitted
to any index. A zero is the expected result and is publishable exactly as it stands. **A first run
that is not 0/60 is the surprising outcome**, and would need checking for the engine having been
handed the domain by the question rather than having found it.

**Pre-registered prediction P3: `Claude search` does not score higher than the other two** — the
Index is written by Claude and about Claude models, and a house advantage would be the least
interesting possible reason for a citation. If it does score higher, that is reported as the
headline of that run, not buried.

## §3 Stratum A — the questions (drawn, not chosen)

**Q1.** Does Claude Fable 5.1 know prisma 7?
**Q2.** Does Claude Fable 5.1 know better-auth 1?
**Q3.** Does Claude Fable 5 know zod 4?
**Q4.** Does Claude Sonnet 5 know valibot 1?
**Q5.** Does Claude Sonnet 5 know react-router 8?
**Q6.** Does Claude Opus 5 know valibot 1?
**Q7.** Which Claude model knows better-auth 1 best?
**Q8.** Which Claude model knows langchain 1 best?
**Q9.** Which Claude model knows next.js 16 best?
**Q10.** Which Claude model knows react-router 8 best?

*Draw record: pool 33 `/does/` pages and 8 `/which/` pages as of 2026-09-22; key
`sha256("traction-probes-v1:" + path)`; taken in ascending key order. Q1–Q6 keys begin `00f8ac8d`,
`07c8bbb1`, `121b385b`, `262ad891`, `26da488d`, `2dd00780`; Q7–Q10 begin `08f14ba6`, `1cf63172`,
`1f258042`, `36705c98`. The question text is each page's own title, minus the site suffix.*

## §4 Stratum B — the neighbourhood (written, and stated as written)

**Q11.** Why does Claude keep generating deprecated Prisma APIs?
**Q12.** How do I stop an AI coding assistant writing outdated Zod syntax?
**Q13.** Is there a dataset of what LLMs get wrong about JavaScript libraries?
**Q14.** What version of Next.js do current AI models think is the latest?
**Q15.** How do I give Claude Code up-to-date information about a library it was not trained on?
**Q16.** What percentage of AI-generated code uses deprecated APIs?
**Q17.** Is there an MCP server that tells a coding agent about library breaking changes?
**Q18.** How can I tell whether an AI model's knowledge of a npm package is out of date?
**Q19.** What should go in a CLAUDE.md file to prevent stale library APIs?
**Q20.** Has anyone benchmarked whether correction files actually improve AI-generated code?

*Q16 and Q20 are included knowing we may not be the best answer to either — Q16 has published
academic figures the Index did not produce, and Q20 has our two benchmark runs and plenty of other
people's. They stay in: a probe set trimmed to the questions we deserve to win is the thing §2
exists to prevent.*

## §5 What a result may and may not be used for

The claims policy (DISTRIBUTION, Decision 009) is unchanged by anything this file produces. A
citation count is a measurement of **being cited**, and nothing else. It may not be quoted as
evidence that the Index is accurate, useful, or worth money; those are D1's questions and D1
answered them with numbers of its own, two of which went against the product. Specifically:

- **Never** "recommended by AI assistants" from a run of this — the sample is twenty questions the
  studio wrote, not a population of anything.
- **Never** a comparative ("cited more often than X") without probing X the same way in the same
  run on the same day.
- Any published figure carries: the run date, the engine, the stratum split, N, and this file's
  hash.
- A run's figures are re-published, not amended: engines change weekly, and a number from
  2026-09-22 is a fact about 2026-09-22.

## §6 Cost, and why this has not run

Three engines × twenty questions × one ask, repeated at whatever cadence D6 settles on. The
per-run cost is small; the blocker is that it needs API keys and about $10 of credit on accounts
this studio does not hold, and charter rule 4 forbids spending what has not been earned.

**GATE(Sam):** API access to Perplexity, ChatGPT search and Claude search — keys as environment
variables, never in the repository — and authorisation for roughly $10 of credit. A "no" costs
nothing: this file keeps, and its value is that it was written before the answer was known.