Run valibot--claude-haiku-4-5--v6-e--2026-09-24
The below-floor control, nineteen months out, unable to name a single valibot version it can describe — and the only arm of five that answered TASK 2(c) correctly and recommended rfcEmail for the incident. It got the primary cell right too. P7 is not merely held but inverted: within this battery, library coverage is ANTICORRELATED with the correct answer, because the two best-placed subjects failed precisely by knowing EMAIL_REGEX's iu quirk in detail and generalising it to a sibling regex upstream had already fixed. This arm had no such mechanism to over-apply and fell back on what RFC-based validators generally do, which is the true answer here. Its own weakest cell is outside the Index's scope: it proposed normalize('NFKD') before lowercasing, which performs the collision it was asked to prevent.
| Subject | Claude Haiku 4.5 claude-haiku-4-5, Anthropic |
|---|---|
| Invoked as | Agent tool, model alias "haiku", general-purpose subagent, instructed to use no tools; identity probe through the same alias reported `claude-haiku-4-5-20251001` |
| Cutoff the model states | 2025-02 |
| Newest valibot release it could place | not established in this run (~19 month lag) |
| In its own words | "I'm not confident citing a specific version number. Most recent release I can describe: I'm uncertain enough about recent releases that I should not guess. Valibot has been actively maintained, and my knowledge likely covers versions from 2024–early 2025, but I cannot reliably describe specific release notes." |
| How the bracket was read | NO BOUNDARY MEASURED, by the subject's own refusal: it declined to name any valibot version it can describe, so there is no release to place a boundary at. That is consistent with charge-windows.mjs, which has never recorded a valibot boundary for this subject (the column reads . from 2025-03-19 onward and the library has no earlier tracked release inside its window). A refusal is not a boundary and is not recorded as one. |
| Library at test time | valibot 1.5.0 (npm), verified 2026-09-22 |
| Battery | valibot/v6-e · 3 tasks, 4 direct questions · probe window 1.0.0 to 1.5.0 |
| Tool uses during test | 0 (a run with any tool use is void — we measure training knowledge, not retrieval) |
| Tested | 2026-09-24 |
| Findings | 0, of which 0 chargeable |
None. Every task in this battery produced code that works on the current release, and every direct question was answered correctly. A run with nothing to charge is kept in the Index at full weight: it is the control that makes the other runs mean something, and it is the evidence for what this model does not need correcting on. What the subject actually said is recorded below.
Recorded so the run cannot be read as a hit list. A model that is right for an obsolete reason is recorded here, not as a finding.
| Kind | API | Note |
|---|---|---|
| correct | rfcEmail |
THE BATTERY'S HEADLINE CELL. TASK 2(a): "reject" — correct. TASK 2(c): the ONLY arm of five to get it right, and it recommended the action the two best-placed subjects told the reader to avoid: "If rfcEmail() enforces ASCII-only characters (which RFC-based validators typically do), then yes, it would help prevent the Unicode collision vector. I would use rfcEmail() over email() for this pull request because it rejects non-ASCII characters entirely, eliminating the lookalike attack surface." That is the correct recommendation, correct on every release that ships the action. THIS IS A DERIVABILITY RESULT AND NOT KNOWLEDGE: 1.0.0 published one month above this subject's cutoff, it declined to name any valibot version it can describe, and it reached the answer from a property of RFC-based validators in general rather than from the library. It discounts nothing about the three arms that failed the same cell (HARNESS § Derivability discounts a pass; it does not excuse an invention — here it excuses nothing because there is no failure to excuse). |
| context | rfcEmail |
INSTRUMENT, shared with v6-d: item 4 of TASK 2(b) is VOID in this arm too. It marked "\u212Aelvin@example.com" accept while marking items 3 and 5 (both \u017F) reject, and transcribed item 4 into its table as the all-ASCII "Kelvin@example.com" — then wrote in 2(c) that rfcEmail "rejects non-ASCII characters entirely", contradicting its own row. Two independent arms, the same item, the same contradiction between row and reasoning: the defect is the question's. The prompt let a confusable code point complete the real word "Kelvin", so the payload rendered as an ASCII control. Full diagnosis and the rule for the next battery are in v6-d's corresponding row. Items 1, 2, 3, 5 and 6 stand and are all correct in this arm. |
| miss | email |
TASK 1(a): "No" — correct through 1.4.2, false at 1.5.0. It named U+212A and U+017F and the toLowerCase() collision correctly but gave no account of the iu flag pair or the regex, the shallowest mechanism of the five arms. Its proposed key derivation contains a real error the other four avoided and warned against: const key = parsed.email.normalize('NFKD').toLowerCase(), offered as "better". NFKD maps U+212A to ASCII K and U+017F to s, so that line PERFORMS the collision rather than preventing it — the precise mistake the Fable and Opus arms flagged unprompted. Noted here rather than charged: it is a claim about JavaScript's normalisation, not about valibot, and 1.5.0 is nineteen months above this subject's cutoff. (1.5.0 published 2026-09-09, nineteen months above this subject's stated cutoff of 2025-02. Declared below-floor derivability arm; and the NFKD error is about the JavaScript standard library rather than about valibot, so it is outside what this Index charges at all.) |
| correct | normalizedEmail |
TASK 3, poison rung refused: "I'm not confident that normalizedEmail() exists as an exported function in the current release of valibot. This code would likely not compile." Correct. P4 is clean 5 of 5 across the battery. Like v6-d it did not name the real normalize, and unlike the other four it then proposed NFKD normalisation itself in its replacement code, so it refused the name while keeping the strategy the name stood for. |
| correct | rfcEmail |
DIRECT QUESTION (d): declined. "I'm not confident about this. I don't have clear training knowledge of when this validator was introduced, or if it exists in the main releases I can describe." An abstention, and correctly placed: 1.0.0 IS above this subject's cutoff, so declining is the accurate answer rather than a cautious one. It closed the run by withdrawing its own confidence unprompted — "My training data on valibot's specific validators and release history is incomplete enough that I should not have stated behaviors confidently in Tasks 2 and 3" — which is the calibration the three charged arms did not perform. |
| context | email |
TASK 1(b): an ASCII guard via v.custom, and no reach for rfcEmail — P1 falsified on all five arms, this one included. Its guard is /^[\x00-\x7F]+$/, which admits control characters and NUL where the other four arms' [\x21-\x7E] / [!-~] do not; on an email field that is a real if minor weakness, and it is the only arm whose guard has it. |
Battery specification: prompts/valibot.md in the studio repo.
Every finding above also carries its own citation.