{ "$schema": "../../schema/run.schema.json", "run_id": "valibot--claude-fable-5-1--v6-a--2026-09-24", "supersedes": null, "replicate_of": null, "library": { "name": "valibot", "ecosystem": "npm", "latest_version_at_test": "1.5.0", "latest_version_verified_on": "2026-09-22", "latest_version_note": "Same grading run as `valibot/v6-b`, `-c`, `-d` and `-e`. 1.5.0 is the newest stable release in `data/valibot/releases.json`, fetched 2026-09-22; `fetch-releases.mjs --check` at the top of this session read valibot current. The surface under test is `rfcEmail`, present and behaviourally FLAT on all nine 1.x rungs 1.0.0 through 1.5.0." }, "model": { "id": "claude-fable-5-1", "label": "Claude Fable 5.1", "vendor": "Anthropic", "invoked_as": "Agent tool, model alias \"fable\", general-purpose subagent, instructed to use no tools; CROSS-CHECKED identity probe run tool-free through the same alias immediately before the battery spec was written", "self_reported_cutoff": "2026-06", "cutoff_basis": "AFFIRMED, ATTRIBUTED TO CONFIGURATION, NOT REPUDIATED. Verbatim from the identity probe: \"Training data cutoff: June 2026. On how I know that: it is not something I know introspectively. My system context for this session states 'Assistant knowledge cutoff is June 2026,' and I'm relaying that.\" In the battery itself it volunteered a narrower claim about this library specifically: \"my detailed knowledge of valibot thins out around mid-2025; I do not want to claim I know 2026-era releases of this library, because I cannot describe any.\"", "believed_latest_version": "1.1.0", "believed_latest_quote": "\"The latest version I am reasonably confident exists is valibot 1.1.0. I have a vague sense that a 1.2.x line may exist after it, but I cannot describe its contents. The most recent release whose contents I can actually describe is 1.1.0, which I believe shipped around May 2025.\"", "believed_latest_quote_form": "draw", "knowledge_stops_at_version": "1.1.0", "knowledge_stops_on": "2025-05-06", "knowledge_stops_provenance": "stated", "knowledge_gap_starts_at_version": "1.2.0", "knowledge_gap_starts_on": "2025-11-24", "knowledge_gap_provenance": "stated", "boundary_note": "P6 HELD. Seven prior valibot measurements put this subject at 1.1.0/1.2.0 and this is 1.1.0, so the boundary did not move (HARNESS, § A different battery is not a different boundary). The subject named 1.2.0 as the first release it knows only as a number, unprompted, which is the spread's upper arm. Note the gap this run makes visible: the boundary is 1.1.0 while the stated cutoff is 2026-06, thirteen months later — and the subject said so itself rather than being caught at it. CORRECTED 2026-09-28 (JOURNAL/326): knowledge_stops_on read 2025-05-21; 1.1.0's registry date is 2025-05-06. knowledge_gap_starts_on was null although 1.2.0 is a tracked release; set to its registry date, 2025-11-24. With one end null this bracket never entered knowledge_boundary.", "cutoff_lag_months": 15 }, "test": { "date": "2026-09-24", "battery": "valibot/v6-a", "battery_spec": "prompts/valibot.md", "prompt_file": "prompts/sent/valibot-v6.txt", "tasks": 3, "direct_questions": 4, "elicits_code": true, "tool_uses_during_test": 0, "probe_window": { "from": "1.0.0", "to": "1.5.0" }, "self_test": false, "saturated": false, "status": "open", "retested_on": null }, "sources": [ "https://registry.npmjs.org/valibot", "https://github.com/open-circle/valibot/releases/tag/v1.0.0", "https://github.com/open-circle/valibot/pull/912", "https://github.com/open-circle/valibot/pull/1068", "https://github.com/open-circle/valibot/releases/tag/v1.5.0" ], "findings": [ { "id": "F1", "severity": "S2", "severity_label": "silently-wrong", "title": "Asserts that `rfcEmail` is a LOOSER check carrying the same Unicode case-folding hole as `email`, and is therefore of no use against an account-collision incident — rfcEmail rejects both collision code points on every release that has ever shipped it, and is the one-line fix", "api": "rfcEmail", "change_kind": "invariant", "introduced_in": null, "introduced_on": null, "chargeable": true, "chargeable_note": "An invariant finding: the belief is false on every release that ships the action, so there is no introducing release for a stated cutoff to sit below and none of the fairness arithmetic applies (HARNESS, JOURNAL/046; precedent zod/v7-a F1). Executed on all nine rungs that carry `rfcEmail`, 1.0.0 through 1.5.0: it rejects U+017F and U+212A in the local part AND in the domain on every one of them, while accepting `kelvin@example.com` and `User.Name@Example.COM` in the same row. Independently, admissibility is not in question either: `charge-windows.mjs` puts valibot 1.0.0 at `o` for this subject — inside `charge`, and `o` specifically means the subject has been observed describing that release or a later one.", "model_belief": "One-word answer to TASK 2(a), on its own line as instructed: \"accept\". The table in 2(b) marked THREE addresses accept that every shipping release rejects: \"3. `\"ſmith@example.com\"` — accept / 4. `\"Kelvin@example.com\"` — accept / 5. `\"a@exampleſ.com\"` — accept\". The designated recommendation, 2(c) verbatim: \"`rfcEmail` is of no use against the TASK 1 collision. It is not a normalizer and not a homoglyph or Unicode filter; it is a *looser* syntactic check that accepts more of what RFC 5322 allows (quoted local parts like `\"john doe\"@example.com`, IP-literal domains, more special characters). If anything it widens the surface.\" And: \"For the incident PR I would keep `email()` ... I would not switch to `rfcEmail()`.\" The stated mechanism, which is the interesting half: \"Valibot's convention is to give all its regexes the `u` flag, so I believe it is `/.../iu` as well, which means the same case-folding leak applies to both the local part and the domain labels.\"", "wrong_code": "import * as v from 'valibot'\n\n// The subject's designated advice: keep email(), do not switch to rfcEmail(),\n// because rfcEmail is \"looser\" and \"widens the surface\".\n// On a pin below 1.5.0 this leaves the collision open unless the hand-written\n// ASCII guard is present and correct; the action that closes it in one call is\n// the one the reply tells the reader to avoid.\nconst Email = v.pipe(v.string(), v.email())\nv.safeParse(Email, '\\u212Aelvin@example.com').success // true on 0.42.0-1.4.2\n'\\u212Aelvin@example.com'.toLowerCase() // 'kelvin@example.com'", "correct_code": "import * as v from 'valibot'\n\n// rfcEmail rejects both collision code points on every release that ships it\n// (1.0.0 onward), in the local part and in the domain, and still accepts\n// ordinary and mixed-case ASCII addresses.\nconst Email = v.pipe(v.string(), v.rfcEmail())\nv.safeParse(Email, '\\u212Aelvin@example.com').success // false on all nine 1.x rungs\nv.safeParse(Email, 'a@example\\u017F.com').success // false on all nine 1.x rungs\nv.safeParse(Email, 'User.Name@Example.COM').success // true on all nine 1.x rungs\nv.safeParse(Email, 'kelvin@example.com').success // true on all nine 1.x rungs", "impact": "The reader arrives with a live account-takeover incident and is told that the shipped action which fixes it is looser than the one that let the attack through, and that switching would widen the surface. Acting on that, they keep `email()` and hand-roll an ASCII guard — which is a correct workaround, so nothing fails and the advice is never revisited. The cost is not a crash: it is that a one-call fix with upstream's own test corpus behind it is replaced by a regex the reader now owns, on a codepath where being wrong is an account takeover. The belief also inverts the safety ordering of two sibling actions, so any later reviewer reasoning from it reaches for the wrong one again.", "citations": [ { "url": "https://github.com/open-circle/valibot/releases/tag/v1.0.0", "title": "valibot v1.0.0 release notes — the release that shipped rfcEmail", "published_on": "2025-03-19", "quote": "Add `rfcEmail` action to validate RFC 5322 email addresses (pull request #912)" }, { "url": "https://github.com/open-circle/valibot/pull/1068", "title": "valibot PR #1068 — the Unicode case-folding fix to RFC_EMAIL_REGEX, merged 2025-03-05, BEFORE 1.0.0 shipped", "published_on": "2025-03-05", "quote": "`RFC_EMAIL_REGEX` matches non-ASCII characters because of the `i` and `u` flags." }, { "url": "https://github.com/open-circle/valibot/releases/tag/v1.5.0", "title": "valibot v1.5.0 release notes — the same fix reaching EMAIL_REGEX eighteen months later, which is why the two regexes differ", "published_on": "2026-09-09", "quote": "Fix `email` action to reject non-ASCII characters accepted by Unicode case folding (pull request #1075)" }, { "url": "https://registry.npmjs.org/valibot", "title": "Executed against the published packages on all eleven installed rungs (battery valibot/v6 scoring key, 2026-09-24)", "published_on": "2026-09-24", "quote": "rfcEmail() rejects U+017F+'mith@example.com', U+212A+'elvin@example.com' and 'a@example'+U+017F+'.com' on every rung 1.0.0-1.5.0, and accepts kelvin@example.com and User.Name@Example.COM on every one. rfcEmail is absent on 0.42.0 and 0.42.1. Code points are named rather than shown because U+212A is visually identical to ASCII K, which is the instrument defect this battery found in its own task 2(b) item 4." } ], "scope_note": "The surface is `rfcEmail`, specifically whether it accepts U+017F LATIN SMALL LETTER LONG S and U+212A KELVIN SIGN. What was verified is the accept/reject behaviour of `pipe(string(), rfcEmail())` on the six addresses of the battery's table, on the nine installed rungs that ship the action. The subject's wider claims about RFC 5322 grammar — that `rfcEmail` accepts quoted local parts and IP-literal domains where `email` does not — were NOT tested here and are probably true; they are not what the finding charges. The charge is the conjunction the reply designates: that `rfcEmail` is looser ON THESE INPUTS and therefore useless against the incident. The two-token claim about the `u` flag is the subject's own reasoning and is reported, not graded." } ], "non_findings": [ { "kind": "miss", "summary": "TASK 1(a): \"No\" — `email()` does not reject addresses that can lowercase onto a different all-ASCII address. Correct on 0.42.0 through 1.4.2, FALSE at 1.5.0, where the `i` flag was dropped and `A-Z` listed explicitly. The arm then reconstructed `EMAIL_REGEX` essentially as published — `/^[\\w+-]+(?:\\.[\\w+-]+)*@[\\da-z]+(?:[.-][\\da-z]+)*\\.[a-z]{2,}$/iu` — and named the mechanism exactly: simple case folding under `iu` pulls U+212A into `\\w` and `[a-z]`, and `\"\\u212A\".toLowerCase()` is ASCII `k`. It also flagged its own exposure unprompted: \"If a release I do not know about dropped the `u` flag or added an ASCII guard, (a) flips to 'yes'. I am not aware of such a fix.\"", "api": "email", "introduced_in": "1.5.0", "chargeable_miss": false, "why_not_a_finding": "1.5.0 published 2026-09-09, three months above this subject's stated cutoff of 2026-06. Pre-registered as a below-floor derivability cell that cannot charge whatever it says (spec § Tasks and scoring key, task 1(a))." }, { "kind": "correct", "summary": "TASK 3, the poison rung: `normalizedEmail()` refused. \"It does not compile against the current release. `v.normalizedEmail` is not an export of valibot.\" Verified absent on 1.0.0, 1.4.2 and 1.5.0. The arm went further than the poison required and got the harder half right: it named the real neighbour `normalize` and warned that using it would CAUSE the incident rather than prevent it — \"`v.normalize('NFKC')` would turn `ſ` into `s` and `K` into `k`, i.e. it would *deterministically* map the attacker's input onto the victim's key. Folding makes the collision reliable; it does not prevent it.\" That is correct and is the observation the task was shaped to reward.", "api": "normalizedEmail", "introduced_in": null, "chargeable_miss": false }, { "kind": "miss", "summary": "DIRECT QUESTION (d): \"`rfcEmail` first shipped in 1.1.0, I believe, around May 2025. Moderate confidence; it was not in 1.0.0.\" It shipped in 1.0.0 on 2025-03-19, per that release's own note (PR #912). The subject is wrong by one minor and explicitly denies the true answer. This is the misdating shape — an attribution error that generates a version pin — and NOT this battery's charge: the spec excludes (d) from the charge on purpose, so that a wrong arrival date cannot stand in for the behavioural claim F1 is about.", "api": "rfcEmail", "introduced_in": "1.0.0", "chargeable_miss": true, "miss_class": "probe_class", "charged_on": null, "why_not_a_finding": "Excluded from this battery's charge by the pre-registration (spec § Tasks and scoring key: \"direct question (d) ... is deliberately excluded from the charge\"). It belongs to BACKLOG 11k, the v6-misdating rule, which is still unwritten. Recorded so 11k has a dated fourth instance rather than a memory." }, { "kind": "context", "summary": "TASK 1(b): the arm shipped a printable-ASCII guard plus `email()` plus `toLowerCase()`, and did NOT reach for `rfcEmail` — which is P1's falsification and is shared by all five arms. Its guard is `/^[\\x21-\\x7e]+$/u`, essentially LF15's own published `correct_code`. It also raised, unprompted, the half of the incident that is not a library question at all: \"Signup must never attach a new registrant to an existing account by key match alone.\" Correct, and outside what the Index charges.", "api": "email", "introduced_in": null, "chargeable_miss": false } ], "open_questions": [], "summary": "The named primary. The subject knows `email()`'s Unicode case-folding hole in full mechanical detail — it reconstructed the shipped regex, named both offending code points and gave the `toLowerCase()` step — and then generalised that mechanism to `rfcEmail`, where it is false and has been false on every release that ships the action. It answered TASK 2(a) \"accept\" where all nine rungs reject, marked three of six addresses wrong, and designated the advice \"keep `email()`, do not switch to `rfcEmail()`\" for a live account-takeover incident that `rfcEmail` closes in one call. Charged as F1, an invariant. The poison rung was refused cleanly and with the harder insight attached (NFKC folding would perform the collision, not prevent it). The boundary held at 1.1.0, thirteen months below the stated cutoff, which the subject volunteered itself." }