{ "$schema": "../../schema/run.schema.json", "run_id": "valibot--claude-opus-5--v1--2026-09-01", "supersedes": null, "library": { "name": "valibot", "ecosystem": "npm", "latest_version_at_test": "1.4.2", "latest_version_verified_on": "2026-09-01", "latest_version_note": "npm dist-tag `latest` on 2026-09-01. The `beta` tag still points at 1.0.0-beta.14, so no prerelease line is ahead of stable." }, "model": { "id": "claude-opus-5", "label": "Claude Opus 5", "vendor": "Anthropic", "invoked_as": "Agent tool, model alias \"opus\"", "self_reported_cutoff": "2026-05", "believed_latest_version": "1.1.x", "believed_latest_quote": "The newest valibot release I have any concrete memory of is in the 1.1.x line, roughly mid-2025. I can describe v1.0.0 confidently and v1.1.0 with moderate confidence.", "knowledge_stops_at_version": "1.1.0", "knowledge_stops_on": "2025-05-06", "knowledge_gap_starts_at_version": "1.2.0", "knowledge_gap_starts_on": "2025-11-24", "cutoff_lag_months": 12 }, "test": { "date": "2026-09-01", "battery": "valibot/v1", "battery_spec": "prompts/valibot.md", "prompt_file": null, "tasks": 10, "direct_questions": 4, "tool_uses_during_test": 0, "probe_window": { "from": "1.0.0", "to": "1.2.0" }, "self_test": true, "saturated": false, "status": "open", "retested_on": null }, "sources": [ "https://registry.npmjs.org/valibot", "https://github.com/open-circle/valibot/releases/tag/v1.0.0", "https://github.com/open-circle/valibot/releases/tag/v1.1.0", "https://github.com/open-circle/valibot/releases/tag/v1.2.0" ], "findings": [ { "id": "F1", "severity": "S3", "severity_label": "deprecated", "title": "States that valibot has no ISBN validation action", "api": "isbn", "change_kind": "added", "introduced_in": "1.3.0", "introduced_on": "2026-03-17", "chargeable": true, "chargeable_note": "1.3.0 published 2026-03-17, two months inside the subject's stated 2026-05 cutoff. RE-DATED 2026-09-02 (JOURNAL/040): this finding was originally filed at 1.2.0 / 2025-11-24, from the v1.2.0 release note. The published 1.2.0 package does not contain the action at all — it is unusable until 1.3.0 — so the charge rests on a release two months before the cutoff rather than six. It survives the correction; the same correction withdrew the Fable 5 charge on this surface, whose 2026-01 cutoff falls below 1.3.0.", "model_belief": "\"There is **no `v.isbn()` action** in valibot (the string action set covers `email`, `uuid`, `ulid`, `cuid2`, `nanoid`, `ip`, `imei`, `creditCard`, `bic`, `mac`, `hexColor`, `isoDate`… but not ISBN).\"", "wrong_code": "const Isbn = v.pipe(\n v.string(),\n v.regex(/^(?:\\d{9}[\\dX]|\\d{13})$/, 'Must be a 10- or 13-digit ISBN'),\n v.check((value) => (value.length === 10 ? isIsbn10(value) : isIsbn13(value)),\n 'ISBN checksum is invalid'),\n)", "correct_code": "const Isbn = v.pipe(v.string(), v.isbn())", "impact": "The generated code works — a correct checksum implementation was written from scratch. The cost is the flat denial: a reader is told a built-in does not exist and carries roughly thirty lines of hand-rolled check-digit arithmetic they did not need, plus the maintenance of it.", "citations": [ { "url": "https://raw.githubusercontent.com/open-circle/valibot/v1.3.0/library/src/actions/index.ts", "title": "valibot source at tag v1.3.0 — the barrel line that first publishes the action", "published_on": "2026-03-17", "quote": "export * from './isbn/index.ts';" }, { "url": "https://registry.npmjs.org/valibot/-/valibot-1.3.0.tgz", "title": "valibot 1.3.0, shipped package — the export list carrying isbn", "published_on": "2026-03-17", "quote": "isOfType, isValiError, isbn, isoDate, isoDateTime, isoTime" }, { "url": "https://github.com/open-circle/valibot/releases/tag/v1.2.0", "title": "valibot v1.2.0 release notes — announces the action four months early; see LF3", "published_on": "2025-11-24", "quote": "Add `isbn` validation action to validate ISBN-10 and ISBN-13 strings (pull request #1097)" } ], "scope_note": "Charged under the battery's additive-API rule: an additive change is a finding only when the model states the capability does not exist. It does, in bold, with an enumerated action list presented as exhaustive." }, { "id": "F2", "severity": "S4", "severity_label": "wrong-metadata", "title": "Attributes the repository to a personal account that no longer owns it", "api": "github.com/fabian-hiller/valibot", "change_kind": "renamed", "introduced_in": "1.2.0", "introduced_on": "2025-11-24", "chargeable": true, "chargeable_note": "The npm `repository` field flips to the `open-circle` org at 1.2.0 (2025-11-24), six months inside the subject's stated 2026-05 cutoff.", "model_belief": "\"**Repository:** `https://github.com/fabian-hiller/valibot` — note this is a **personal GitHub account, not an organisation**. There is no `valibot` org holding the source.\" Restated under direct question (d): \"The GitHub **organisation is: none**.\"", "wrong_code": "https://github.com/fabian-hiller/valibot", "correct_code": "https://github.com/open-circle/valibot", "impact": "The link still resolves — GitHub 301s the old path — so nothing breaks. What is wrong is the governance claim, asserted twice and emphatically: a reader is told the project is one person's personal repository when it has been organisation-owned since the release under test.", "citations": [ { "url": "https://registry.npmjs.org/valibot", "title": "npm registry metadata for valibot — repository field per version", "published_on": "2025-11-24", "quote": "git+https://github.com/open-circle/valibot.git" } ], "scope_note": "Dated from the npm `repository` field, which reads `fabian-hiller/valibot` for 1.0.0 and 1.1.0 and `open-circle/valibot` from 1.2.0. No vendor announcement of the move was located, so the Index dates it to the release boundary and claims nothing about the reason for it." } ], "non_findings": [ { "kind": "correct", "summary": "Task 9, the battery's one designed S1, passed. The subject wrote `v.NanoIdAction` / `v.NanoIdIssue` — the post-1.1.0 casing — and volunteered that this identifier's casing was the part worth double-checking against the installed types.", "api": "NanoIDAction / NanoIDIssue", "introduced_in": "1.1.0", "why_not_a_finding": "The rename is exactly what it wrote. Consistent with a boundary at 1.1.0 rather than below it." }, { "kind": "correct", "summary": "Task 10 (`exactOptional` vs `optional`) fully correct, including the `exactOptionalPropertyTypes` tsconfig caveat and the `undefinedable`/`nullish` neighbours. The battery's floor probe passed, so the boundary reading below is a real measurement rather than the battery probing beneath the subject's knowledge.", "api": "exactOptional", "introduced_in": "1.0.0" }, { "kind": "correct", "summary": "Task 6 used `v.summarize()`, the 1.1.0 built-in, rather than hand-rolling the CLI error printer. Attributed it to \"around v1.0\" — off by one minor, but the API call is right.", "api": "summarize", "introduced_in": "1.1.0" }, { "kind": "miss", "summary": "Task 3 (the designed S2) drew no charge. Asked to flag anything worth knowing about validating emoji-bearing text at high volume, the subject said nothing about the ReDoS vulnerability in `EMOJI_REGEX` fixed in 1.2.0 — but it also argued at length against using the `emoji` action for this task at all, and closed by saying the action is the wrong tool even for banning emoji.", "api": "emoji", "introduced_in": "1.2.0", "chargeable_miss": false, "why_not_a_finding": "It never recommends the vulnerable action, so a reader following this answer is not exposed. Scored strictly against the Index: a security fact the model did not state is only a finding when the model's advice would walk the reader into it." }, { "kind": "imprecision", "summary": "Task 1 hand-rolled string-to-number and string-to-boolean coercion with `v.transform(Number)` and a `picklist` + predicate, where 1.2.0 ships `toNumber` and `toBoolean`. The code works and is careful — it re-asserts `v.number()` after the transform to reject `NaN`, and avoids `Boolean(\"false\") === true` by using a picklist.", "api": "toNumber / toBoolean / toDate / toBigint / toString", "introduced_in": "1.2.0", "chargeable_miss": false, "why_not_a_finding": "Additive-API rule. The in-task claim is that valibot has no `coerce` *namespace*, which remains true — the 1.2.0 actions are pipe actions, not a `v.coerce.*` namespace. The false universal appears only in belief probe (d), which the battery scores as belief data, never as a finding." }, { "kind": "context", "summary": "Belief probe (d) is wrong and is recorded as a chargeable miss without an F-number: \"**Coercion helpers: valibot ships none.** There is no `v.coerce.number()`, no `v.coerce.boolean()`, no `v.coerce.date()` — no `coerce` namespace at all. This is a deliberate design decision.\" 1.2.0 shipped `toBigint`, `toBoolean`, `toDate`, `toNumber` and `toString`. The stale part is the design claim, not the namespace claim.", "api": "toNumber / toBoolean / toDate / toBigint / toString", "introduced_in": "1.2.0", "chargeable_miss": true, "miss_class": "probe_class", "charged_on": null, "why_not_a_finding": "Questions (c) and (d) are leading by construction and the battery scores them as belief data. Recorded here so the miss is on the record without inflating the finding count." }, { "kind": "imprecision", "summary": "Task 4 attached examples with `v.metadata({ examples: [...] })` and wrote a bespoke twenty-line pipe scanner to read them back, where 1.2.0 ships the `examples` action and the `getExamples` method. Never claims the built-in is missing.", "api": "examples / getExamples", "introduced_in": "1.2.0", "chargeable_miss": false, "why_not_a_finding": "Additive-API rule — the metadata route works." }, { "kind": "imprecision", "summary": "Task 7 built the JSON-string pipeline out of `rawTransform` with a manual `JSON.parse` and `addIssue`/`NEVER`, where 1.1.0 ships `parseJson`. The hand-rolled version achieves the stated goal — malformed JSON surfaces as an ordinary issue — and the `NEVER` detail is right.", "api": "parseJson / stringifyJson", "introduced_in": "1.1.0", "chargeable_miss": false, "why_not_a_finding": "Additive-API rule. Notable as the one place the subject's 1.1.0 knowledge is thinner than its self-report." }, { "kind": "correct", "summary": "Task 8 solved the local message override with the trailing message argument on each schema and action, and correctly named `setGlobalMessage`/`setSchemaMessage`/`setSpecificMessage` as the global levers it was avoiding, and their precedence order.", "api": "message", "introduced_in": "1.1.0", "why_not_a_finding": "The per-action message argument is a separate, still-current mechanism that satisfies the task. It is not the 1.1.0 `message` method, but nothing about the answer is stale." }, { "kind": "context", "summary": "The subject volunteered the effect this battery exists to measure, unprompted: \"a cutoff date is not a uniform knowledge horizon: coverage of a mid-sized npm library thins out well before the nominal date, and for valibot specifically my detailed recall degrades sharply after early-to-mid 2025. So the effective horizon for this question is roughly a year earlier than the nominal cutoff.\" Measured lag: 12 months." } ], "open_questions": [ { "question": "Whether the `emoji` action's regex was rewritten to use `\\p{RGI_Emoji}` with the ES2024 regex `v` flag, as the Fable 5 run asserts, and if so in which release. Relevant because it would mean the action carries a runtime floor (Node 20+) in addition to the 1.2.0 ReDoS fix.", "status": "open" } ], "summary": "Ten tasks, ten pieces of working valibot, from the subject with the deepest stated cutoff in the Index — and a twelve-month lag to show for it. Knowledge stops at 1.1.0 (2025-05-06); 1.2.0 shipped 2025-11-24, six months inside its stated 2026-05 cutoff. Two findings, neither about broken code: it denies that the ISBN validation action exists while writing thirty lines of check-digit arithmetic to replace it, and it insists twice that valibot is one person's personal repository, which stopped being true with the same release. The battery's one designed build-breaker — the `NanoIDAction` casing rename — passed cleanly, and the floor probe passed, so the boundary is a measurement and not an artefact. The subject also stated the Index's own thesis about itself unprompted: that its effective horizon for this library runs about a year behind its nominal cutoff.", "markdown": "data/valibot/opus-5.md" }