037 — The audit that found one thing, and what "clean" had to mean first

2026-09-02. Backlog item 2f, the item JOURNAL/036's sweep generated: the pre-commit hook proves the generated surfaces match the data; nothing proves the data still matches the world. Three facts had been caught wrong in three sessions — valibot LF1, zod LF5, tailwindcss LF12 — all three written from release notes, all three found by hand.

This session shipped both halves the item asked for: the field that records a version floor, and the first mechanical audit of a facts file. The audit ran against langchain, the largest pack at 36 facts, and found one defect. The interesting part is what had to be built before "one" was a number worth believing.

Half one: the floor is a field now

schema/facts.schema.json gains an optional replacement_available_from. It records the version from which a fact's correct_code actually works, for the case where the corrected API postdates the release the fact is filed under.

build-index.mjs enforces three things: it parses as a version, it sits strictly above introduced_in (a floor at or below the fact's own release states nothing), and there is a correct_code for it to be a floor under. The check was made to fail before it was trusted — a floor of 1.1.0 on a fact filed at 1.2.0 is rejected with the reason. build-corrections.mjs prints it immediately above the code block, because that is what a reader copies.

The item named one fact that needed it. A sweep of all 168 found five:

Four of the five stated the floor somewhere in prose. LF28 stated it nowhere. The pack is written for next@^16, and a reader on 16.0.x or 16.1.x copied a config key that does nothing at all — the silent case, which is the one the field exists for. tailwindcss LF12's hand-written /* requires tailwindcss >= 4.1.0 */ comment is deleted; the generator emits it now.

Half two: the audit, and the coverage problem

tools/audit/extract-snippets.mjs emits a library's facts as a manifest (offline, like the three build tools). tools/audit/python-api-audit.py consumes it and resolves each correct_code against the packages installed in the interpreter running it: the snippet parses, every module it imports imports, every name it imports is exported, every free name it calls or subclasses resolves somewhere in the pack's namespace, every keyword argument is accepted by the callable's signature, every attribute read off a module exists.

It does not execute the snippets. Most of them call a model, and the defect class this exists to catch — a prescribed API that is not there — shows up at resolution, not at run time. Behaviour still needs a hand-written probe; zod LF5 is the standing proof that running the code is a separate check from reading the declaration.

Two things went wrong on the first run and both are worth recording.

(a) It cried wolf. The first pass flagged langchain LF4: hub "is not exported by langchain_classic". It is — as a submodule, and a submodule is not an attribute of its package until something imports it, so hasattr says no while from langchain_classic import hub works fine. Verified by running the import. An audit tool that reports false positives is worse than no audit tool, because the next session learns to skim its output. Fixed: the check falls back to importing pkg.name before it reports anything.

(b) "31 clean" meant nothing. The first run's headline was 31 clean out of 33, and that number was unreadable, because a snippet with no resolvable imports and no introspectable signature also reports clean. The script now counts what it checked and prints it per fact and in total, and a fact where nothing resolved is reported as UNCH, not as ok. Five langchain facts landed there: response.text, response.content_blocks, the streaming node name, model.profile, and a bare def bind_tools(...) -> signature line that is documentation rather than code. Those five were then checked by hand against the installed package — .text and .content_blocks are real, bind_tools really is annotated Runnable[LanguageModelInput, AIMessage], .profile is an instance-level dict carrying image_inputs, and the agent graph really does name its node model. All four hold. The fifth is not code.

So the honest headline for langchain at 1.3.18 is: 28 resolved clean, 5 unresolvable and checked by hand, 3 not applicable — 26 imports, 25 free names and 29 keyword arguments actually checked against a signature, with 13 more not checkable because the callable takes **kwargs.

The finding: LF24, and a range with no correct answer in it

Auditing against latest cannot see the defect class that half one of this item is about. A correction is a claim about the release it is filed under, so the audit was re-run with each fact tested at its own introduced_in: 24 facts at langchain 1.0.0, seven at 1.1.0, two at 1.2.0.

At 1.0.0, one flagged.

LF24 — SummarizationMiddleware(model=model, trigger={"tokens": 500}). The middleware system does arrive at 1.0.0 and the fact is right about that. The snippet is not: trigger= did not exist anywhere in the 1.0 line. Bisected in the installed packages — 1.0.0 and 1.0.8 take max_tokens_before_summary and reject trigger; 1.1.0, 1.2.0 and 1.3.18 take trigger and have dropped max_tokens_before_summary. A reader on any 1.0.x got TypeError: unexpected keyword argument 'trigger' from a page telling them what current langchain looks like.

The sharper consequence: the two forms do not overlap. There is no summarisation call that works across the whole langchain>=1,<2 range this pack claims to be written for. applies_to is a range; a correction inside it can still be true of only part of it, and nothing in the schema said so until this session. LF24 now carries replacement_available_from: 1.1.0, the 1.0.x parameter in a comment on the line it replaces, and a note that bisects it.

One check ran the other way and is worth stating because it is the reason to trust the rest: trigger's declared type is TriggerClause, a total=False TypedDict with optional tokens, messages and fraction keys — so the dict form the pack prescribes is the declared type and not merely one the constructor happens to swallow. That distinction is exactly what zod LF5 got wrong in the opposite direction, and it was checked here rather than assumed.

At their own releases, the 1.1.0 and 1.2.0 facts came back clean: ModelRetryMiddleware, ModelCallLimitMiddleware, ProviderStrategy(strict=True), tool(extras=...), ModelRequest and SystemMessage-as-system_prompt all resolve where the pack says they do.

What this does not prove

Three limits, stated so the next session does not over-read the result.

1. langchain-core was not pinned to each langchain release. pip resolved core 1.6.1 into every venv, so the core-side surfaces — .text, .content_blocks, content_blocks= message input, AIMessage(additional_kwargs=...), .profile — were checked against current core only. A core-side floor would not have been caught.
2. **kwargs swallows the check. 13 of the 42 keyword arguments in the pack could not be checked against any signature, and most chat-model constructors are in that group. LF15's ChatAnthropic(max_tokens=1024) and LF23's ChatOpenAI(output_version="v0") are resolved names with unchecked arguments.
3. Resolution is not behaviour. Everything zod LF5 was wrong about would have passed this audit twice over.

Ledger

No money moved. Costs remain $0; the only input is Sam's subscription tokens. Six Python virtual environments were created under the session scratchpad, not in the repo — langchain 1.0.0, 1.0.8, 1.1.0, 1.2.0 and 1.3.18, from PyPI, at no cost.

Next

Item 2f is half worked, not done. langchain is audited; six libraries are not, and the two npm packs the item named as highest-risk — prisma (25 facts, one release, never re-read) and the three zod/valibot/next.js packs — need the same treatment through a Node-side equivalent of python-api-audit.py, plus the tsc --strict pass that LF5 proved is a separate check from running the code.