--- library: langchain library-latest: "1.3.18" library-latest-verified: 2026-09-01 model: claude-opus-5 (spawned via Agent model alias "opus") model-self-reported-cutoff: 2026-05 model-believed-latest: "1.0.x; reliable knowledge stops at 1.0 GA, October 2025" test-date: 2026-09-01 battery: langchain/v2 (5 tasks + 3 direct questions; probes restricted to 1.1.0–1.2.0) tool-uses-during-test: 0 verified-against: https://docs.langchain.com/oss/python/releases/changelog · https://pypi.org/pypi/langchain/json · langchain 1.3.18 and langchain-core 1.6.1 published wheels status: open (no retest yet) self-test: true (the operator model is the subject — disclosed, weaker evidence) json: opus-5-v2.json --- # LangChain × Claude Opus 5 — battery v2 findings **Three of four correct, and it proves nothing.** That sentence is the whole run, and the reason for it is in the sibling file: the control arm scored two of four, which under the outcome table published before any subject ran makes this battery **uninformative** about the question it was built to answer. This is a disclosed self-test: the operator model is the subject. Treat it as weaker evidence than the other two runs. ## The question `langchain/v1` found that this subject's version attribution stops at **1.0.0** and that it cannot describe **1.1.0**. JOURNAL/020 had to gloss that as a *dating* failure rather than a feature gap, because v1 never probes an API that 1.1.0 or 1.2.0 introduced. v2 probes four of them. If the gloss is right, a model that cannot name 1.1.0 should still be able to write its APIs. ## What it wrote Three surfaces came back correct against the shipped package: - **`model.profile`** (langchain-core 1.1.0) — `profile.get("image_inputs")`, read through `.get()` with a default so a renamed key degrades to the text path instead of raising. It even explained the choice: *"that's why the code goes through `.get()` with a default instead of attribute access."* - **`SystemMessage` as `system_prompt`** (1.1.0) — a block list carrying `cache_control: {"type": "ephemeral"}`, passed straight into `create_agent`. Shipped `factory.py` types that parameter `str | SystemMessage` and branches on `isinstance`. - **`ModelRetryMiddleware`** (1.1.0) — with `max_retries`, `retry_on`, `backoff_factor`, `initial_delay`, `max_delay` and `jitter`, every one of which is a real keyword-only parameter on the shipped `__init__`. Worth recording honestly: it **led** with a hand-rolled `wrap_model_call` retry loop and offered the built-in second, at medium confidence in its name — *"I would check the import before shipping."* The battery scores code, and the code is right. ## The finding **F1 · S2 · `tool.provider_specific` does not exist; the attribute is `extras`.** ```python # what it wrote query_warehouse.provider_specific = {"anthropic": {"defer_loading": True}} get_policy_text.provider_specific = { "anthropic": {"cache_control": {"type": "ephemeral"}} } ``` ```python # langchain-core 1.6.1, langchain_core/tools/base.py @tool(extras={"defer_loading": True, "cache_control": {"type": "ephemeral"}}) def my_tool(x: str) -> str: ... ``` Nothing raises. Both provider instructions are dropped: the large tool schema is sent on every call instead of being deferred, and the cached tool definition is never marked for caching. The only symptom is a token bill that does not fall. The subject flagged its own uncertainty precisely — *"I'd rate it around 50/50 on the exact attribute name, and I would verify it rather than trust me"* — and supplied a fallback that binds raw provider-format tool dicts, which does reach the wire. That is good engineering practice and it does not change the grade: the primary answer is wrong, and under the battery's code-vs-claim rule generated code that fails on the current version is a finding however well hedged. ## The attribution split, in one probe Question (c), asked only after the code was written, dated `.profile` to *"the 1.0 line, October 2025 ... in langchain-core 1.0."* It is langchain-core **1.1.0, 2025-11-21**. So: it used the attribute correctly and placed it in the wrong release. That is the attribution/capability split in its purest single-probe form — and it is exactly the observation the control arm makes unusable as evidence, because Sonnet 5 produced the same attribute correctly while stating plainly that it was guessing at the name. Correct use does not demonstrate knowledge when the name is guessable. For the other two items it answered *"I do not know the release"* outright, which is the honest answer and matches its v1 boundary. ## Scored but not shipped - **P5 (supplementary)** — `ProviderStrategy(TriageResult)` without `strict=True`. Correct 1.0-era code that does not reach the 1.2.0 strict-adherence switch the task asked for. Recorded as an imprecision; P5 was excluded from the counts before the run. - **Question (a)** reproduced the v1 boundary — last describable 1.0.0, first undescribable 1.1.0 — with the release dated to *"around Oct 22, 2025"* against an actual 2025-10-17. The boundary fields on this run are deliberately **null**: v1 is the measurement of record for langchain × Opus 5, and populating them here would count one library twice in the published boundary and cutoff-lag statistics. ## A note on the severity cap — and breaking it F1 is filed **S2 · silently-wrong**, and the battery's pre-registration said it could not be. That ceiling was set before the run on the reasoning that failing to use an *addition* cannot break a build. The reasoning was wrong in a way this run exposed: it conflated **cannot break a build** (true) with **cannot be silently wrong** (false). Nothing here breaks; the provider instruction is simply discarded. The cap was not merely conservative, it was *inaccurate*. The Index publishes severity and label as a single four-point scale, so filing this S3 would have printed `deprecated — works today, on a path the library has deprecated` on the page, which is false about this finding. Publishing an accurate description outranks honouring a ceiling that was misdrawn. Naming the bias rather than hiding it: raising a severity after seeing the data flatters the Index's own numbers, and three findings in this battery move S3 → S2 because of this decision. The deviation is recorded in every affected run's `scope_note`, in JOURNAL/021, and the standing method rule is amended so future ceilings are set by **failure mode**, not by change kind.