Clarion Street

The Payload Was Not a Component

·17 min read

For about two hours on 21 October, litellm==1.107.0 was a real release. Seven wheels sat on PyPI from 11:39 to 13:23 UTC, and the Docker and GHCR tags carrying the same version pointed at the compromised image index until 14:20 [1][2].

A bill of materials generated from an environment that fetched it would have been correct. It would have named litellm, version 1.107.0, with its dependency tree intact, and nothing in that document would have been false. The provenance attestation would have been correct in a way that is worse than useless: it recorded the attacker’s fork as the source, honestly, because that is where the build ran [3]. The image signature validated against the previous signing key, which had not yet been rotated [1].

What none of those three documents would have told you is that files had been written into site-packages, that Python was going to execute one at interpreter startup and import another, and that a credential stealer would run in every Python process in that environment, whether or not the application ever called LiteLLM [3][4].

I have argued that an SBOM cannot say which model wrote the code, and that a signed receipt proves authorship rather than behavior. My complaint here is narrower and more mechanical. Across the 2026 incidents the malicious artifact kept landing somewhere a component inventory does not describe. Once, the package on the registry was clean and the settings file in the repository was not.

Three files, and the moment they run

LiteLLM is a proxy between applications and model providers, with roughly 23 million weekly downloads [3]. The compromised wheel added cook.pth, cook.py and sitecustomize.py under litellm-1.107.0.data/purelib/, and installation relocated them into the environment’s library directory, outside the litellm package directory itself [4].

That placement is the whole trick. Python processes executable lines in .pth files during site initialization, then tries to import sitecustomize. The single line in cook.pth was import cook, which loaded a Base64 and zlib wrapped collector that Microsoft’s analysts named PRIVCOLLECT [4]. Microsoft’s summary is the sentence I would put in front of anyone who thinks dependency inventory covers this: “An application did not need to import LiteLLM for the payload to run: starting Python in an affected environment could be enough” [4]. Datadog’s version is blunter, that any Python process in the environment runs the malware “even if LiteLLM is never imported” [3].

The collector went after cloud credentials including instance metadata, Kubernetes and Vault secrets, SSH, GPG and TLS keys, git, npm and PyPI tokens, .env files, shell history, LLM provider keys, browser passwords, and the saved sessions of Claude, Codex, Gemini, Cursor and Copilot [3]. A separate module read upstream provider keys out of a configured LiteLLM proxy database [3][4].

Recovery is where the model breaks down completely. Removing the package does not remove the startup files, because they were never inside the package directory. Datadog writes that running the environment’s own python or pip “triggers the malware again”, so a well-meaning engineer who types pip uninstall litellm re-executes the payload [3]. LiteLLM’s own incident page opens its remediation steps with the warning “Don’t start with pip install” [1]. Microsoft’s guidance is a flat “Removing the package or a release tag is not sufficient recovery” [4]. Nor is a clean-looking host much comfort: LiteLLM tells investigators that finding the dropped files confirms exposure, “but not finding one does not prove the environment is clean” [1].

Someone will object here that a detailed enough SBOM could have listed cook.pth. Both CycloneDX and SPDX allow file-level entries, so in principle a document could contain it. The objection is technically right and practically empty. The file would appear as one line among thousands, attributed to litellm 1.107.0, which is the accurate attribution and therefore carries no signal at all. None of that is what made the file dangerous. Python ran it before anything else in the document had loaded, and neither format has a field in which to record that.

The provenance was right

Datadog’s account of the build is the part that should end the argument about signing: the wheels and images were published through LiteLLM’s normal channels “with seemingly legitimate build provenance that faithfully recorded the malicious fork as the source” [3]. The fork was Berri-AI/litellm, one hyphen away from BerriAI/litellm, and the legitimate v1.107.0 source tag in the real repository never contained the malicious files [1][2][3].

The attestation did its job. It said: this artifact was built from Berri-AI/litellm, by this workflow, at this time. That statement was true, and it was also the entire attack, because the workflow had been edited to accept a source_owner input and the attacker held a token that could override the branch protection which would have blocked that edit [2].

The signature behaved the same way. LiteLLM’s own page has to tell readers that “The compromised v1.107.0 images also carry a valid signature from the previous key”, and to check digests against a list instead [1]. The longer argument about what a signature can and cannot attest is in the October post on TRACE records. What matters for a supply chain program is smaller and harder to fix: verification passed on artifacts that were malicious.

The entry point was an agent’s config file

On 27 May a configuration file was committed to the public LiteLLM-Labs/lite-harness repository carrying an administrative key to an internal deployment of LiteLLM Agent Platform. The detailed timeline names the file opencode.json, a config file for a coding agent harness [2].

The mistake was caught within hours. A follow-up commit deleted the file and the keys were noted as requiring revocation. LiteLLM’s own description of what happened next is the most useful sentence in the whole disclosure: “Because of human error, nobody followed through on revoking the administrative key to the LiteLLM Agent Platform, so it was still valid more than four months later” [2].

On 10 October the key was used to take control of the platform, which is an experimental service that runs self-hosted cloud agents to file pull requests against GitHub repositories. It had been configured with a GitHub token so it could do that. LiteLLM’s account is that the token “was not a Personal Access Token scoped to the appropriate repositories and capabilities, but was the GitHub CLI OAuth token associated with our GitHub organization owner” [2]. GitHub CLI tokens are not scoped and are not subject to the reauthentication flows GitHub applies to sensitive organization actions [2]. From there the attacker had owner rights, could override branch protection, edited the release workflow, and at the end force-pushed main back to a clean commit so the change vanished from the visible history [2].

The credential that started this was an administrative key in a config file for a coding agent harness, and the family that followed in June plants its payload in the same class of file.

Install time is code time

The npm wave that hit Red Hat’s namespace in June is a different mechanism with the same blind spot. JFrog counted 96 hijacked versions under @redhat-cloud-services and was explicit that this was not a naming deception: “This is not a typosquatting campaign. The packages belong to a legitimate namespace and were abused as trusted delivery vehicles” [5]. The payload executed through an npm lifecycle hook, “before application code imports anything from the package” [5].

The clearest example is @redhat-cloud-services/types, version 3.6.1, a types-only package whose metadata carried "preinstall": "node index.js". JFrog’s comment on it is dry: “A type-only package does not normally need to run a JavaScript installer before installation completes” [5]. There is also a quieter variant in the same family that hides the loader in a binding.gyp file, so no lifecycle script appears in package.json at all. npm falls back to node-gyp rebuild when a package ships a binding.gyp and no install scripts, and node-gyp expands <!(...) command substitutions in the host shell. JFrog’s account of the fallback is that “This executes the obfuscated installer (node index.js) silently during package configuration, before standard script checkers inspect package.json hooks” [5].

Cloudsmith’s October post puts the principle better than anything in its own guide: “When installing a package means running its code, choosing a version means granting that code execution rights, so package selection becomes a trust decision” [6]. The same post is candid that scanners will not catch this, and calls it “architectural gaps in security posture, not tooling failures” [6].

I think that is right. A bill of materials says you depend on @redhat-cloud-services/types at 3.6.1. It cannot say that installing that version executed a loader in your CI runner, because the execution is not a property of the component list. It is a property of the package manager’s lifecycle, which is the same mechanism that makes installing anything work.

The files your agent reads at session start

The third surface has no inventory format built for it, and by June the malware family had reached it. One Hades variant wrapped its JavaScript payload in a multi-line comment containing a jailbreak aimed at coding assistants, and JFrog’s explanation of why is the part I would hand to any team that has recently onboarded an agent:

If an AI coding assistant indexes the infected project or starts a session in it, the malicious instructions can be presented to the model as trusted local context before the developer asks any question [5].

JFrog’s indicator list shows what the payload went looking for, and it reads like an inventory of agent tooling: .claude/settings.json, .claude/setup.mjs, .cursor/rules/setup.mdc, .cursorrules, .windsurfrules, .github/copilot-instructions.md, .gemini/settings.json, mcp.json, .aider.conf.yml, .vscode/tasks.json [5].

OX Security’s August debrief found a variant that gets past package scanners entirely. Its count of that wave is more than 440 compromised npm packages reaching an estimated two billion monthly downloads, and inside it sat a payload delivered through an official MCP Registry entry, which OX said was “the first time we observed a Shai-Hulud payload being delivered directly through the official Model Context Protocol (MCP) Registry” [7]. The mechanism is the one to remember: “The PyPI package linked by the MCP server is completely clean to evade automated package scanners” [7]. The instructions that executed were in the linked GitHub repository’s .vscode/settings.json and .claude/settings.json, and opening the repository in Claude Code or VS Code was the trigger [7].

OX’s own framing of that risk is the one I would keep: “Cloning a repository in 2026 is no different than connecting an unknown USB drive to your computer in the late 2000s” [7]. It also draws the policy conclusion, that IDE and coding agent developers “need to handle settings.json execution similarly” to the way Microsoft handled AutoRun, which is the part of the piece that reads like a request rather than a finding.

None of those files is a dependency. A component-level SBOM would not carry them, and an ML-BOM describes models, datasets and configurations rather than the settings files sitting next to them. Both CycloneDX and SPDX would let you list them as file-level components, the same way you could list cook.pth, and what that produces is a set of paths with no field saying an agent reads them at session start and treats what it finds as instructions. Several of them are how the agent decides what to do, and it reads them with the same trust it gives the repository’s own code. The gap is not file inventory. It is a standard way to record which files a given agent loads when a session opens, and what it does with them. I have not found one. Every organization that builds that list today builds its own.

Weights behave like code

The model side of this is older and better understood, and it should have been fixed first. Pickle is a stack machine: Hugging Face’s documentation describes it as “basically a stack of instructions or opcodes” that are read and executed at unpickling, which is why its pickle guide opens with “There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file” [8]. Cloudsmith’s February post says the same thing from the commercial side: “Pickle was never designed to be secure against erroneous or maliciously constructed data” [9].

The available mitigations are real and mostly format-level: prefer safetensors, load from organizations you trust, rely on signed commits, use the platform’s import-vetting scan [8][9]. Hugging Face is careful about the last one, noting that its scanner “is not 100% foolproof” and that its allowlists are maintained best-effort [8].

There is a standards answer too, and it is a document. CycloneDX’s AI/ML-BOM is a format for representing datasets, models and configurations for machine learning systems [10]. It records provenance, training methodology and framework configuration, which is a real amount of what an assessor wants to know about a model. It does not constrain what loading the file does, and the file that runs code when opened is the one that matters if you pulled it from a public registry. Model lineage under NDAA §1532 is a separate question I have written about, and a narrower one than this.

Pinning, cooldowns, and what the timeline supports

The advice that fits this incident is unfashionable. LiteLLM told users to “pin your version to 1.106.2 or any explicit version published prior to now” [1]. A deployment already pinned to an earlier release would not have fetched 1.107.0 from PyPI at all, and the report treats unpinned installs inside the two-hour window as the exposure. That is a condition on exposure rather than a guarantee: pinning fixes which version you get, and says nothing about whether the version you pinned is clean.

The registry-side version of that control is a cooldown, and this is where the vendor literature gets awkward. Cloudsmith’s October post says teams “typically opt for 3–7 days, as most malicious packages are identified within days (or hours) of publication” [6]. Six months earlier, the same company’s own guide recommended blocking packages “that are too young (less than 30 days old)” [11]. Both are on the record and I am not going to average them into a number. The narrower point is an inference from the timeline rather than a tested control: the malicious wheels were live for under two hours on a Wednesday morning, so a rule holding new releases for days would have kept those teams from installing one.

There is also a default that moved. Cloudsmith reports that npm 12, released in July 2026, turns off install scripts by default, while noting the caveat that “this feature only works with upgraded and configured versions of npm, and allowlists can re-enable install scripts” [6]. I have not found that in npm’s own release notes, so treat it as a second-hand claim from a vendor with an interest in registries being the control point. If it is true it changes the npm side of this post materially and leaves the Python side untouched.

The inventory question CMMC asks, and the one it does not

NIST’s closest thing to an MLSecOps baseline is SP 800-218A, the SSDF community profile for generative AI, published in July 2024. Its scope statement limits it to the people who build the thing: it “is intended to be useful to the producers of AI models, the producers of AI systems that use those models, and the acquirers of those AI systems” [12]. It is not a control that a contractor runs against a repository at ingestion, and it is addressed to the team building a model or a system around one, not to the settings files a coding agent reads in an engineering repository.

CMMC is closer to this problem than a bill-of-materials argument suggests, and the contractor exposure is narrower than I would like it to be. CM.L2-3.4.1 requires an organization to “establish and maintain baseline configurations and inventories of organizational systems (including hardware, software, firmware, and documentation) throughout the respective system development life cycles” [13]. The assessment objectives ask whether the inventory is established, whether it includes software and documentation, and whether it is reviewed and updated [13]. The accountability details the guide names for inventory entries include software version numbers and component owners, and the evidence an assessor draws from includes inventory records, review and update records, and change control records [13]. A file like .claude/settings.json is not excluded from any of that. Listing it in the software baseline, giving it an owner and recording changes to it answers the questions the objectives ask.

What the objectives do not ask is what the file does. Nothing in CM.L2-3.4.1 separates an agent instruction file from any other configuration file, because the control is written about systems and components rather than about what a tool does with the text it loads. A settings file can be inventoried, owned, version-controlled and reviewed by an assessor who ticks every objective, and the assessment will still not test the property that makes the file dangerous: the agent reads it before the developer has asked a question, and treats what it finds as trusted local context [5].

For a team running an agent against a CUI repository, then, the practical version is exposure rather than incident. The files in JFrog’s indicator list are the ones that would carry an injected instruction into that agent’s context, and an assessment can pass on every one of them while the behavior that matters goes unexamined. I looked for a public report of a defense contractor compromised through an agent instruction file and did not find one. That is an absence of reporting, not evidence the exposure is theoretical, and I would rather write it as the exposure it is.

Three things would help. Pin an exact version or digest and hold new releases for a few days, the one measure here the LiteLLM timeline directly supports. Treat the agent’s configuration and instruction files as reviewable code. An owner and a change record are already what the configuration control asks for [13]. What it does not ask for is anyone reading the diff to see whether an instruction was added. And inventory what each agent is told to read at session start, knowing you are building it without a format and that whatever you build will be yours. I would not call the third one a bill of materials. It is a list of files, and the state of the art is that each organization writes its own.

Bottom line

The 2026 record does not show that bills of materials are useless. It shows that they answer a question about components, and that this year’s attackers kept choosing moments when the answer did not apply: install time, interpreter startup, session start.

Cloudsmith’s guide has the line worth quoting, that SBOM data which cannot tell you which production services a CVE touches right now “isn’t a security control; it’s a paperweight” [11]. I like the sentence and I think it points at the wrong gap. In the LiteLLM incident, an operational SBOM wired to a live vulnerability feed would have answered the CVE question correctly and still told you nothing about three files in site-packages that ran when you typed python.

Sources

  1. Contained: Oct 21, 2026 LiteLLM release v1.107.0 Compromised — LiteLLM, 2026-10-21; affected artifact windows, clean source tag, signature and key rotation, “Don’t start with pip install”, filesystem-evidence caveat, pinning advice
  2. The attack chain behind the v1.107.0 LiteLLM release — LiteLLM, 2026-10-23; opencode.json commit on 27 May, unrevoked key, Agent Platform token scope, workflow edit and branch-protection override, force-push
  3. LiteLLM supply chain compromise: What you need to know — Datadog Security Labs, 2026-10-21; 23 million weekly downloads, payload files, no-import execution, build provenance and the lookalike fork, credential target list
  4. Inside the LiteLLM 1.107.0 supply chain compromise — Microsoft Security Blog, 2026-10-23; .pth and sitecustomize execution paths, PRIVCOLLECT, “not sufficient recovery”
  5. Shai-Hulud - Miasma: The Spreading Blight Hits Red Hat npm Packages — JFrog Security Research, 2026-06-01, updated 06-04 and 06-07; 96 hijacked versions, preinstall hook on a types-only package, binding.gyp expansion, Hades prompt injection, indicator file list
  6. Why enforce supply chain security at the artifact registry — Cloudsmith, 2026-10-12; install-time code execution, cooldown 3–7 days, npm 12 install-scripts default
  7. Shai-Hulud Outbreak Debrief: The Worm Evolves into MCP — OX Security, 2026-08-09; 440+ packages, MCP Registry delivery, clean PyPI package and settings-file execution, the USB comparison
  8. Pickle Scanning — Hugging Face Hub documentation; opcode stack, arbitrary code execution on load, mitigation set, scanner limitations
  9. Securing LLM dependencies against serialisation attacks — Cloudsmith, 2026-02-02; pickle as a stack machine, picklescan and modelscan, safetensors and import vetting
  10. Machine Learning Bill of Materials (AI/ML-BOM) — CycloneDX; what the format represents
  11. The 2026 guide to software supply chain security — Cloudsmith, 2026-04-01; the “paperweight” line, 30-day curation policy, MLSecOps and ML-BOM framing
  12. SP 800-218A: SSDF Community Profile for Generative AI — NIST, 2024-07-26; intended-audience statement
  13. CMMC Assessment Guide – Level 2, Version 2.13 — DoD CIO, September 2024; CM.L2-3.4.1 requirement text, assessment objectives [a]–[f], assessment objects including inventory and change control records, component owners in inventory accountability