Forty Minutes, Five Months, 2,500 Companies: The LiteLLM Supply Chain Bill

I wrote about the litellm compromise on March 25, the day after it happened. The facts were clear and the scope was not.

Somebody had slipped a booby-trapped version of a popular software component into the public library that Python projects download from. It was there for about forty minutes. Every machine that downloaded it during that window handed over its passwords and access keys.

The question nobody could answer was who got hit. On August 11, five months later, CloudSEK published an answer.

The Bill

• More than 2,500 organisations in the reconstructed dataset.
• Around 434,000 build pipelines touched. A build pipeline is the automated server that assembles and tests a company’s software, and it holds the keys to almost everything.
• Named in reporting: NVIDIA, Samsung, Cisco, Siemens, S&P Global, ServiceNow, Deloitte, Vodafone, X Corp, Zscaler, FedEx, Volkswagen, Thales, and the London Stock Exchange Group.
• Per-organisation counts where reported: 3,459 secrets attributed to X Corp, 462 to Deloitte, 327 to Cisco.

High confidence refers to the strength of the exposure match, not proof of successful compromise.

— CloudSEK, on their exposure dataset

Sources also disagree on what 434,000 counts: some say build pipelines, others files taken from them. Those are very different things and I could not reconcile them.

How the Number Was Produced

This matters more than the total.

CloudSEK did not get it from download logs, from victims reporting in, or from anyone’s monitoring. They pieced it together from the attackers’ own haul, part of which was encrypted and shipped to a web address impersonating the real vendor, and part of which the attackers left sitting in public repositories online.

So the definitive census of a global supply-chain compromise exists because the thieves left the swag on the street.

For five months, every organisation in that dataset had no way to work out whether it had been hit. Not through negligence. A build server downloads the component. The hidden code runs automatically the moment anything starts up, without the component ever being used. The stolen keys go out over an ordinary encrypted connection to an address that looks like the vendor’s own. Then the build finishes successfully. Nothing fails, nothing is logged, and there is nothing to find later.

The Chain

The way in was not litellm. It was a security scanner.

• 27 February: attackers steal a powerful access key from Trivy, a widely used free security scanner made by Aqua Security, by abusing a setting that let outside contributions run with insider privileges.
• Aqua’s key rotation is incomplete. The attacker keeps access.
• 19 March: poisoned versions of Trivy go out. Aqua later confirmed the attackers rewrote 76 of the scanner’s 77 published releases.
• 24 March: litellm’s own build downloads Trivy without locking to a version, so it gets the poisoned one. The scanner reads litellm’s publishing key out of memory, and two booby-trapped versions go up under litellm’s name.
• 26 March: the US cyber agency adds it to its list of flaws under active attack.
• 2 July: the FBI issues an alert on the campaign, tracked as TeamPCP.

The wrong lesson is available and people are taking it. When a Rust package was compromised the same way on August 20, the popular comment was this:

All those folks telling me to update my dependencies, this is why I don’t do it. It’s not laziness, it’s undeniable foresight.

— vatsachak, Hacker News, 20 August 2026

Good joke, terrible advice. Never updating leaves you exposed to every flaw already found in what you have. The actual defence is locking to exact versions and choosing when to move, which is the opposite of both never updating and taking whatever is newest.

Two details deserve pulling out. The remediation failed silently: Aqua rotated keys, believed it closed, and the attacker was still inside. Incomplete rotation is the most ordinary failure in incident response, and it turned a contained breach into a five-week campaign. And a security scanner inside your build has exactly as much power as the build. Nobody thinks of it that way, because a scanner arrives labelled as the thing that protects you.

Why This One Concentrates

litellm is a switchboard for AI models, sitting between your software and every provider, holding an account key for each. Break into an ordinary component and you get whatever that machine had. Break into the switchboard and you get the keys to every AI service your company pays for, plus the wiring diagram. The malware’s shopping list reflected that.

That is now a category of target rather than an accident. Switchboards are the single richest place to steal from, precisely because their job is to hold every key in one place. On August 19 Stripe agreed to buy the biggest one on the market for around $8 billion, on the reasoning that it sits at a chokepoint. Attackers are reading the same map.

What This Isn’t

• Exposure is not compromise. Credentials were captured, which is not proof anyone used them against you.
• Not a Python problem. The same chain runs through the equivalent libraries for JavaScript and container images, and this campaign used all of them within a fortnight.
• The fixes are old and boring. Lock build tools to exact versions. Give publishing keys the narrowest power. Prefer short-lived credentials. And when you rotate after an incident, rotate everything.

The Number That Matters

Everyone quoted the forty minutes, in March and again this month, because it is satisfying: a tiny window, a lucky escape.

It was never the relevant one. Forty minutes measures how long the trap was open. It says nothing about the value of what walked into it. A cloud key with no expiry, a server login nobody has changed in years, an AI provider key pasted in once and forgotten: every one was still working on August 11, unless somebody happened to change it for an unrelated reason.

The half-life of the incident is the lifetime of the credentials it stole. On that measure it is still running.