LiteLLM on PyPI: how TeamPCP used the supply chain to poison 95 million monthly downloads

# LiteLLM on PyPI: how TeamPCP used the supply chain to poison 95 million monthly downloads

On March 24, 2026, two malicious versions of LiteLLM — 1.82.7 and 1.82.8 — were published to the Python Package Index. The last known clean version is 1.82.6, published on March 22. In the window of roughly one day between the clean version and the malicious ones, LiteLLM went from being one of the most-downloaded AI infrastructure libraries in the Python ecosystem to a mass credential-exfiltration channel. What makes this incident especially severe is that LiteLLM handles language model API keys by design — every legitimate installation of the package has access to secrets an attacker would want to steal. The potential compromise scale is therefore proportional to the adoption scale: 95 million monthly downloads, present according to Wiz in roughly 36% of cloud environments scanned at the time of the incident, with users including Stripe, Netflix, and Google ADK itself.

Who TeamPCP is and why it matters

TeamPCP is a threat actor that self-identified with that name — the initials appearing in artifacts such as `tpcp.tar.gz` are the most direct clue — and that orchestrated between March 19 and March 24, 2026 a four-wave campaign against the supply chain of security and development tools. The particularity that distinguishes TeamPCP from other supply chain actors is that they started by compromising security tools: first Aqua's Trivy, then Checkmarx's KICS and AST GitHub Actions. Tools that any reasonable DevSecOps team would have in their pipeline, running with broad access to credentials and source code. The compromise chain was sequential: Trivy compromised stole PyPI publishing tokens that LiteLLM used in its CI/CD; those tokens were used to publish the malicious versions on PyPI; the malicious versions stole credentials from environments where they were installed, which were used in the next wave. It is the supply chain attack pattern in its most chilling form: each wave produces the credentials that feed the next.

CVE-2026-33634, with a CVSS of 9.8, covers the Trivy binary and GitHub Actions compromise. PYSEC-2026-2 covers the malicious LiteLLM PyPI packages. The Checkmarx KICS and AST GitHub Actions compromise has no assigned CVE at the time of writing this article, which is a reminder that not all TeamPCP activity is fully covered by the formal vulnerability identification system.

Anatomy of the malicious payload

The malicious code in LiteLLM 1.82.7 and 1.82.8 operates in three stages. The first is credential collection. The payload searches more than 50 known paths where secrets are stored in typical Python environments: AWS credentials files, AWS Secrets Manager contents, SSM parameters, environment variables containing AI provider tokens such as OpenAI, Anthropic, or Google, Kubernetes configurations at `~/.kube/config`, and locally stored GitHub tokens. Exfiltration is directed at an attacker-controlled domain: `models.litellm.cloud`, a name carefully chosen to look legitimate to casual observers.

The second stage is lateral movement. In Kubernetes environments, the payload attempts to deploy privileged pods on every node of the cluster using whatever Kubernetes credentials it managed to collect. The deployed pods have enough access to persist even after the original container is destroyed. In non-Kubernetes environments, lateral movement takes other forms — SSH connections to known machines, reading cloud instance metadata, pivot attempts through internal services the collected credentials can reach.

The third stage is persistence. The payload installs a Python script at `~/.config/sysmon/sysmon.py` and a systemd unit at `~/.config/systemd/user/sysmon.service`, with the display name "System Telemetry Service" to disguise itself as a legitimate system service. The systemd unit starts at boot and maintains communication with the attacker's server, awaiting additional payloads or update instructions. The persistence is designed to survive reboots and, in many cases, updates to LiteLLM itself.

The .pth file variant and why it is more dangerous

LiteLLM 1.82.7 contained the stealer embedded inside the `litellm/proxy/proxy_server.py` file, which executes when someone imports LiteLLM in their Python code. That limits activation to environments where LiteLLM is actually used — a development server, a pipeline, an application — but excludes environments where LiteLLM is installed but not imported.

LiteLLM 1.82.8 added a second activation path: a `.pth` file in the package, named `litellm_init.pth`. `.pth` files in Python are processed automatically by the interpreter every time it starts, without the package needing to be imported. That means any Python process that starts on a machine where LiteLLM 1.82.8 is installed executes the code from the `.pth` file, regardless of whether LiteLLM is imported. A CI/CD script that runs `python something.py` where LiteLLM is in the environment executes the payload. A Jupyter notebook that starts executes the payload. A testing tool that starts Python to validate code executes the payload. The effective scope of the compromise multiplies.

There is a detail that reveals the actor's level of improvisation: the involuntary fork bomb. The `.pth` file uses `subprocess.Popen` to launch a child Python process, but that child process also processes the `.pth` files, including the same `litellm_init.pth`, creating an exponential recursion of forks that exhausts system resources and locks it up. It is a bug in the malware — the malicious LiteLLM partially self-destructs — but it is also an example of how even attackers make mistakes when they iterate quickly.

How the PyPI publishing tokens were compromised

LiteLLM published to PyPI from a CI/CD pipeline that used Aqua's Trivy for vulnerability scanning. When Trivy was compromised in TeamPCP's first wave, the malware inside Trivy could access the memory of the CI/CD runner where it executed. The PyPI publishing tokens LiteLLM had configured as repository secrets were in that memory or were accessible from the runner context. TeamPCP exfiltrated them and used them two days later to publish the malicious versions, bypassing LiteLLM's legitimate release process — in fact, versions 1.82.7 and 1.82.8 have no corresponding tag or release in LiteLLM's GitHub repository; they appeared directly on PyPI.

This detail is crucial to understanding the chain. The attacker did not need to compromise the LiteLLM repository directly or its maintainers. They needed to compromise an upstream tool in the pipeline, and the tokens fell as a side effect. It is exactly the supply chain attack pattern the industry conversation has been describing for years, made real in the most uncomfortable way: attacking the security tools the industry presumed would defend against that same kind of attack.

Affected versions and confirmed clean line

Affected versions are LiteLLM 1.82.7 and LiteLLM 1.82.8, both published on March 24, 2026 — 1.82.7 at 10:39 UTC, 1.82.8 at 10:52 UTC. The 13-minute gap between them and the addition of the `.pth` vector in 1.82.8 indicates the attacker was actively iterating during the attack window. The last confirmed clean version is LiteLLM 1.82.6, published on March 22, 2026. PyPI quarantined the entire LiteLLM project around 13:30 UTC on March 24; the quarantine was later lifted and maintainers handled the recovery.

For any organization, the immediate action is: audit every environment where LiteLLM is installed and confirm the exact version. If 1.82.7 or 1.82.8 appears anywhere — production, CI/CD, developer workstations, ephemeral containers — it must be treated as a compromised environment. Rotation of all credentials that have been accessible from that environment is mandatory, not optional.

What to do in a compromised environment

The response to a LiteLLM 1.82.7 or 1.82.8 compromise is not just uninstalling the package and reinstalling the clean version. The payload already executed, and indicators of compromise are likely on the system. The reasonable response has six steps.

First, isolate the environment. If the machine is a production server, take it off the network immediately to prevent active lateral movement. If it is a developer workstation, ask the developer to stop working on it until the investigation is complete.

Second, preserve evidence. Capture a disk image or at least a snapshot of the key files: `~/.config/sysmon/` if it exists, the contents of `~/.aws/credentials`, `~/.kube/config`, any environment variable containing AI provider tokens, CI/CD runner logs if the compromise was in a pipeline.

Third, rotate all credentials that have been in that environment. This includes: AWS access keys, Kubernetes tokens, GitHub tokens, AI provider API keys, secrets from any service the machine could reach. Rotation must be complete; it is not enough to rotate only "the ones that look compromised" because the payload searches more than 50 paths and we do not always know which ones it reached.

Fourth, search for persistence indicators. The `~/.config/sysmon/sysmon.py` script and the systemd unit are the best known, but the attacker may have installed other mechanisms. Look for anomalous cron jobs, unknown SSH keys in `~/.ssh/authorized_keys`, suspicious entries in `/etc/passwd` and `/etc/shadow`, active systemd services that do not correspond to known deployments.

Fifth, rebuild from scratch. A machine compromised by this kind of payload should not be "cleaned"; it should be rebuilt. The persistence of the payload is designed to be difficult to detect completely, and the cost of residual compromise vastly exceeds the cost of rebuilding.

Sixth, review your own supply chain. If your organization had compromised LiteLLM, it may also have had any other pipeline tool compromised that shares CI/CD runners. Review the rest of the pipeline, not just LiteLLM.

What changes after TeamPCP

This incident accelerates several conversations the supply chain industry has been having for years. The first is about separation of responsibilities: when a security tool is the compromise vector, who is responsible? Aqua distributed compromised Trivy; Progress distributed LiteLLM; Checkmarx distributed compromised KICS. All three are companies with formal security teams. The reality is that the supply chain is only as strong as its weakest link, and one compromised link can poison the rest even if every individual link has reasonable security practices.

The second conversation is about publishing verification. PyPI does not have a robust attestation mechanism that allows a consumer to verify that a published version actually corresponds to the source code in the repository. in-toto and sigstore tools exist, but their adoption is voluntary and fragmented. As long as that remains the case, anyone with compromised publishing tokens can publish malicious versions without the consumer being able to detect it by automatic cryptographic means. It is an Internet infrastructure problem, not a LiteLLM-specific one, and it needs attention at the standard level.

The third conversation is about supply chain monitoring. Traditional SCA (software composition analysis) tools scan dependencies for known vulnerabilities, but do not scan for malicious behavior. After TeamPCP, that limitation is more evident than ever: the malicious LiteLLM versions were new versions, with no known vulnerabilities in advisory databases, with behavior that only becomes evident at runtime. Supply chain monitoring must evolve toward behavioral analysis of packages in sandboxes, not just comparison against vulnerability databases.

Closing

TeamPCP used the supply chain as the vector, not the target. The target was the credentials that any legitimate LiteLLM installation has access to. The chain — Trivy, then LiteLLM, and the credentials that will come — is an uncomfortable reminder that security tools are not immune to being compromised, and that the cost of a single weak link multiplies downstream. For organizations, the operational lesson is direct: audit versions, rotate credentials, rebuild affected environments from scratch, and start taking supply chain monitoring seriously beyond traditional scanning. For the industry, the lesson is structural: the modern supply chain needs robust cryptographic attestation by default, behavioral sandbox monitoring by default, and effective separation of responsibilities between publishers, registries, and consumers. Without those changes, TeamPCP — or the next actor iterating on the same pattern — will succeed again.

What defenders can do today while standards catch up

While the supply chain ecosystem evolves toward cryptographic attestation and mandatory sandbox monitoring, defenders cannot wait. There are practical steps that reduce exposure today. Pinning specific versions in lock files is the most basic — refusing to install anything outside the pinned range, including transitive dependencies. Pinning does not stop a malicious version from being installed if it has already been published, but it stops automatic upgrades from picking up a malicious version silently. Hash verification on top of pinning, available in tools like pip with `--require-hashes`, adds another layer: even if a malicious version sneaks past pinning, a hash mismatch will block installation. Neither measure is foolproof, but together they raise the cost of the attack.

Private package mirrors are another practical defense. Organizations running their own mirror of PyPI — or using a service like Sonatype Nexus, JFrog Artifactory, or Cloudsmith — can curate the set of packages available to internal environments. A malicious version of LiteLLM that reaches public PyPI does not necessarily reach the internal mirror if the mirror is configured to allow only specific, vetted versions. The mirror becomes a policy enforcement point that complements the version pinning in lock files.

Behavioral monitoring at the runner level is more sophisticated but increasingly feasible. Tools that sandbox Python processes and observe what files they touch, what network connections they make, and what subprocesses they spawn can flag suspicious behavior even from packages that look legitimate. The performance cost of sandboxing has dropped enough that it is viable for CI/CD runners in many organizations. After TeamPCP, that investment looks less like a luxury and more like table stakes for any organization that runs non-trivial CI/CD pipelines.

What the response coordination actually looked like

The TeamPCP incident is also a study in how the security community coordinated a response under pressure. Aqua Security, Progress (LiteLLM maintainers), Checkmarx, PyPI maintainers, and a long list of independent researchers at Endor Labs, JFrog, Cycode, Phoenix Security, and others all converged on the problem within hours of the malicious versions being identified. Public write-ups appeared within the same day; the PyPI quarantine happened within roughly three hours of the second malicious version; the maintainers of LiteLLM communicated publicly and directly through their incident response process. That coordination is part of why the long-term damage, while real, was bounded — many organizations learned about the malicious versions before they could affect them, because the disclosure happened quickly.

The lesson there is positive but also fragile. The coordination worked because the actors involved — researchers, vendors, package registries — had established relationships and clear playbooks for supply chain incidents. That is not guaranteed for the next incident, especially if the affected package is more obscure, the maintainers less resourced, or the disclosure happens through less coordinated channels. Building the muscle memory of supply chain incident response before the next TeamPCP arrives is itself a form of preparation that pays off the first time it is needed.