A Microsoft engineer running database benchmarks noticed his SSH logins were taking 500 milliseconds longer than usual. That half-second of irritation is the only reason a backdoor planted in nearly every major Linux distribution didn't reach production servers worldwide. The attacker had spent more than two years earning trust before pulling the trigger — and almost got away with it because of one exhausted volunteer and an industry that confuses popular with audited.
The Setup
XZ Utils is a compression library. You've probably never heard of it, and that's the point. It ships inside almost every Linux distribution, gets pulled in by package managers, and silently links into critical services like SSH on Debian and Fedora derivatives. Compromise XZ, and you're one dependency away from servers running banks, hospitals, cloud providers, and government infrastructure.
In late March 2024, a backdoor was discovered in XZ versions 5.6.0 and 5.6.1. It was caught roughly two weeks after release, before it propagated into stable distributions. The fix was quick. The implications were not, and they still aren't.
What Everyone Assumed
The reigning belief about open source security is Linus's Law: "Given enough eyeballs, all bugs are shallow." More contributors, more reviewers, more scrutiny, therefore safer code. This belief shapes how enterprises trust dependencies they have never read a line of.
What the assumption ignores is staffing. Most foundational open source libraries are maintained by one or two volunteers in their spare time. XZ Utils had exactly one — Lasse Collin — and he had publicly admitted he was burned out. The "many eyes" do not actually look at C compression libraries. The reviewers everyone imagines are mostly imaginary.
What Actually Happened
A contributor calling themselves Jia Tan started submitting patches to XZ Utils in 2021. The patches were useful. Over months, this contributor became helpful, dependable, technically sharp — exactly the kind of volunteer an exhausted maintainer dreams of finding.
Then the pressure campaign began. New accounts with names like Jigar Kumar and Dennis Ens, none with any broader project history, started appearing on mailing lists complaining about slow patch reviews, demanding fresh maintainers, framing Lasse as the bottleneck. These accounts are now widely assessed to be part of the same operation. The pattern is textbook social engineering: isolate the target, manufacture urgency, present a solution. The solution was Jia Tan, conveniently already trusted.
By 2023, Jia Tan had commit access. In early 2024, version 5.6.0 shipped with a backdoor disguised as test fixtures — binary blobs labeled as "corrupted" test data that the build system would quietly stitch into the final library only under specific conditions. The payload hooked into a function used during SSH authentication, allowing whoever held a specific private key to execute commands as root on any affected machine. The activation logic only triggered where sshd was linked against liblzma, a quirk of how Debian and Fedora patch OpenSSH for systemd integration, not a behavior of upstream OpenSSH itself. The targeting was surgical.
It was discovered by accident. Andres Freund, a PostgreSQL developer at Microsoft, was benchmarking and noticed sshd was burning unusual CPU. He pulled the thread out of annoyance, not suspicion. The entire supply chain of modern Linux was saved by one engineer's irritation at slow logins.
Decoded
Stop thinking of supply chain attacks as code problems. They are trust problems running on multi-year timelines. The XZ attacker did not break cryptography or burn a zero-day. They exploited the loneliest fact about open source: critical infrastructure depends on a small number of unpaid, exhausted humans, and trust in those humans is transitive. Once you trust the maintainer, you trust the maintainer's collaborators, and you trust the new co-maintainer who showed up two years ago being helpful.
The mental shift is this: every dependency in your stack has a human attack surface, not just a code attack surface. The question is not only "is this library secure?" — it is "who has commit access, how recently did they get it, and how was that decision made?" That second question is almost never asked, and it is the one that mattered here.
What I keep coming back to is how close this was. If Andres had been a little less curious, or if the attacker had been slightly more careful with the activation cost, we would be having a very different conversation, probably months or years from now, in retrospect. The defense was not a tool. It was one person who refused to ignore a weird signal. So here is what I want to know from you: when was the last time you investigated something that was just slightly off, instead of dismissing it as nothing? Hit reply — I read every response.
Think clearly, — DJ Brar, SKBSEC | SKB Decoded www.skbsec.com