On a flight back from visiting his parents in Germany, Andres Freund noticed a few odd Valgrind errors in a log of automated tests. He was jet-lagged, they did not seem urgent, and he filed them away. A few weeks later, running micro-benchmarks on his Debian unstable machine, he saw sshd processes burning far more CPU than immediately-failing logins could explain. He profiled sshd and found the time going into liblzma, a compression library that should barely be doing anything at all, with perf unable to attribute it to a symbol. He got suspicious. He remembered the valgrind complaint from weeks earlier, after a package update. Then he did the least glamorous thing in software: he went and read the source.
What he found, on March 29, 2024, was the most serious supply chain attack yet found in open source. The upstream xz repository and its release tarballs had been backdoored. Someone using the name "Jia Tan" had spent more than two years earning trust in a project maintained largely by one burned-out volunteer, Lasse Collin - sock puppets complained loudly about slow releases, Jia Tan kept showing up with useful patches, and by early 2024 the helpful newcomer was one of the project's two main developers and was cutting its releases. The malicious code landed in versions 5.6.0 and 5.6.1, and the target was not even xz itself. It was OpenSSH, the remote login door on nearly every Linux server.
The design was patient in a way that still bothers me. The injected code triggered only on x86-64 Linux built with gcc and the GNU linker, only inside Debian or RPM package builds, and once running it checked that the process was named /usr/sbin/sshd, that TERM was unset, that LANG was set, that no debugger flags were in the environment. Started under a researcher's clean environment, it behaved. It hid from analysis the way a thief avoids a camera's field of view. And it never touched sshd's own code: sshd does not use liblzma at all. Distributions that patch OpenSSH to support systemd notification pull in libsystemd, which depends on lzma, and that dependency chain was the delivery route into a program that never asked for the code.
Every criterion I know for judging a secure system would have passed this machine. The commit history read like a maintainer fixing bugs. The releases built, passed their tests and shipped to beta builds of the major distributions. Red Hat assigned the issue CVE-2024-3094 after Freund reported it.
What was not silent was the half second. On Freund's own machine, an ssh login attempt went from 0.299 seconds to 0.807 seconds with the backdoored library installed - and the user-mode CPU time was identical in both runs, 0.202 seconds. The extra half second was the backdoor parsing symbol tables in memory and redirecting a cryptographic function to its own code before sshd opened for business. The disclosure email carried the finding in one flat line: "The upstream xz repository and the xz tarballs have been backdoored."
I keep asking myself what I would have done with a 500 ms slowdown in ssh logins on my own servers. I know the answer, and it is embarrassing: I would have blamed the hypervisor, or DNS, or something I changed last week - if I checked at all. Half a second is exactly what a busy site owner reboots around. Freund had two advantages no product sells. The first is context: he was micro-benchmarking, so the noise sshd added landed in front of the one person who needed a quiet machine. The second is that when his profiler showed him a weird picture, he did not close the tab. He pulled on the thread - first through valgrind, then through the source, then to the tarball diff where the backdoor's first stage lived, a line present in the release tarballs and absent from the git tree.
That is the uncomfortable takeaway for anyone who runs a site: the attack that nearly ended in a master key to the internet's servers was caught by a person who noticed it was half a second slower than it should be, and cared. Not by a scanner. Not by a threat feed. By familiarity with normal.
A reproducible-builds rig that diffs the release tarball against the git tree would have flagged the first stage, and that discipline deserves wider adoption. But scanners, dashboards and alerts only fire on thresholds you already imagined, which is exactly what the xz author was counting on: the whole design was a profile of what maintainers, reviewers and automated checks look at, and it stayed quiet on every one of those channels. The delay was not a monitored metric. It was a side effect of the exploit doing its work in process memory, in the seconds before a login is allowed or refused - and side effects are the one channel no attacker fully controls.
Familiarity is also something you can actually buy, in a small way, for the kind of sites most of us run. A checkout that normally settles in 1.2 seconds and one day takes 4 is telling you something. A webhook queue that never backs up suddenly crawling is telling you something. On Bid Spotlight, where I operate, a listing only climbs when Stripe's verified checkout.session.completed webhook lands - no redirect, no promise on a page, ever moves rank. The value is not the happy path. It is that the event stream is small and legible enough that I can actually read it, so when a payment settles oddly I can pull on the thread the same way Freund did.
The internet got lucky. Alex Stamos told the New York Times this "could have been the most widespread and effective backdoor ever planted in any software product" and that undetected it would have "given its creators a master key to any of the hundreds of millions of computers around the world that run SSH." The affected versions had only reached pre-release builds when the disclosure froze them out of the major distributions, and the clean 5.6.2 release followed in May 2024. The luck was one person who knew what half a second of sshd felt like.
Know one number on your site the way Freund knew sshd. The cheapest intrusion detector you will ever own is your own ability to say: that usually takes half a second, and today it does not. If you run a site and want a rank you can verify that way, bid your way to the top spot on Bid Spotlight.