One of the most fascinating and frightening incidents in computer security history started in 2022 with a few pushy emails to the mailing list for a small, one-person open source project.

A user had submitted a complex bit of code that was now waiting for the maintainer to review. But a different user with the name Jigar Kumar felt that this wasn’t happening fast enough. “Patches spend years on this mailing list,” he complained. “5.2.0 release was 7 years ago. There is no reason to think anything is coming soon.”.

A month later, he followed up: “Over 1 month and no closer to being merged. Not a suprise.” [sic]

And a month after that: “Is there any progress on this?” Kumar stuck around for about four months complaining about the pace of updates and then was never heard from again.

A few weeks ago, the world learned a shocking twist. “Jigar Kumar” does not seem to exist at all. There are no records of any person by that name outside the pushy emails. He — along with a number of other accounts — was apparently part of a campaign to compromise nearly every Linux-running computer in the world. (Linux is an open source operating system — as opposed to closed systems from companies like Apple — that runs on tens of millions of devices.)

That campaign, experts believe, was likely the work of a well-resourced state actor, one who almost pulled off an attack that could have made it possible for the attackers to remotely access millions of computers, effectively logging in as anyone they wanted. The security ramifications would have been huge.

Here’s how events played out: In 2005, software engineer Lasse Collin wrote a series of tools for better-compressing files (it’s similar to the process behind a .zip file). He made those tools available for free online, and lots of larger projects incorporated Collin’s work, which was eventually called XZ Utils.

Collin’s tool became one part of the vast open source ecosystem that powers much of the modern internet. We might think that something as central to modern life as the internet has a professionally maintained structure, but as an XKCD comic published well before the hack shows, it’s closer to the truth that “all modern digital infrastructure” rests on “a project some random person in Nebraska has been thanklessly maintaining since 2003.” XZ Utils was one such project — and yes, you should find it a little worrying that there are many of them.

Starting in 2021, a user going by the name “Jia Tan” — he, too, doesn’t seem to exist anywhere else — started making contributions to the XZ project. At first, they were harmless small fixes. Then, Tan started submitting larger additions.

The way an open source project like this one works is that a maintainer — Collin, in this case — has to read and approve each such submission. Effectively, Tan was overloading Collin with homework.

That’s when “Kumar” showed up to complain that Collin was taking too long. Another account that doesn’t seem to exist joined the chorus. They argued that Collin clearly wasn’t up to the task of maintaining his project alone and pushed for him to add “Jia Tan” as another maintainer.

“It seems likely that they were fakes created to push Lasse to give Jia more control,” engineer Russ Cox writes in a detailed timeline of the incident. “It worked. Over the next few months, Jia started replying to threads on xz-devel authoritatively about the upcoming 5.4.0 release.” He’d become a trusted “maintainer” who could add code to XZ Utils himself.

Why does any of this matter? Because one of the many, many open source tools that happened to incorporate XZ Utils was OpenSSH, which is used to remotely access computers and is used by millions of servers around the world.

“Tan” carefully added to XZ Utils some well-disguised code that compromised OpenSSH, effectively allowing the creators to log in remotely to any computer running OpenSSH. The files containing the (heavily disguised) code were accepted as part of the larger project.

Fortunately, almost all of the millions of potentially targeted computers were not affected because it’s routine for such a new update to first be released as “unstable” (meaning expected to have some bugs), and most administrators wait for a subsequent “stable” release.

Before that happened, “Jia Tan”’s work got caught. Andres Freund, a software engineer at Microsoft, was off work and doing some testing on a computer that had the “unstable” new release. Under most circumstances, the hack ran seamlessly, but under the circumstances he was testing in, it slowed down SSH performance. He dug deeper and quickly unraveled the whole scheme.

Which means that, thanks to one Microsoft engineer doing some work off-hours, your computer remains secure — at least, as far as I know.

There was nothing inevitable about this hack getting discovered. Lots of other people were running the unstable new build without noticing any problems. What made Freund suspicious in the first place wasn’t the suspicious code but a bug that had been accidentally introduced by “Jia Tan.”

If the “Jia Tan” team had avoided that error, they might well have pulled this off. Catching the suspicious code “really required a lot of coincidences,” Freund said later on Mastadon.

No one wants to believe that modern computer security essentially relies on “a lot of coincidences.” We’d much rather have reliable processes. But I hope this narrative makes it clear just how hard it is to reliably defend the jury-rigged internet we have against an attack like this.

The people behind “Jia Tan” spent more than two years building the access they needed for this attack. Some of the specifics have to do with the dynamics of open source software, where decades-old projects are often in a quiet maintenance stage from which, as we saw, an aggressive actor can seize control. But with the same resources and dedication that were behind “Jia Tan,” you could get hired at a software company to pull off the same thing on closed-source software too.

Most of all, it’s very hard to guess whether this attempted attack was unprecedented or unusual simply in that it got caught. Which means we have no idea whether there are other land mines lurking in the bowels of the internet.

Personally, as someone who doesn’t work in computer security, the main thing I took away from this was less a specific policy prescription and more a sense of awe and appreciation. Our world runs on unsung contributions by engineers like Collin and Freund, people who spend their free time building stuff, testing stuff, and sharing what they build for the benefit of everyone. This is inconvenient for security, but it’s also really cool.

I wasn’t able to reach Collin for comment. (His website said: “To media and reporters: I won’t reply for now because first I need to understand the situation thoroughly enough. It’s enough to reload this page once per 48 hours to check if this message has changed.”) But I hope he ultimately comes to think that being personally targeted by this fairly extraordinary effort to make his work on XZ utils feel inadequate is, in fact, a remarkable vindication of its importance.