Anyone who works hard in the trenches of the internet will tell you that it is not a well-oiled machine that runs without the slightest problem.
Rather, it is a collection of disorganized parts that have been assembled over decades and are only held together by the digital equivalent of duct tape and mascara gum. Much of the Internet depends on open source software that is maintained by the work of a small army of volunteer programmers who no one thanks for fixing the bugs, patching the holes, and making sure that rickety contraption, which runs billions, of dollars in global gross domestic product can barely keep going.
It is very likely that last week one of those programmers saved the Internet from a huge problem.
His name is Andrés Freund. He is a 38-year-old software engineer who lives in San Francisco and works for Microsoft. Part of his job involves developing a piece of open source software for managing databases known as PostgreSQL. If he could properly explain what this software is about (something I definitely can’t do), maybe he’d just bore them to death.
Recently, while performing some routine maintenance tasks, Freund inadvertently discovered a hidden backdoor in a piece of software that is part of the Linux operating system. That backdoor may have been the prelude to a major cyberattack that, experts say, could have caused terrible damage if it had come to fruition.
Now, in a Hollywood twist, several tech industry leaders and cybersecurity researchers are calling Freund a hero. Microsoft CEO Satya Nadella praise his “curiosity and skill.” A fan what was described as “the gorilla leader of the nerds.” There’s been an old web comic strip circulating among engineers, famous among programmers, whose premise is that all modern digital infrastructure depends on a project maintained by some guy in Nebraska (they say Freund is that guy).
In an interview this week, Freund—who is actually a soft-spoken, German-born programmer who did not want his photo taken for this article—said that becoming an online folk hero has caused him great confusion.
“It seems very strange to me,” he said. “I’m a pretty private person who just sits in front of the computer and produces code.”
The saga began earlier this year, during Freund’s flight home from visiting his parents in Germany. While reviewing an automated test log, he noticed that there were a few error messages that he didn’t recognize. At the time he was suffering from jet lag and the messages didn’t seem urgent, so he filed them away in his memory.
But a few weeks later, while running other tests at home, he noticed that an application called SSH, which is used to remotely log into computers, was using more processing power than usual. After searching for the source of the problem, which he traced to a set of data compression tools called xz Utils, he wondered if it was related to errors he had seen before.
(Don’t worry if these names sound like I’m speaking Chinese; in reality, you just need to know that they are small fragments of the Linux operating system, which is perhaps the most important open source software in the world. The vast majority of servers world—including those used by banks, hospitals, governments, and Fortune 500 companies—run on Linux, so its security is of global importance.)
Like other popular open source software, Linux is updated frequently and most bugs are due to innocent mistakes. However, when Freund took a closer look at the xz Utils source code, he found clues that indicated that someone had intentionally altered it.
In particular, he discovered that someone had planted malicious code in the most recent versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine.
At first, Freund doubted his findings. Had he really discovered a backdoor in one of the most scrutinized open source programs in the world?
“I felt like it was surreal,” he said. “I thought several times that maybe she had slept badly and was delirious.”
But as he continued analyzing, he identified new evidence, so last week Freund shared his findings with a group of open source software developers. The news soon caused alarm in the technological world. In just a few hours, a fix was created and some researchers credited Freund with preventing what could have been a historic cyberattack.
No one knows who planted the backdoor, but the plan was apparently so elaborate that some researchers are convinced it could only have been tried by a nation with tremendous skills at devising cyber attacks, such as Russia or China.
According to some researchers who have reviewed the evidence, everything seems to indicate that the attacker used a pseudonym, “Jia Tan”, to suggest changes to xz Utils as early as 2022 (many open source software projects are governed by a hierarchical system; developers They propose changes to a program’s code, and then more experienced programmers are responsible for reviewing and approving the changes.)
The attacker, using the name Jia Tan, is believed to have worked for several years to gradually gain the trust of other xz Utils developers and gain more control over the project, until he rose through the internal hierarchy and eventually inserted the code . with the backdoor hidden earlier this year (although the new manipulated version of the code had already been released, it was not yet in widespread use).
Freund said that since his findings were made public, he has dedicated himself to helping teams trying to reverse-engineer the attack to identify the culprit. So he’s been too busy to rest on his laurels. The next version of PostgreSQL, the database management software he works on, is due out later this year, and Freund is still looking to get some last-minute changes accepted before the deadline.
“I don’t really have time to go have some drinks to celebrate,” he said.
Kevin Roose He is a Times technology columnist and host of the podcast hard fork. More from Kevin Roose
