On March 28th, 2024, while benchmarking the database software that he worked on, called ‘PostgreSQL,’ Microsoft engineer Andres Freund noticed something peculiar, a delay of just over 500 milliseconds when authenticating. Investigating further, he discovered a supply chain attack giving a malicious actor backdoor access to the billions of computers running Linux worldwide. This supply chain attack, a type of cyberattack where hackers add malicious code to a library depended on by their target, was accomplished by backdooring the XZ Utils library, which is used by many linux distributions. Thankfully it was caught mere weeks before becoming a threat, but this has prompted the cybersecurity community to ask, what prevents something similar from happening in the future?
The attack was a long time in the making. Having started in 2021, the attackers relied on social engineering in order to accomplish their goals. A user by the name of “Jia Tan” began committing patches to the XZ Utils repositories, pieces of code that provide compression and decompression tools for .xz files. The majority of these commits were helpful and did not contain any malicious code. However, in 2022, a pair of users named “Jigar Kumar,” and “Dennis Ens,” both likely also personas from the same malicious actor, began to pressure the maintainer, Lasse Collin, on email lists to give Jia Tan the privileges of co-maintainer on the repository. These emails appear to be social engineering attempts to pressure Dennis Ens into ceding access to the repository, saying things such as:
“Is XZ for Java still maintained? I asked a question here a week ago and have not heard back. When I view the git log I can see it has not updated in over a year.” – Dennis Ens
The aforementioned XZ for Java is a version of the library written for use with Java, since the other XZ implementation is written in C.
“Patches spend years on this mailing list. There is no reason to think anything is coming soon.” – Jigar Kumar
“Over 1 month and no closer to being merged. Not a surprise.” – Jigar Kumar
These pressuring emails served to force Dennis Ens to give maintainer access to Jia Tan, who would later abuse his privileges. The use of multiple personas to trick Ens is much more effective, as it introduces social pressure and allows the hackers to manipulate the situation however they want.
“I am sorry about your mental health issues, but its important to be aware of your own limits. I get that this is a hobby project for all contributors, but the community desires more. Why not pass on maintainership for XZ for C so you can give XZ for Java more attention? Or pass on XZ for Java to someone else to focus on XZ for C? Trying to maintain both means that neither are maintained well.”
“As I have hinted in earlier emails, Jia Tan may have a bigger role in the project in the future. He has been helping a lot off-list and is practically a co-maintainer already. 🙂 I know that not much has happened in the git repository yet but things happen in small steps. In any case some change in maintainership is already in progress at least for XZ Utils.”
Ens eventually did give Tan maintainer access. His initial work was innocuous, but a new, likely fake persona named Hans Jansen was laying groundwork for the backdoor. Two years later, in February 2024, they committed and merged malicious binary code to a test files directory. This wasn’t flagged as suspicious because binary files, which aren’t readable to humans, are commonly stored there for testing the library. Tan then added code to attach the newly added backdoor binary to the final executable. Weeks later, when they discovered the malicious code, Tan was stripped of his maintainer role, the code was removed, and all of their past commits were heavily scrutinized.
As of this article’s publication, the identity of “Jia Tan,” the account involved in the attack, is unknown. Given the deliberate and long nature of their actions, it is likely not to be a personal account, but rather a state sponsored hacking group.
Almost all of the internet is built on top of free, open-source software. These projects have publicly available source code with licenses granting everyone the freedom to read, modify, and redistribute its source code. A similar concept is the creative commons license on images. Compared to proprietary software, such as windows, open-source software gives more power to users. Some common open-source software you may have used includes Linux, Chromium, and Git. Critical projects like these benefit from having an open development model because anyone can audit and contribute to them.
In this attack, hackers were able to target a single person, maintaining a single project, responsible for the continued operation of almost the entire internet. This thankless volunteer, who was not paid for his work, was busy with his life and became a weak link in the security of countless critical internet systems. This sheds light on a major problem with how open-source development is currently structured, where companies that benefit from maintainers’ volunteer work do not properly give back in the form of maintenance and funding. It makes sense that companies should give back to these projects, both out of moral duty and to secure their infrastructure. Specifically, if companies sponsored the maintainers’ work or offered their own employees to help, attacks like this one would be less likely to succeed, since the scenario of an unpaid solo-maintainer would be mitigated.
In the future, it is critical that more care be taken to monitor and secure the open software that the world depends on. Had this attack been successful, the hacking group behind it could have gained access to almost every updated Linux server computer, bringing the internet to its knees and resulting in immense damages. Given that it was only detected almost by accident, better ways to detect and prevent such attacks in the future are necessary.
In the future, it is critical that more care be taken to monitor and secure the open software that the world depends on. Had this attack been successful, the hacking group behind it could have gained access to almost every updated Linux server computer, bringing the internet to its knees and resulting in immense damages.