The IT Security community had an interesting Easter weekend. Somebody very smart almost hacked 20 million internet servers, but they were found out in the nick of time by one guy from San Francisco called Andres Freund. If it was a movie plot, you’d call it far-fetched.
The incident unfolds
The situation began to unfold on Good Friday with a concerning post on Mastodon by Andres Freund, a Microsoft engineer specialising in the PostgreSQL open-source database. Freund had noticed unusual timing behaviour on one of his test servers: the SSH service, used for remote login to Linux, was using far more resources than normal.
Being an expert on system performance, he wanted to know why, and he knew how to pin down such problems. He found that the culprit was a common software library called XZ. It is used by many programs for data compression.
The reason for the slowdown was that a backdoor had been added to XZ, specifically targeting SSH to allow an attacker to access systems with their own private login key, without the owner of the system knowing about it.
The potential impact
Scans of the internet show 20 million IP addresses with the SSH service listening for connections. Web servers, e-mail servers, infrastructure servers, database servers, all sorts of servers. If this software had spread to all of them, the attackers would have been able to remotely make these servers do whatever they want. Delete them, quietly steal information that they handle, change data, … anything. It would not be an exaggeration to say that they would have been able to control a large part of the internet, and to listen in on a lot of confidential communications.
Fortunately, Andres runs systems with much more recent software than 99% of us. So this had only just happened, and was not yet included in any mainstream Linux releases. Huge sigh of relief all around. Very lucky escape.
How could this happen?
The backdoor was very cleverly hidden. It can’t be found by looking at the source code for XZ. Somebody added it to a release script, a small program that builds the software, packs it up and sends it off “downstream” to be included in Linux operating system releases. Only after XZ is built from source code is the backdoor code injected into the files that are sent to be run on other systems.
Also, the backdoor is built so that it is not detectable from the network. If XZ is included in SSH, it acts only when a login request is received from someone who has a specific, secret key. Then, and only then, will the backdoor run a command for its owner. If you don’t have that secret key, there is no way to know that a server is vulnerable. Only the server owner could find it, provided, of course, (s)he knows about the issue and knows where to look.
If Andres hadn’t noticed the slight timing issue, this might have taken a very, very long time to be discovered.
Whodunnit
The XZ version with the backdoor was released by somebody calling themselves Jia Tan. It is now believed that this is not a real person.
Lasse Collin is the owner and inventor of XZ. He maintained the software for free, for many years. After being pressured to work harder on his project[1], and suffering from illness, he gave a very friendly new volunteer called Jia Tan access to edit the software and to make releases, and took some well-earned vacation. We now think that the pressure campaign, which started in 2022, was orchestrated so that “Jia Tan” could become the maintainer of XZ and thus be able to release malicious software into the larger internet ecosystem.
Getting control of a widely-used project, developing a complicated backdoor, and hiding it, must have taken serious investment. Some people are saying that some of the large criminal hacking groups can afford to build something like this, but the main suspect would be a national intelligence service. We can’t be sure, of course, but who else would spend years building this, and getting it included into an important piece of internet infrastructure by finding a widely-used project maintained by only one overworked person?
[1] Look at this e-mail exchange from 2022, for example: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html – “Jigar Kumar” and “Dennis Ens” demanding that Lasse give control of his project to others; conveniently, “Jia Tan” then volunteers
What do I do?
We’re very lucky that this was caught in time. “Jia Tan” was busy just last week pushing Linux maintainers to quickly adopt the latest version of XZ into their main releases, but this had not happened yet. So, unless you are running “bleeding edge”, unstable testing releases of Linux, you are most probably fine. But, update just the same, to make extra sure: updates have now been released that roll back the latest, infected releases of XZ (5.6.0 and 5.6.1).
Also, a recommendation that we at Securance always give to our clients is not to expose services like SSH, that are meant to be used only by a few IT staff, to the open internet. Always limit access in the firewall to internal IP addresses, and perhaps a few home addresses of trusted staff. That way, if a new vulnerability in the service is discovered, attackers simply won’t be able to connect to your servers to exploit it.
Lessons for the future
Supply chain attacks are here to stay, because complicated supply chains will continue to exist. A well-known cartoon puts it like this:
We (internet users) need to support these “random persons” a little more, and be aware of their importance to our security. Lasse Collins was building and maintaining this critical piece of software for us in his spare time, for nothing. And there are many more Lasses out there.
We at Securance support some open source projects whose excellent software we use for free. We will look at supporting more of them with a monthly donation. No matter how small – these can really make a difference.
