From what I have read so far: speculative, at best.
It’s all based on the sophistication, the level of psychological manipulation, years of effort, multiple actors, and a working theory based on commit timings.
Generally speaking, with attacks of this sophistication, if it's not a state-backed APT, it's someone who operates with the unofficial backing of a government to give plausible deniability.
I suppose it could be a few amateurs who took on a passion project, but... it's unlikely.
Could be someone at a company that hacks into stuff and sells it to governments too. They're always looking for backdoors and I assume would do stuff like this.
AIUI, it was the dynamic loader audit hook checking symbol names against strings stored in a trie as a way to hide them. Tons of symbols, and slow process for each one.
My working theory is they got sloppy. A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).
They were rushing to get it out into the wild and thus made many mistakes, including the performance degradation.
From what I recall, some distros were reporting build errors in bleeding edge versions. Actors were essentially debugging issues in production, and recommending to distro maintainers ways to “fix” (really hide) the build issue with xz
I've worked on long projects for a while, you can be relatively invisible for a year and let them know you're getting close (meaning a couple months) and the management will freak out and want it yesterday if it's near the end of a quarter and they think it will look good on quarterly reports. I assume the same can happen with government actors and "security companies".
A reasonable person would expect many more open source projects have been infiltrated and have similar exploits waiting to be found. Hard to prove until the next one is found, but if a second one like this is discovered we probably have a huge problem on our hands.
Since the part that takes longest is getting commit rights and there's a lot of waiting to do, what are the chances the attackers put all their eggs on infiltrating one specific project? I'd be trying to infiltrate at least 10 or 20 even if I were alone, to increase my chances. If you dedicate one or two years to build reputation with the current maintainers by doing a PR here and there, you can do that for a few projects in parallel.
I'm not saying it was a bad target. It was a perfect target. How many more would fullfil these criteria? Those would be the type of repositories I'd go and investigate.
For there to be another repository affected by the same attack you don't even need to think this specific attacker did it twice. You can just consider that multiple attackers had the same idea. I think it's way more likely that there's more repositories affected by the same type of attack than not.
I'd go as far as saying this is an issue of national / global security and there's space for an agency like the IMF but for software security that would be funded by all the countries of the world to bring more resources and assuredness to this type of dependencies.
Because even if this attacker didn't do more than N=1 and other attackers didn't have the idea before, they surely have it now.
Yes, this has happened. See this example from OpenJSF:
"The OpenJS Foundation Cross Project Council received a suspicious series of emails with similar messages, bearing different names and overlapping GitHub-associated emails. These emails implored OpenJS to take action to update one of its popular JavaScript projects to “address any critical vulnerabilities,” yet cited no specifics" [0,1].
I watched the TC interview with Durov (Telegram) and apart from it being a big Telegram ad what struck me is that he told a story about his employee being approached by "an intelligence agency" and asked to reveal information about what open-source libraries they use.
It is kind of strange since their apps are supposed to be open source, maybe he meant the backend? Nonetheless, it seems this has been their modus operandi for a long time.
Remember that NSA is openly interested in systemd and how it works. It's a double edged sword. They wanted to be sure that it's hardened as they like it, and note any "useful features" that might come handy later.
The thing is, as computers proliferate and we start to use them in more places, the effects of possible holes moves closer to our homes. From distant infra to near infra; from borderlines to our homes and transportation we use everyday. Even to our pockets via smartphones and other smart devices we host in our homes.
I would start with projects where the maintainer(s) suddenly got a very helpful contributor. I am sure there's a typical pattern. Then if a maintainer seems suspect investigate a little bit more. Here some balance is needed because probably many maintainers are not Mallory.
I would have paid money to see the look on the team’s face behind this back door as they realize the years of effort and planning go down the drain because of 1 person doing due diligence.
CISA (https://cisa.gov) are investigating. They got the emails between me and "Jia Tan" and other information we have. They can subpoena Google to get IP addresses behind the emails, but it depends on whether and where a VPN was used if they'll get any further than that. If we will ever see a fully public report is anyone's guess.
If it's a state actor, it's not unreasonable to think they'd spin up their own VPN
All you need a Raspberry Pi and some public Wi-Fi network to create a jump point and hide among the 100s of devices going on and off that one public IP. With projects like TailScale you could set it up and plant it somewhere in a matter of hours
If it is a state actor like China or North Korea, they own the gateways and firewalls and can ensure any traffic they do not want to be identified ever will be.
I also recently wrote a single ephemeral socks5 proxy over a hidden service in Rust. Since they’ve probably compromised other machines in the past, they could’ve easily used something similar to proxy their connection through tor and to some random computer (access some vulnerable router through tor, proxy through it, etc).
Hiding your tracks aren’t hard
Nothing but speculation so far based on the times they were active. No real signatures left behind.
All we know is they were active during business hours of the UTC+1/3 timezones. They used a chinese sounding name as one of their front figures, the one who made most of the commits of malicious code. They wrote english messages that some people claim appear to be a Russian trying to write in English. But that last argument is pretty weak as they could have just masked their origin using Google translate, or even ChatGPT.
One of the biggest clues is actually one little commit that they let slip with a middle name to the chinese persona. And some people claim the names don't match up with different chinese cultures.
After that you can use all your psych 101 experience to try and analyze them but you'll just be guessing.
Personally I think using a chinese front name is a very interesting detail. They could have used any name in the world, there are open source contributors from all over the world.
Using a name in a country where most people have a very little insight is a good way of not being uncovered as fake of course. But why not use a Ugandan name? Also using a Chinese name can be perceived as pointing the blame at China, which might be sensitive geopolitically.
They also used a bunch of other fake names (Jigar Kumar, Dennis Ens) for the personas that put social pressure on the original maintainer. And the ifunc changes were done by one "Hans Jansen" who also later opened merge request to Debian. These are fairly eclectic names with no pattern, which was the point I guess. We don't know what level of misdirection is going on with the names, so "Jia Tan" is interesting but not particularly meaningful either way.
I would argue that Hans Jansen and Dennis Ens (and Jigar Kumar? I don't know about Indian perceptions of names) are a a bit off as well. They contain rhymes that make them sound slightly silly. Most parents would want to avoid these. Maybe chosen to avoid matching real people while still seeming commonplace.
So these names seem somewhat similar to "Jia Tan" in their almost-but-not-quite real quality.
Jia Cheong Tan is anagram for CIA Agent John. If you consider how much went into planning the backdoor operation, it seems virtually certain it's an internal joke.
And why an Asian name? It's certainly exploiting a psychological bias. Apart from what you noted, the names are much more generic than Ugandan names and IMO it's virtually impossible to track a legend down. And the number of OSS contributors from East Asia is much larger than from Africa. Hence a more normalized/frequent occurrence. And lastly if things start going south, some play on the r-card is always up the sleeve of PsyOps personnel.
A Chinese name is consistent with the timezone - alternatively Australia, Russia etc.
Here is my speculation.
I would look at the test framework - the attacker needs the complexity of it to hide away the attack and it is first thing he publishes - probably has been planned from the very beginning.
This is a homemade thing based on something called "Seatest".
How common is Seatest? Not that common as far as I can tell - maybe someone can correct me?
The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind
The thwarted XZ Utils supply chain attack was years in the making. Now, clues suggest nation-state hackers were behind the persona that inserted the malicious code.
...
At a glance, Jia Tan certainly looks East Asian—or is meant to. The time zone of Jia Tan’s commits are UTC+8: That’s China’s time zone, and only an hour off from North Korea’s.
Indeed. If it's nation state and years in the making then the idea of all the commit times being faked isn't a stretch.
Hell, even five-eyes (UK+AU+US+etc) could seek a linux backdoor and use Perth, Australia (GMT+8) timezones or have a submission bot in the UK that just commits at preset time from preset IP proxy.
Even if the real origin is identified, that news piece will have much less circulation that the initial suspicion that it's Chinese.
So there are also PR reasons to create a fake trail.
The best lie confirms your confirmation bias. Everybody heard about the Chinese hackers attacking US, Ugandan hackers however would raise a few eyebrows.
And from Microsoft / GitHub - who would have a lot additional information (logs, ip-adresses, use of two-factor auth etc.)? Have they made a statement?
Usually Microsoft etc. don't hold back identifying "threat actors".
Is there a US police investigation ongoing that could ask Microsoft? The target of this attack has been US firms / persons so if they report it, I assume a US police investigation would be required.
> And from Microsoft / GitHub - who would have a lot additional information (logs, ip-adresses, use of two-factor auth etc.)? Have they made a statement?
Based on a HN comment from a couple of weeks ago, by analyzing the attackers IP addresses from IRC chat logins, it seems they used a VPN service. If you think about it, it makes sense to always use VPN when doing an operation like this. So I think the ip addresses won't be of much use.
I have seen NordVPN’s response to a subpoena. Their response was that they had no records connecting an IP address at a specific date/time to any particular person.
it would be the end of their business if they did, as they have a strict no retention policy. This would mean they are lying to all their customers, so it is not going to happen.
> using a chinese front name is a very interesting detail
I have recently read that book https://www.ifitssmartitsvulnerable.com/ The author mentions a malware which at the first glance looked to originate from China (e.g. saved with Mandarin-localized MS Office) but was actually developed in Russia.
Is there anyone who have made complication on the most famous (searched by the whole world) anonymous people. No 1 must be Satoshi Nakamoto, and No 2 maybe this one? Is there any others?
Satoshi disappeared one day before Gavin went to the CIA for a Bitcoin presentation. He left the CAlert key and disappeared.
I learned about Bitcoin either days prior or days later. As a newbie I did not care who created Bitcoin, but now I regret not knowing and speaking with him directly.
I thought it is established by now Paul Le Roux is almost certainly Satoshi. He had the motivation (money laundering), the cryptography knowledge and him never moving his coins after they became quite valuable is easily explained by the fact he is in prison. As far as I am aware there is no one else for whom all three would be present.
Whoever Satoshi Nakamoto is, once he (she?) moves the coins they become the target of lots of criminal enterprises. Those bitcoins make them a VERY rich person and draws a very bright target on their back. Maybe they just prefer not having to deal with this?
I remember reading the article proposing this and I seem to remember some pretty big differences in coding style (TrueCrypt vs. Bitcoin) and assumed development platform. I'm not convinced tbh
While libcurl is an alluring target it is really ill suited for this, Daniel Stenberg is paid to work on it full time and his track record speaks. The situation is really different from xz.
On very old glibc ldd used to set an env variable and run the executable. The assumption being that PT_INTERP is set to glibcs dynamic linker. The dynamic linker would dump resolved libraries and stop execution if the variable was set. My guess is they wanted one and only one code path, whereas a separate resolver in `ldd` could get out of sync with the dynamic linker and therefore report incorrect results. But obviously binaries do not have to have an interpreter set at all, or they may point it to any other executable that's not glibcs dynamic linker, and the result is it would just execute like normal.
The fact that the lecturer is aware of it makes `sudo ldd` look even more odd.
Because it’s just another attempt by people who have emotional issues around systemd to derail a technical thread while adding nothing new or insightful. The people who actually work on these projects have been making efforts to reduce dependency depth and otherwise protect against a general class of attack which could have been used to target many other things, both users of liblzma and more broadly other popular projects with limited resources.
I hope that one positive development from this saga will be that open source users do not want to been as the next Jugar Kumar and choose to act less entitled and demanding towards maintainers.
I've never minded ignoring -anyone- who wasn't paying me if they're being disrespectful. I try my best generally, so I don't feel guilty, and if that's not good enough, they can go pound sand and cry about it; a good cry can definitely help you when you're at peak emotions is my thinking. A lot of people have other things going on and just snap at the smallest thing when they get frustrated.
We now know the answer to the question of "who would win?"
1) A years-long nation-state-backed hacking effort to infiltrate a software project and compromise most servers in the Western world
or
2) A German's obsession with efficiency and precision in engineering
Is that actually known?
It’s all based on the sophistication, the level of psychological manipulation, years of effort, multiple actors, and a working theory based on commit timings.
I suppose it could be a few amateurs who took on a passion project, but... it's unlikely.
It seems crazy to me that they spend so much time keeping this hidden and then never check for performance degradation.
They were rushing to get it out into the wild and thus made many mistakes, including the performance degradation.
From what I recall, some distros were reporting build errors in bleeding edge versions. Actors were essentially debugging issues in production, and recommending to distro maintainers ways to “fix” (really hide) the build issue with xz
Normally it would be much quicker but potentially could have been picked up downstream by other developers when the code became more mainstream.
He did an interview for Risky Business #743.
Whoever was trying to get this into the wild was really sloppy in execution of an otherwise “beautiful” attack.
Stuxnet effectively put a stopgap in Iranian’s nuclear program. This backdoor could barely make it out of the alpha phase of development.
> A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).
Since the part that takes longest is getting commit rights and there's a lot of waiting to do, what are the chances the attackers put all their eggs on infiltrating one specific project? I'd be trying to infiltrate at least 10 or 20 even if I were alone, to increase my chances. If you dedicate one or two years to build reputation with the current maintainers by doing a PR here and there, you can do that for a few projects in parallel.
* It is extremely critical. Used at multiple places in the boot process, with root or kernel mode
* Nevertheless an unnoticeable dependency. Nobody thougt about its security implications
* Stable and trustworthy for years (decades?)
* Managed mostly by 1 person
* That person was more than average vulnerable for pressure, but not in a very visible way.
For there to be another repository affected by the same attack you don't even need to think this specific attacker did it twice. You can just consider that multiple attackers had the same idea. I think it's way more likely that there's more repositories affected by the same type of attack than not.
I'd go as far as saying this is an issue of national / global security and there's space for an agency like the IMF but for software security that would be funded by all the countries of the world to bring more resources and assuredness to this type of dependencies.
Because even if this attacker didn't do more than N=1 and other attackers didn't have the idea before, they surely have it now.
See Tim Bray's proposal about "OSQI": https://www.tbray.org/ongoing/When/202x/2024/04/01/OSQI
* Has random binary test files in its repo, ideal for hiding exploit payloads.
"The OpenJS Foundation Cross Project Council received a suspicious series of emails with similar messages, bearing different names and overlapping GitHub-associated emails. These emails implored OpenJS to take action to update one of its popular JavaScript projects to “address any critical vulnerabilities,” yet cited no specifics" [0,1].
[0] https://openjsf.org/blog/openssf-openjs-alert-social-enginee... [1] https://www.schneier.com/blog/archives/2024/04/other-attempt...
It is kind of strange since their apps are supposed to be open source, maybe he meant the backend? Nonetheless, it seems this has been their modus operandi for a long time.
The thing is, as computers proliferate and we start to use them in more places, the effects of possible holes moves closer to our homes. From distant infra to near infra; from borderlines to our homes and transportation we use everyday. Even to our pockets via smartphones and other smart devices we host in our homes.
Thanks
Stephen Smalley in question works at NSA [3].
[0]: https://news.ycombinator.com/item?id=9863896
[1]: https://www.phoronix.com/news/NSA-KDBUS-Credentials
[2]: https://lkml.iu.edu/hypermail/linux/kernel/1507.1/01758.html
[3]: https://www.linkedin.com/in/stephen-smalley
All you need a Raspberry Pi and some public Wi-Fi network to create a jump point and hide among the 100s of devices going on and off that one public IP. With projects like TailScale you could set it up and plant it somewhere in a matter of hours
I thought most VPNs sold usage data to ad companies?
I also recently wrote a single ephemeral socks5 proxy over a hidden service in Rust. Since they’ve probably compromised other machines in the past, they could’ve easily used something similar to proxy their connection through tor and to some random computer (access some vulnerable router through tor, proxy through it, etc). Hiding your tracks aren’t hard
All we know is they were active during business hours of the UTC+1/3 timezones. They used a chinese sounding name as one of their front figures, the one who made most of the commits of malicious code. They wrote english messages that some people claim appear to be a Russian trying to write in English. But that last argument is pretty weak as they could have just masked their origin using Google translate, or even ChatGPT.
One of the biggest clues is actually one little commit that they let slip with a middle name to the chinese persona. And some people claim the names don't match up with different chinese cultures.
After that you can use all your psych 101 experience to try and analyze them but you'll just be guessing.
Personally I think using a chinese front name is a very interesting detail. They could have used any name in the world, there are open source contributors from all over the world.
Using a name in a country where most people have a very little insight is a good way of not being uncovered as fake of course. But why not use a Ugandan name? Also using a Chinese name can be perceived as pointing the blame at China, which might be sensitive geopolitically.
So these names seem somewhat similar to "Jia Tan" in their almost-but-not-quite real quality.
And why an Asian name? It's certainly exploiting a psychological bias. Apart from what you noted, the names are much more generic than Ugandan names and IMO it's virtually impossible to track a legend down. And the number of OSS contributors from East Asia is much larger than from Africa. Hence a more normalized/frequent occurrence. And lastly if things start going south, some play on the r-card is always up the sleeve of PsyOps personnel.
Here is my speculation.
I would look at the test framework - the attacker needs the complexity of it to hide away the attack and it is first thing he publishes - probably has been planned from the very beginning.
This is a homemade thing based on something called "Seatest".
How common is Seatest? Not that common as far as I can tell - maybe someone can correct me?
https://www.wired.com/story/jia-tan-xz-backdoor
The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind
The thwarted XZ Utils supply chain attack was years in the making. Now, clues suggest nation-state hackers were behind the persona that inserted the malicious code.
...
At a glance, Jia Tan certainly looks East Asian—or is meant to. The time zone of Jia Tan’s commits are UTC+8: That’s China’s time zone, and only an hour off from North Korea’s.
Indeed. If it's nation state and years in the making then the idea of all the commit times being faked isn't a stretch.
Hell, even five-eyes (UK+AU+US+etc) could seek a linux backdoor and use Perth, Australia (GMT+8) timezones or have a submission bot in the UK that just commits at preset time from preset IP proxy.
The time zone is consistent with Western Australia and the work starts by "branching out" a not widely used Australian test framework.
The actual attack is probably done by someone else.
Which Jia Tan turns into "s-test".
So there are also PR reasons to create a fake trail.
The best lie confirms your confirmation bias. Everybody heard about the Chinese hackers attacking US, Ugandan hackers however would raise a few eyebrows.
Usually Microsoft etc. don't hold back identifying "threat actors".
Is there a US police investigation ongoing that could ask Microsoft? The target of this attack has been US firms / persons so if they report it, I assume a US police investigation would be required.
Based on a HN comment from a couple of weeks ago, by analyzing the attackers IP addresses from IRC chat logins, it seems they used a VPN service. If you think about it, it makes sense to always use VPN when doing an operation like this. So I think the ip addresses won't be of much use.
I have recently read that book https://www.ifitssmartitsvulnerable.com/ The author mentions a malware which at the first glance looked to originate from China (e.g. saved with Mandarin-localized MS Office) but was actually developed in Russia.
I learned about Bitcoin either days prior or days later. As a newbie I did not care who created Bitcoin, but now I regret not knowing and speaking with him directly.
ldd has an unfortunate side effect of actually executing code within some libraries (the exact details, I don’t know. Maybe it’s library specific)
The fact that the lecturer is aware of it makes `sudo ldd` look even more odd.
https://archlinux.org/news/the-xz-package-has-been-backdoore...
https://youtu.be/Q6ovtLdSbEA?feature=shared&t=1491
I have just closed issues where the reporter or “user”, as you put it, is just unhelpful, rude, or non responsive.
No response needed most of the time.