I thought this story was remarkable. The Chinese military has been placing teeny-tiny chips in the microchips China makes for the whole world that provide a backdoor into all kinds of confidential information on servers. Big companies like Apple and Amazon figured this out, and rather than making it public, have been quietly blacklisting major suppliers. But weirdly, everyone is denying it.
But that’s just what U.S. investigators found: The chips had been inserted during the manufacturing process, two officials say, by operatives from a unit of the People’s Liberation Army. In Supermicro, China’s spies appear to have found a perfect conduit for what U.S. officials now describe as the most significant supply chain attack known to have been carried out against American companies.
One official says investigators found that it eventually affected almost 30 companies, including a major bank, government contractors, and the world’s most valuable company, Apple Inc. Apple was an important Supermicro customer and had planned to order more than 30,000 of its servers in two years for a new global network of data centers. Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.
In emailed statements, Amazon (which announced its acquisition of Elemental in September 2015), Apple, and Supermicro disputed summaries of Bloomberg Businessweek’s reporting. “It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental,” Amazon wrote. “On this we can be very clear: Apple has never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server,” Apple wrote. “We remain unaware of any such investigation,” wrote a spokesman for Supermicro, Perry Hayes. The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.” The FBI and the Office of the Director of National Intelligence, representing the CIA and NSA, declined to comment.
But other sources say otherwise.
The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information.
The devices targeted were circuit boards in servers that do ubiquitous stuff, like compressing video so you can Netflix & chill, or doing language processing so Siri can figure out what you’re saying around a mouthful of Doritos. It’s all around us, and we take it for granted.
One country in particular has an advantage executing this kind of attack: China, which by some estimates makes 75 percent of the world’s mobile phones and 90 percent of its PCs.
See? This is what you get when you want all the slick new gadgets but you’re only willing to pay starvation wages to Chinese peons to get it done — all the fundamental work flees expensive America to cheap Asia. If we’d actually supported a semi-conductor industry in this country, just think…it could have been American spies bugging everyone’s computer.
It is – they’re just doing it in different layers.
I am surpised at how little I’m suprised at this. Indeed, my dominant reaction is: “Of course they did. Why weren’t you checking for that?”
Intel has also built backdoors into processors. And the NSA has a whole operation for adding bugs to laptops.
From here it looks as though the governments of the world are united in not wishing “their” citizens to be able to avoid ubiquitous surveillance. It’s the one thing they agree on.
The story is remarkable. But is it true?
Or is it an attempt to unjustly throw suspicion at China while the current administration is trying to get the upper hand on them in trade?
The chips have networking capability? What does that even mean? Network access would have to go through the ethernet ports. The chip pictured appears to have only a single digit number of electrical contacts. How then is it going to intercept and modify traffic between the CPU and memory as described?
Curious how no person, or even their agency, backing the claim is named.
Yellowcake.
I recall a story a few years back about the US govt fitting tiny spy chips to export boards, though at this point I have no idea if that was a Russian plant story or this is a similar plant story by US agencies.
The fact that everyone involved denies what the security agency is saying, might just suggest that it’s not exactly true.
The chips inject a tiny bit of code into the software installed on them that then accesses a larger reservoir of nefarious code elsewhere. So they don’t have networking capability, they just bootstrap parasitic code that exploits the networking capability of the operating system.
Also, why would major companies deny this? Because, as mentioned, China is the world’s biggest supplier of these circuit boards, and admitting that they were compromised would a) undermine confidence in the gadgets they sell, and b) force them to find other suppliers, which would probably be more expensive, and cut into profits. Or, short answer, capitalism.
I’m not sure that we should be blaming consumers for needing affordable electronics for the low wages in other countries. In the modern world, most people can’t get along to do the daily business of life without a cell phone and/or a computer. The blame lies with the US companies that don’t want to pay US workers a reasonable salary. These companies outsource because they know the workers in those countries will accept less. In the business world it’s more important for stock holders (and CEOs) to keep making obscene profits that increase every quarter. These companies then make the same argument – “It’s not us! It’s those greedy consumers that make just enough to pay their bills every month that demand “cheap” whatever it is!” – investors need to realize that they are gamblers and that gambling does not guarantee anyone a win, let alone continuing wins at ever increasing volumes.
Please stop.
The quote that the chips have networking capability is taken directly from the Bloomberg article you are using as your primary source.
Where are you proposing this “larger reservoir of nefarious code” is situated? Is it elsewhere on the motherboard, which means another item has been overlooked; or is it out on the network, in which case you need the networking capability to reach it?
Relying on the operating system would mean you would need to know which version of which operating system.
Read further. Read the section titled “how it worked”.
Reginald Selkirk:
The chips have networking capability? What does that even mean? Network access would have to go through the ethernet ports.
Cellular protocols.
Very few organizations do a wireless spectrum analysis. Those that do – are shocked and horrified. I assume that these interfaces, like Intel’s backdoors or the backdoor in the Via processor, have some kind of unlock event that turns on the full stack. That keeps them hidden for a good long time – these things are only used in carefully targeted environments.
There are other vectors into systems that would be pretty good: GPUs and network interfaces. Both of those have bus mastery and can do DMA into wherever they want. A GPU is a very complicated device and there is so much room to hide – and the drivers run in kernel mode. A network interface is the obvious place. I have seen demos of an android phone being rooted remotely via a buffer overrun in a smart antenna controller from across a room: boom – a shell changes to uid=0 magically.
The US, with its huge infrastructure of “stingrays” could easily inject such attacks and I assume they do. I have a friend who used to do spectrum audits around hosting facilities, airports, and tech offices (including Lawrence Livermore) – he found hundreds of network devices that were offering connectivity. Presumably the US does not allow the GRU to install those. The worst he found was a home-built stingray running RH Linux, close to a catholic girls’ high school. I consider this “highly credible” – not tinfoil hat brigade stuff. I used to own a cryptophone myself and watching the logs from that was a real eye-opener.
@8
“because we refuse to pay them enough to afford our things at a higher prices their pay would require.”
that is the thinking and management “skills” that earn them multi-million dollar compensation packages.
as for the “chips” it seems trivially easy to do these kinds of things to get the results of easy government access to computer controlled devices. It is not just chips that can used to supply “back-doors” software is also available for hacking and modification.
the story reveals yet again how difficult it is to keep things secret. The bigger the secret the harder it is to keep it hidden.
uncle frogy
Reginald Selkirk @9:
Not really, if you parse the article’s words carefully. Why would you need networking capability if you can indirectly communicate with the networking hardware elsewhere on the motherboard?
That depends. The chip is located near the southbridge, if Bloomberg’s illustration is accurate, so in theory it could hop on a bus and send off its own network packets. PCI-e devices can directly access system memory without accessing the CPU, so you could also carve out some storage for a malicious payload. A clever design could defer the OS-specific portions to the payload, and get that payload running by searching for then tweaking an execution jump.
Having said that, server boards would usually be running one of a handful of Linux kernels from Red Hat or Canonical. Juggling a few hundred possibilities, all very similar to one other, isn’t very hard.
Reply to #11: You are addressing the wrong question. Yes, there are possible ways of hacking hardware. But you are expanding beyond the specifics of the case. We are told that this one small chip is the compromise. Suggesting that a GPU (on a server?) could be hacked does not address that. Bringing in the network interface? That was not part of the accusation.
That’s nice. But an android phone has a cellular antenna. Are you proposing that this one small chip includes a cellular antenna? Or that there is a separate cellular antenna on the motherboard that is used by the chip? And from across the room? OK. But you would need to know which room.
Let’s see…
Environmental destruction on a horrific scale? CHECK!
Corporate control of government at all levels? CHECK!
Obscene wealth disparity? CHECK!
Political and cultural balkanization? CHECK!
Near ubiquitous access to a world wide computer network? CHECK!
Massive amounts of computer crime. CHECK!
A glib libertarian economic and cultural ethos (or rather, lack of an ethos)? CHECK!
Terrible street violence? CHECK!
Yup, “cyberpunk” is pretty much how I’d describe the present. Only know it’s Russian oligarchs pulling the world’s strings rather than Japanese Zaibatsu. All we need are street mercenaries who deliberately replace their limbs and sensory organs with prosthetics and we’re set!
Edit: Only now…
Akira, @ #15: This is not the sci-fi dystopia I ordered! I wanted the cool one, with the designer body mods and the really good drugs. This one sucks.
Reginald Selkirk:
Did you see where I said “cellular protocols” or did you maybe overlook that?
This is not a formal debate. The other information I was commenting on was stuff I consider interesting and related – I didn’t want to just make a two word comment.
There’s networking capability in the BIOS.
Reginald Selkirk @14:
Intel has been bundling GPUs into their CPUs for years. Why would they stop doing that for a server chip, given that it costs tonnes of money to build a new processor line? GPUs are also a hot commodity in computing clusters, due to their efficiency.
You’re sending the signal metres instead of kilometres, so why not? Cheap off-the-shelf WiFi chips are larger than what Bloomberg shows, but not by much. But I agree it’s highly unlikely they’d use WiFi or cellular; every server board has some sort of Ethernet controller (the one in Bloomberg’s illustration is in the upper-right), which works just as well but over much further distances.
The notion that outsourcing of manufacture is the reason China specifically is responsible for the manufacture of microchips is slightly misdirected. There are plenty of places in the world where wages are lower, but China has the peculiar combination of developing economy wages and high tech education. There’s also the fact that Chinese companies have a de-facto monopoly on the entire global reserve of rare earths, meaning that nobody else could acquire the necessary materials even if they could man the factory. How much of this was planned by the government of China, and the direction of cause and effect, are difficult questions to answer.
Hj Hornbeck:
I have been personally involved in incidents in which there were cellular protocol transmitters embedded in systems in manufacturing facilities hosted in China. I’m not just talking theory here.
Ian King makes a good point. This stuff does not accurately track national origin. The NSA once “enhanced” a Swedish cipher machine (Hag m209) with backdoor’d crypto because they knew Hagelin was going to sell them to Iran. #awkward.
I get blood-squirtingly angry when American dumbasses go “Chineeeeese company, probably backdoored!” As if the Chinese would only backdoor Chinese companies? What, do Americans think the Chinese are stupid? It’s incredibly parochial.
Example: Congress gets all bent out of shape about Huawei. OMG Huawei is a Chineeeeeeeeese company! Not so fast: anyone who has worked in computing for more than 20 years ought to know that Huawei is a joint venture between 3COM and the Chinese government! OMG Hauwei is Amerrrrrrrrrrucan!!
This is a very complex and subtle problem and there are a lot of people with opinions and a small number of people with facts.
Dunc:
The MINIX instance in the Intel Management Engine (which is in most chips made by Intel since 200?8) is a full blown multitasking operating system with its own IP stack, that exists unknown to the rest of the CPU. AND it can talk to the onboard NIC, which has a nice standard interface because the Intel mobo chipset includes the phy. Thanks Intel! Thanks NSA!
Reginald: clearly you’ve never heard of JTAG. For those of the “TLDR” school, here’s the money quote from Wikipedia:
And as just one data point for connectivity, I’m guessing you also weren’t aware that every modern PC server designed for use in a data center already has a second computer, running its own (usually very insecure, in my experience) OS, sharing the primary Ethernet jack. You think it would be that hard to add a third? I’ll leave it as an exercise for the reader to guess how many pins you need to talk a serial protocol like Ethernet.
The technical implementation of this attack isn’t trivial. But it’s extremely plausible, especially on systems where you already have another computer controlling the main computer to the degree these do.
(The sum total of the non-remote setup needed to install an OS to a bare drive on these servers when they’re fresh from the manufacturer is to ask someone to plug in power and Ethernet cables. The rest, starting from turning on the computer, is done over the network, usually by someone who’s never been and never will be within hundreds or thousands of kilometers of the hardware.)
Robert Asprin predicted this in 1977, well before “cyberpunk” was even a thing. The Cold Cash War.
Favorite quote from the article:
Yes, if you’re building the motherboard you generally you know the exact version of (in SuperMicro’s case) Linux, BusyBox, etc. that’s installed on the baseboard management controller because it’s installed at manufacturing time. If you have control of this you’ve hacked the system, and have working network connectivity with a full network stack, even if there’s no OS installed on the main computer and the power switch is off.
Why is Bloomberg telling us about this? Seems like they shouldn’t be leaking classified information without good reason. I mean, as an honest question for the experts around here.
Marcus Ranum @ 24:
Unknown to the CPU? Hell, it was unknown to Andy Tannenbaum, the creator of MINIX, who only heard Intel was using it in their Management Engine when security researchers started finding flaws in it!
To expand on 27, the article says that the chip is connected to the baseboard management controller, and that’s the route that they use to get network access.
My guess is that it modifies the baseboard management controller firmware while its being loaded from SPI or I2C EEPROM, which is plausible attack for the low pin count part shown.
Marcus Ranum @22:
Sorry if I was implying otherwise. In fact, I should point out I’m working more on the theoretical end, as my CompSci training involved little security work. Still, you don’t have to read many articles on bridging air gaps to get a rough idea of what’s viable.
Siggy @28:
Politics, of course. The Trump White House doesn’t want to admit they got help from the Kremlin, but can’t deny they got help from a foreign power. Solution: talk more about Chinese hacking and shift the blame. It helps that the Chinese government is genuinely engaged in high-tech espionage.
HJ @31: While I don’t doubt that the White House is trying to play up China, the article sounds more like a Bloomberg analyst spotted a weird shift in major company buying patterns and then started pulling on the thread.
Siggy:
Why is Bloomberg telling us about this? Seems like they shouldn’t be leaking classified information without good reason. I mean, as an honest question for the experts around here.
Back in 1997, a sales rep working for a firewall vendor (Raptor Systems) floated a rumor that there was a Mossad backdoor in the market leader, Israeli-made Checkpoint Firewall-1. Pretty soon there were ultra-nationalists in the DOD saying that the US Government should not buy Checkpoint. And, that’s how it happened. Raptor made good inroads into the federal IT market for a while as a result.
Similarly, antivirus product company Kaspersky has been tagged with “ooh, they are Russian!” and it has definitely hurt their market share in the US. Ditto Huawei (as I mentioned above)
Sometimes these leaks are a form of marketing. You concoct a story and get the press to carry it for you. It’s just politics by another name. The reality of the situation is that an experienced intelligence officer would get a software engineer in any manufacturer that was notable, and you’d have weird things like the Intel Management Engine crop up on product plans as a result.
Well, you have them. Only they’re not physical devices. They have funny names like “Microsoft Windows”, “Google”, “DMCA”, “Amazon”, “Apple” etc.
In the 1950’s Australia was one of the world leaders in computing technology with a fledgling industry that need some government support and protection. The conservative free-market ideologues that still dominate its government appointed a British judge to head an enquiry to decide if this should happen. Naturally the judge put his loyalty to Britain first and recommended that Australia didn’t need to support its fledgling computer industry because it could always buy its computers from Britain. The meddling by China is a just one example of why this idea was stupid. For this and other acts of treason the pro-British PM of the day received two knighthoods and a lordship.
angela78 @34:
Nah, that’s rather inelegant. If you own the network, you effectively own the data.
garydargan@35–
Agree with your criticism of Australia’s history of stupid political decisions on science and industry, but I doubt it would have made much difference. The UK invented the universal computer but never made much effort to make itself a centre of excellence after WW2. The US invented integrated circuits and personal computing but then handed over their manufacturing to China. Even with the industries where Australia had good infrastructure, we’ve tended to move manufacturing offshore just because it’s cheaper. There’s no reason to think we’d have handled computing any differently.
@ 4 Reginald Selkirk
Yellowcake.
Not proven but did you notice the Americans are having an election soon?
Mike Pence is alleging massive Chinese interference in that election (besides this)? He had proof: … an example of this interference a four-page advertisement that the state-owned China Daily newspaper placed in the Des Moines Register, criticizing Trump’s trade policies. My god!
Sergei Skripal and his daughter, Yulia, were poisoned just before the World Cup.
The Syrian army (allegedly) launched a gas attack on Ghouta just before the Syrian army was to begin a major campaign, essentially a final one, against jihadists who were, more or less Western clients
And now the Russians are hacking WADA (who say they have no evidence of this), and the Organization for the Prevention of Chemical Weapons who might just be an embarrassment to the USA re the (alleged?) chemical attacks in Syria.
Curioser and curioser.
And the US and other Western countries will never have to prove these accusations.
@19 for the win. BIOS based malware, executing even before the preexecution environment is loaded. Toss and the additional core or two inside of the onboard GPU or even any other decent sized die chip, have a very real single board computer equivalent operating at Xeon clock speeds or higher, without the OS being allowed to know of its existence, stealing a few cycles on the bus during idle to report on what it’s found via DMA access at the hardware level.
I remember a certain HP jetdirect card equipped computer being intercepted before being sent to Iraq, just before a war… And a sudden failure of a network, just before the attack.
And shortly after, having to redesign the network to have such devices on a vlan that can’t see the production network or internet.
All, reported by Joint Taskforce, Global Network Operations, the predecessor of Cybercom.
I think Marcus has it right: that horse bolted long ago.
—
OP:
Global supply chains. Distributed expertise. Existing infrastructure.
(https://www.vox.com/technology/2018/9/13/17851052/apple-iphone-price-china-trump-us-trade)
China may not be the lowest cost option, but yes, they are the lowest cost that has the high-tech knowledge, as others have mentioned.
Chris that is exactly the problem. Too many politicians completely lacking in vision for the country with an eye to the corporate balance sheet and not the country’s economic development and infrastructure security. They are traitors in my book.
@36 Hj Hornbeck
Owning the network helps for sure, but is not necessary -you still have Echelon and such. Also, you can still use end-to-end encryption to at least make difficult to peek at your data over the network.
But you can do nothing if your closed-source OS is the one who steals your data.
Or if the free cloud service which you use because it’s so cool and provides you so much features actually is an NSA funnel.
Nowadays there is actually no need to place hidden chips to steal information. People willingly gives away his data for free.
That’s not quite true – we did make a fairly significant effort in that direction, but it was hamstrung by bureaucratic incompetence, short-sightedness, classism, and sexism. Marie Hicks’ book “Programmed Inequality – How Britain Discarded Women Technologists and Lost Its Edge in Computing” is an excellent examination of the multi-faceted problems affecting the British computer industry with a particular focus on gender, which I think many people here would find interesting.
@#43, angela78
Back in the Obama administration, we already found out that the open-source OSes have also been hacked by the NSA. They’ve made greater strides since then. You’re just as compromised with Linux as you are with Windows or Mac OS X, don’t kid yourself that open-source helps even the slightest bit with this problem.
I’m sure others use the same tech but for different reasons. Just pissed China thought of it first. Anyway, now The Base can blame the Chinese if Trump loses.
@ 25 Marcus
What, do Americans think the Chinese are stupid?
Yes.
It’s incredibly parochial.
Yes. Or were those rhetorical questions?
As a Canadian, I notice these things. I have seen Americans on the Internet assure me that we are still ruled by Britain and that the British are responible for the policing in Dublin. At second hand, I heard the interesting story of the American woman who complained that Bejing looked run-down and insisted that its owners in Europe should fix it up.
Perhaps the crowning touch was the couple from Albany NY I ran into in a small town about 100km from Ottawa and who had visit the area every summer for years. The woman (M. Ed) and her husband (Elec. Eng., IIRC) seemed surprised to learn that it was the capital of Canada.
@45 The Vicar
nobody said that opensource can’t be stuffed with spyware. The huge difference is, when you find this out the whole community can (and will) go through the entire codebase to look for vulnerabilities and fix them.
With closed-source you can’t do that.
This is one of the reason for Linux’s inherent robustness and safety compared to Windows.
Except that it’s not always easy to look through the entire codebase, because the entire codebase might not be in the source code:
Except that Unix is not Linux, and it’s not opensource.
GCC instead is opensource.
This does’nt mean that such an attack is impossible, only that it’s much more difficult and, what is more important, that it can always be detected and removed.
@angela78: You do not understand the attack.
The attack described in the paper assumes that the target has full access to the source and can review and recompile it at will. While Unix wasn’t open source to the world, within the environment where the attack was demonstrated all the targets had the same access to the compiler binaries and sources as you have to GCC binaries and sources; effectively the situation was equivalent.
I suggest you re-read (or read, if you haven’t already) the paper carefully and the consider exactly how you would detect this attack if it were done on GCC on your personal Linux system.
@Curt Sampson: I do understand the attack described. What you do not understand is that you can actually complie everything, from the compiler onwards, on a fully opensource system.
Obviously you could have a low level (hardware) compromised system which alters your code, but this is really hard to do (and you don’t build your compiler on the same machine…)
@angela78:
Given that I’ve been doing (open source) operating system development since the 90’s, yes I understand very well how to compile everything. I’ve done it many, many times.
Recompiling your compiler doesn’t help; this is explicitly addressed in the paper. You really need to read it, because it’s very clear you do not understand the attack at all.
Curt, I don’t quite follow you.
Do you mean you’re already using a compiler to compile the compiler source, and the compiler you’re using is already compromised?
Because, it seems to me, if you have the source code for the compiler in front of you, you should be able to follow what it does if you know the language. And presumably a compiler’s source code is not supposed to be obfuscated, rather the opposite.
@John Morales: Yes, that’s exactly what I mean. That’s why I said “the entire codebase might not be in the source code” and quoted this from Wikipedia:
It’s all described in further detail on the Wikipedia page I linked and Thompson’s paper and other sources linked from there.
Curt, in that case, the compiler source code you are compiling is not the compiler source code of the compiler you’re that’s compromised.
When you wrote about having the source for the compiler and still not being able to tell, I thought you were talking about the source code for the compromised compiler. If one had that, the compromising code would be rather more evident.
Correct. You have the compromised compiler only as a binary.
No I was not. You have only the source for the non-compromised compiler. Nonetheless, using the compiler you have, building a new compiler from that non-compromised source code will produce a compromised compiler.
The references I linked seemed pretty clear to me, but maybe that’s just because I’m familiar with the attack. (Or maybe people aren’t bothering to read them before claiming it’s not not possible to attack open source software with this.)
What I mean is you are talking about two different compilers; one (C1, compromised) is in machine code, and the other (C2, not compromised) is in source code.
So, when you wrote “the entire codebase might not be in the source code”, you meant that the codebase for C1 might not be in C2. Sure.
—
(Or maybe people aren’t bothering to read them before claiming it’s not not possible to attack open source software with this.)
But the proposed attack relies on one’s compiler already being already compromised!
So, the claim amounts to this: if one’s compiler is compromised, compiling clean source will result in another compromised compiler. If it isn’t, then to get a compromised executable one would need to compile the compromised source code. Right?
(Huh)
—
PS, I garbled that first line in my previous, meant to delete “you’re using” as unneeded, but missed the first. :)
Curt, you miss the point.
Thompson’s attach assumes I use a binary, compromised version of a compiler. I’m telling you that this is by no means a mandatory step, and that you can easily build a clean compiler provided you have the code, time, skills and reasons to do so.
There are so many other ways to compromise a system: third part libraries, for example are the weakest point in modern Linux systems, because they are useful -therefore are often installed and used- but not opensource.
I’m not sure what you mean by this; it sounds kind of like saying, “an attack where someone uses your password relies on them having your password.” Certainly the attack relies on someone downloading and using a compromised compiler, but when was the last time anybody you know bootstrapped the compiler they used “from scratch,” without using a previously built version of that compiler? (I used to do this from time to time for the Glasgow Haskell compiler, but I still had to hope that the GCC I was using for that didn’t have a compromise designed to compromise GHC.)
Correct. But how do you know the compiler you’re using, C₀, isn’t compromised, thus compromising anything it builds? You can’t try to go forward out of that because even building from source that you’ve inspected and ascertained is uncompromised you’ll still end up with another compromised compiler.
What you need to know is both that your current compiler was built from uncompromised source and that the compiler that built your compiler, C₋₁, was not compromised. But in turn, that depends not only on the source used to build C₋₁ having been uncompromised but also that the compiler C₋₂ which built it was uncompromised. And so on, continuing back.
Verifying that “chain of custody” back as far as you need to go is, well, non-trivial. (I invite you to try it for the current compiler shipping with Red Hat or Debian systems.)
@angela78: And what compiler do you use to build that clean compiler? Feel free to go into as much detail as necessary as to how you would get a guaranteed clean GCC.
Curt,
A proper analogy would be “an attack to get your password which relies on them having your password.”
Anyway, I’ll drop the derail after this, but to me an “attack” means an attempt to compromise, not the brute fact of being compromised.
Curt: I hope you are not asking me “how do you compile a C compiler without a C compiler?”. You are not bound to using a C compiler, you know…
angela78: Let me dash your hopes. Sure, let’s start with, “How do you build GCC without using a C compiler?” Get as technical as you like, and assume you’re explaining to someone who’s done plenty of compiler and OS builds.
For bonus points, show how often your method is actually used, rather than just building GCC with a previously built version of GCC.
Curt, still missing the point, and still asking me to “get technical” without need.
Building a C compiler from scratch is not something you do everyday, but is something that you can do -at least, something that I can do, even if it’d take me way too much time. But the point is not if you have to do it anytime, it’s that if you want to do it you can, and that a whole community of people working on a public piece of code is much safer and much more difficult to compromise than closed-source sw.
I can think hundreds of ways to attack both opensource and closed sw, most of them more realistic and easy to implement than your scenario. If you think that opensource means 100%safe than you don’t understand computer science, but if you think that closed source is not waaay more dangerous than os than…the same.
I have no idea where you’re coming up with the idea that I might think “opensource means 100%safe” when I am pointing out an attack that works on open source software. Nor have I ever even hinted at a claim that, overall, open source is not more resistant to attacks in general than closed source.
But I’m not discussing attacks in general here; I’m discussing a particular attack.
That is absolutely true and I am in complete agreement with you there. It’s also, as far as this attack goes, entirely irrelevant.
In the context of Thompson’s original attack implementation, the source code was indeed “open” within the community of Unix source licencees. Those licensees had the source code to the OS and compiler and could and did build it themselves. Today’s licensees of Linux and GCC are no different.
In fact they’re both in the exact same boat. If they carefully examine every line of the source for the compiler they’re about to build and verify that the compromise is not present in that source, they still end up with a compromised compiler after compiling that uncompromised source with a compromised compiler. It’s not an attack on the source; it’s an attack on the way the compilers are built.
Nor is this attack invalidated because you consider it difficult and can think of other, easier attacks. The same is true of the motherboard attack described in this blog post; surely you aren’t saying that because it was such a difficult attack we can ignore it.
I go back to my original question: how do you know that the GCC on your system right now isn’t compromised in this way, and what have you done to ensure this? (The answer is, I’m pretty sure, you don’t and are simply trusting that whoever gave you your compiler binary hasn’t given you a compromised one.)
Hmm, I wonder if you can get a clean compiler by filtering through several cross-compilers for different processor architectures, with object-level inspection.
The Thompson compiler would necessarily have to have the code to be injected in one of two forms: either a small binary add-on or source (presumably obfuscated in some way).
If object code, then the first time you cross-compile to another processor architecture the injected code fails since it won’t be compatible. Then you either have a clean compiler or one that crashes. Either way, attack is exposed.
If obfuscated source, the sanitizing is somewhat harder. Cross-compile with debugging symbols, then inspect the object code looking for a chunk of data that is not plausibly corresponding to known source code. As a matter of fact, the cross-compile step wouldn’t be needed, though it would filter the binary-blob method. Of course the sanitizing needs to be done by someone sufficiently familiar with how the compiler works and the object code for that particular platform.
So what did I miss? Other than the probability of someone funding that kind of effort of course.
es
p.s. Of course your source-object inspection step could be compromised if whatever you’re using as an editor or decompiler is similarly compromised. I think you can minimize the chances by writing your own tool to group debugging symbols with object code, presumably a compromised compiler would be unlikely to figure out your new code enough to hide the injected blob (especially one written with the twisted wackiness I’d come up with). And there are so many editors around it’d be unlikely they could all be compromised the same way.
BTW, would it help avoid the problem if you use brand A’s compiler to compile brand B’s source code, and vice-versa? Presumably a Thompson attack wouldn’t be flexible enough to successfully inject into a different design? I don’t know how inherently different compiler brands are…
es
p.p.s. Of course you’d have to go through all this with your dependent libraries also, especially wherever your printf() is coming from.
eternalstudent wrote:
In that particular example, perhaps. But what do you do if your burned-in bootstrap code is what’s compromised? Modern CPUs and SoCs are so complicated, it would not necessarily be difficult to use the various features to embed a backdoor that waits for a trigger to activate. And there are so many places to hide such a thing: from EFI drivers to peripheral firmware, there are countless points where code can be injected into your system, often without you having any way to examine it.
Like others, I remain somewhat skeptical of the claims, but not about the ability to create such a technology. It wasn’t all that long ago that some in the United States government freaked out about Canadian quarters being secret spy devices because the Royal Canadian Mint found a way to get ink to stick to metal coins and a poppy design was printed on them, so unwarranted paranoia isn’t exactly uncommon.
@eternalstudent writes:
Now there’s a thought! However, the attack, at least in theory, should be able to avoid being discovered this way since it should have completed its work by the time of generation of the intermediate representation of the compiled code that’s generic to all architectures.
And bingo, as written right in the section of the Wikipedia page I originally linked:
It’s not perfect but this method significantly increases the difficulty of an attack. Then again, it also significantly increases the difficulty of your build process. (There’s a reason that David A. Wheeler was awarded a Ph.D. for the work above.) As just one example of the many issues involved, you may not even be able to verify the compiler shipped with your OS; it’s the first problem Wheeler ran into when trying to verify GCC (thesis §3.7.1). So if you’re sitting there in front of a Fedora system, even if it’s the exact same version he used, his work doesn’t mean that your GCC isn’t backdoored.
By the way, the Countering Misconceptions section of Wheeler’s page on trusting trust is well worth reading. He also goes on after that to talk about attacks on hardware (the topic of this blog post) and further attacks in the wild.
And for those who don’t want to get into the heavy going of the serious analyses of build chain attacks, here are two somewhat lighter bits of reading that should at least give you a better sense of the problem.
_ 1. Mick Stute has a What is a coder’s worst nightmare story about a compromised compiler. (I have no idea if it’s true, but it’s plausible.) At the end that they fell into the same error as some others have here: they had the vendor bring in clean, uncompromised compiler source and tried to rebuild it, but ended up with a compromised compiler anyway. (Though they relatively quickly figured out the problem and copied the compiler binaries from an uncompromised system.)
_ 2. It’s not the exact attack we’ve been talking about, but XcodeGhost is an attack on the Xcode build chain that ended up putting thousands of malicious Apple iPhone apps in the App Store. The creators of the apps had no idea that they were unintentionally creating malware.
Looking at the Trusting Trust attack more closely, I’m becoming more and more convinced that this attack is actually quite as practical as, and probably similar in cost to, the hardware attack described in this blog post. Ironically, open source makes this easier in certain ways because your targets, especially if they’re aware enough to build their own code but not aware enough to be able to really, really harden their procedures for doing so, may have build processes that are considerably easier to attack than those of a company like Microsoft.
For those still interested, Patrick Kennedy on ServeTheHome has an article that offers reasons why this particular chip-on-mobo attack event may be a hoax. You can skip down to the section “The Counterpoint Published Outside of the Main Story” if you don’t want to be bored by the technical re-iteration of things already covered in the article and this comment thread.
Note that, while what they’re saying makes sense, it does not in any way invalidate the idea of the chip-on-mobo attack, which they consider practical.