Notes on the Bloomberg Supermicro supply chain hack story

Bloomberg has a story how Chinese intelligence inserted secret chips into servers bound for America. There are a couple issues with the story I wanted to address.


The story is based on anonymous sources, and not even good anonymous sources. An example is this attribution:

a person briefed on evidence gathered during the probe says

That means somebody not even involved, but somebody who heard a rumor. It also doesn't the person even had sufficient expertise to understand what they were being briefed about.

The technical detail that's missing from the story is that the supply chain is already messed up with fake chips rather than malicious chips. Reputable vendors spend a lot of time ensuring quality, reliability, tolerances, ability to withstand harsh environments, and so on. Even the simplest of chips can command a price premium when they are well made.

What happens is that other companies make clones that are cheaper and lower quality. They are just good enough to pass testing, but fail in the real world. They may not even be completely fake chips. They may be bad chips the original manufacturer discarded, or chips the night shift at the factory secretly ran through on the equipment -- but with less quality control.

The supply chain description in the Bloomberg story is accurate, except that in fails to discuss how these cheap, bad chips frequently replace the more expensive chips, with contract manufacturers or managers skimming off the profits. Replacement chips are real, but whether they are for malicious hacking or just theft is the sticking point.

For example, consider this listing for a USB-to-serial converter using the well-known FTDI chip. The word "genuine" is in the title, because fake FTDI chips are common within the supply chain. As you can see form the $11 price, the amount of money you can make with fake chips is low -- these contract manufacturers hope to make it up in volume.

The story implies that Apple is lying in its denials of malicious hacking, and deliberately avoids this other supply chain issue. It's perfectly reasonable for Apple to have rejected Supermicro servers because of bad chips that have nothing to do with hacking.

If there's hacking going on, it may not even be Chinese intelligence -- the manufacturing process is so lax that any intelligence agency could be responsible. Just because most manufacturing of server motherboards happen in China doesn't point the finger to Chinese intelligence as being the ones responsible.

Finally, I want to point out the sensationalism of the story. It spends much effort focusing on the invisible nature of small chips, as evidence that somebody is trying to hide something. That the chips are so small means nothing: except for the major chips, all the chips on a motherboard are small. It's hard to have large chips, except for the big things like the CPU and DRAM. Serial ROMs containing firmware are never going to be big, because they just don't hold that much information.

A fake serial ROM is the focus here not so much because that's the chip they found by accident, but that's the chip they'd look for. The chips contain the firmware for other hardware devices on the motherboard. Thus, instead of designing complex hardware to do malicious things, a hacker simply has to make simple changes to software, and replace the software.

Thus, if investigators are worried about hacking, they'll look at those chips first. When they find fake ones, because some manager tried to skim $0.25 per server that was manufactured, then they'll find evidence confirming their theory.

But if that were the case, investigators can simply pull the malicious software off the chip, reverse engineer it, and confirm its maliciousness. The Bloomberg story doesn't verify this happened. It's like a story of UFOs the rely upon the weight of many unconfirmed reports rather than citing a single confirmed one.

This story could be true, of course. And even if it's not true in this one case, there are probably other cases. The manufacturing process is so lax it's probable that somewhere some intelligence organization has done this. However, the quality of reporting is so low, quoting anonymous sources that appear not to have sufficient expertise, focusing on sensationalistic aspects, and not following up on background, that I have to question this story.

---

Update: Apple has a very direct refutation of Bloomberg's allegations.
https://www.apple.com/newsroom/2018/10/what-businessweek-got-wrong-about-apple/

Amazon likewise directly refutes the story:
https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/