Tuesday, October 16, 2018

Notes on the UK IoT cybersec "Code of Practice"

The British government has released a voluntary "Code of Practice" for securing IoT devices. I thought I'd write some notes on it.

First, the good parts

Before I criticize the individual points, I want to praise if for having a clue. So many of these sorts of things are written by the clueless, those who want to be involved in telling people what to do, but who don't really understand the problem.

The first part of the clue is restricting the scope. Consumer IoT is so vastly different from things like cars, medical devices, industrial control systems, or mobile phones that they should never really be talked about in the same guide.

The next part of the clue is understanding the players. It's not just the device that's a problem, but also the cloud and mobile app part that relates to the device. Though they do go too far and include the "retailer", which is a bit nonsensical.

Lastly, while I'm critical of most all the points on the list and how they are described, it's probably a complete list. There's not much missing, and the same time, it includes little that isn't necessary. In contrast, a lot of other IoT security guides lack important things, or take the "kitchen sink" approach and try to include everything conceivable.

1) No default passwords

Since the Mirai botnet of 2016 famously exploited default passwords, this has been at the top of everyone's list. It's the most prominent feature of the recent California IoT law. It's the major feature of federal proposals.

But this is only a superficial understanding of what really happened. The issue wasn't default passwords so much as Internet-exposed Telnet.

IoT devices are generally based on Linux which maintains operating-system passwords in the /etc/passwd file. However, devices almost never use that. Instead, the web-based management interface maintains its own password database. The underlying Linux system is vestigial like an appendix and not really used.

But these devices exposed Telnet, providing a path to this otherwise unused functionality. I bought several of the Mirai-vulnerable devices, and none of them used /etc/passwd for anything other than Telnet.

Another way default passwords get exposed in IoT devices is through debugging interfaces. Manufacturers configure the system one way for easy development, and then ship a separate "release" version. Sometimes they make a mistake and ship the development backdoors as well. Programmers often insert secret backdoor accounts into products for development purposes without realizing how easy it is for hackers to discover those passwords.

The point is that this focus on backdoor passwords is misunderstanding the problem. Device makers can easily believe they are compliant with this directive while still having backdoor passwords.

As for the web management interface, saying "no default passwords" is useless. Users have to be able to setup the device the first time, so there has to be some means to connect to the device without passwords initially. Device makers don't know how to do this without default passwords. Instead of mindless guidance of what not to do, a document needs to be written that explains how devices can do this both securely as well as easy enough for users to use.

Humorously, the footnotes in this section do reference external documents that might explain this, but they are the wrong documents, appropriate for things like website password policies, but inappropriate for IoT web interfaces. This again demonstrates how they have only a superficial understanding of the problem.

2) Implement a vulnerability disclosure policy

This is a clueful item, and it should be the #1 item on every list.

Though they do add garbage on top of this, but demanding companies respond in a "timely manner", but overall this isn't a bad section.

3) Keep software updated

This is another superficial understanding of the problem.

Software patching works for desktop and mobile phones because they have interfaces the user interacts with, ones that can both notify the user of a patch as well as the functionality to apply it. IoT devices are usually stuck in a closet somewhere without such interfaces.

Software patching works for normal computers because they sell for hundreds of dollars and thus have sufficient memory and storage to reliably do updates. IoT devices sell for cut-throat margins and have barely enough storage to run. This either precludes updates altogether, or at least means the update isn't reliable, that upon every update, a small percentage of customer devices will be "bricked", rendered unusable. Adding $1 for flash memory to a $30 device is not a reasonable solution to the problem.

Software patching works for software because of its enormous margins and longevity. A software product is basically all profit. The same doesn't apply to hardware, where devices are sold with slim margins. Device makers have a hard time selling them for more because there are always no-named makers of almost identical devices in Shenzen willing to undercut them. (Indeed, looking at Mirai, it appears that was the majority of infected devices, not major brands, but no-named knock-offs). 

The document says that device makers need to publish how long the device will be supported. This ignores the economics of this. Devices makers cannot know how long they will support a device. As long as they are selling new ones, they've got incentive and profits to keep supplying updates. After that, they don't. There's really no way for them to predict the long term market success of their devices.

Guarantees cost money. If they guarantee security fixes for 10 years, then that's a liability they have to account for on their balance sheet. It's a huge risk: if the product fails to sell lots of units, then they are on the hook for a large cost without the necessary income to match it.

Lastly, the entire thing is a canard. Users rarely update firmware for devices. Blaming vendors for not providing security patches/updates means nothing without blaming users for not applying them.

4) Securely store credentials and security-sensitive data

Like many guides, this section makes the superficial statement "Hard-coded credentials in device software are not acceptable". The reason this is silly is because public-keys are a "credential", and you indeed want "hard-coded" public-keys. Hard-coded public-key credentials is how you do other security functions, like encrypted and signature verification.

This section tells device makers to use the trusted-enclave features like those found on phones, but this is rather silly. For one thing, that's a feature of only high-end CPUs, not the low-end CPUs found in such devices. For another thing, IoT devices don't really contain anything that needs that level of protection.

Storing passwords in clear-text on the device is almost certain adequate security, and this section can be ignored.

5) Communicate securely

In other words, use SSL everywhere, such as on the web-based management interface.

But this is only a superficial understanding of how SSL works. You (generally) can't use SSL for devices because there's no secure certificate on the device. It forces users to bypass nasty warnings in the browser, which hurts the entire web ecosystem. Some IoT devices do indeed try to use SSL this way, and it's bad, very bad.

On the other hand, IoT devices can and should use SSL when connecting outbound to the cloud.

6) Minimise exposed attack surfaces

This is certainly a good suggestion, but it's a platitude rather than an action item. IoT devices already minimize as much as they can in order to reduce memory/storage requires. Where this is actionable requires subtler understanding. A lot of exposed attack services come from accidents. 

A lot of other exposed attack surfaces come about because device makers know no better way. Actual helpful, meaning advice would consist of telling them what to do in order to solve problems, rather than telling them what not to do.

The reason Mirai-devices exposed Telnet was for things like "remote factory reset". Mirai infected mostly security cameras which don't have factory reset buttons. That's because they are located high up out of reach, or if they are in reach, they don't want to allow the public to press the factory reset button. Thus, doing a factory reset meant doing it remotely. That appears to be the major reason for Telnet and "hardcoded passwords", to allow remote factory reset. Instead of telling them not to expose Telnet, you need a guide explaining how to securely do remote factory resets.

This guide discussed "ports", but the reality is that the attack surface in the web-based management interface on port 80 is usually more than all other ports put together. Focusing on "ports" reflects a superficial understanding of the problem.

7) Ensure software integrity

The guide says "Software on IoT devices should be verified using secure boot
mechanisms". No, they shouldn't be. In the name of security, they should do the opposite.

First of all, getting "secure boot" done right is extraordinarily difficult. Apple does it the best with their iPhone and still they get it wrong. For another thing, it's expensive. Like trusted enclaves in processors, most of the cheap low-end processors used in IoT don't support it.

But the biggest issue is that you don't want it. "Secure boot" means the only operating system the device can boot comes from the vendor, which will eventually stop supporting the product, making it impossible to fix any security problem. Not having secure boot means that customers can still be able to patch bugs without the manufacturer's help.

Instead of secure boot, device makers should do the opposite and make it easy for customers to build their own software. They are required to do so under the GNU Public License anyway. That doesn't mean open-sourcing everything, they can still provide their private code as binaries. But they should allow users to fix any bug in open-source and repackage a new firmware update.

8) Ensure that personal data is protected

I suppose giving the GDPR, this section is required, but GDPR is a pox on the Internet.

9) Make systems resilient to outages

Given the recent story of Yale locks locking people out of their houses due to a system outage, this seems like an obviously good idea.

But it should be noted that this is hard. Obviously such a lock should be resilient if the network connection is down, or their servers have crashed. But what happens when such a lock can contact their servers, but some other component within their organization has crashed, such that the servers give unexpected responses, neither completely down, but neither completely up and running, either?

We saw that in the Mirai attacks against Dyn. It left a lot servers up and running, but took down on some other component that those servers relied upon, leaving things in an intermediate state that was neither unfunctional nor completely functional.

It's easy to stand on a soapbox and proclaim devices need to be resilient, but this is unhelpful. What would instead be helpful is a catalog of failures that IoT will typically experience.

10) Monitor system telemetry data

Security telemetry is a desirable feature in general. When a hack happens, you want to review logfiles to see how it happened. This item reflects various efforts to come up with such useful information

But again we see something so devoid of technical details as to be useless. Worse, it's going to be exploited by others, such as McAffee wanting you to have anti-virus on TV sets, which is an extraordinarily bad idea.

11) Make it easy for consumers to delete personal data

This is kinda silly in that the it's simply a matter of doing a "factory reset". Having methods to delete personal details other than factory resets is bad.

The useful bit of advise is that factory resets don't always "wipe" information, they just "forget" it in a way that can be recovered. Thus, we get printers containing old documents and voting machines with old votes.

On the other hand, this is a guide for "consumer IoT", so just the normal factory reset is probably sufficient, even if private details can be gleaned.

12) Make installation and maintenance of devices easy

Of course things should be easy, everyone agrees on this. The problem is they don't know how. Companies like Microsoft and Apple spend billions on this problem and still haven't cracked it.

My home network WiFi password uses quotes as punctuation to improve security. The Amazon Echo app uses Bluetooth to pair with the device and set which password to use for WiFi. This is well done from a security point of view.

However, their app uses an input field that changes quotes to curly-quotes making it impossible to type in the password. I instead had to go to browser, type the password in the URL field, copy it, then go back to the Alexa app and paste it into the field. Then I could get things to work.

Amazon is better at making devices easy and secure with Echo and they still get things spectacularly wrong.

13) Validate input data

Most security vulnerabilities are due to improper validation of input data. However, "validate input data" is stupid advice. It's like how most phishing attacks come from strangers, but how telling people to not open emails from strangers is stupid advice. In both cases, it's a superficial answer that doesn't really understand how the problem came about.

Let's take PHP and session cookies, for example. A lot of programmers think the session identifier in PHP is some internal feature of PHP. They therefore trust it, because it isn't input. They don't perceive how it's not internal to PHP, but external, part of HTTP, and something totally hackable by hackers.

Or take the famous Jeep hacker where hackers were able to remotely take control of the car and do mischievous things like turn it off on the highway. The designers didn't understand how the private connection to the phone network was in fact "input" coming from the Internet. And then there was data from the car's internal network, which wasn't seen as "input" from an external source.

Then there is the question of what "validation" means. A lot of programmers try to solve SQL injection by "blacklisting" known bad characters. Hackers are adept at bypassing this, using other bad characters, especially using Unicode. Whitelisting known good characters is a better solution. But even that is still problematic. The proper solution to SQL injection isn't "input validation" at all, but using "parameterized queries" that don't care about input.

Conclusion

Like virtually every other guide, this one is based upon platitudes and only a superficial understanding of the problem. It's got more clue than most, but is still far from something that could actually be useful. The concept here is virtue signaling, declaring what would be virtuous and moral for an IoT device, rather than something that could be useful to device makers in practice.
















Sunday, October 14, 2018

How to irregular cyber warfare

Somebody (@thegrugq) pointed me to this article on "Lessons on Irregular Cyber Warfare", citing the masters like Sun Tzu, von Clausewitz, Mao, Che, and the usual characters. It tries to answer:
...as an insurgent, which is in a weaker power position vis-a-vis a stronger nation state; how does cyber warfare plays an integral part in the irregular cyber conflicts in the twenty-first century between nation-states and violent non-state actors or insurgencies
I thought I'd write a rebuttal.

None of these people provide any value. If you want to figure out cyber insurgency, then you want to focus on the technical "cyber" aspects, not "insurgency". I regularly read military articles about cyber written by those, like in the above article, which demonstrate little experience in cyber.

The chief technical lesson for the cyber insurgent is the Birthday Paradox. Let's say, hypothetically, you go to a party with 23 people total. What's the chance that any two people at the party have the same birthday? The answer is 50.7%. With a party of 75 people, the chance rises to 99.9% that two will have the same birthday.

The paradox is that your intuitive way of calculating the odds is wrong. You are thinking the odds are like those of somebody having the same birthday as yourself, which is in indeed roughly 23 out of 365. But we aren't talking about you vs. the remainder of the party, we are talking about any possible combination of two people. This dramatically changes how we do the math.

In cryptography, this is known as the "Birthday Attack". One crypto task is to uniquely fingerprint documents. Historically, the most popular way of doing his was with an algorithm known as "MD5" which produces 128-bit fingerprints. Given a document, with an MD5 fingerprint, it's impossible to create a second document with the same fingerprint. However, with MD5, it's possible to create two documents with the same fingerprint. In other words, we can't modify only one document to get a match, but we can keep modifying two documents until their fingerprints match. Like a room, finding somebody with your birthday is hard, finding any two people with the same birthday is easier.

The same principle works with insurgencies. Accomplishing one specific goal is hard, but accomplishing any goal is easy. Trying to do a narrowly defined task to disrupt the enemy is hard, but it's easy to support a group of motivated hackers and let them do any sort of disruption they can come up with.

The above article suggests a means of using cyber to disrupt a carrier attack group. This is an example of something hard, a narrowly defined attack that is unlikely to actually work in the real world.

Conversely, consider the attacks attributed to North Korea, like those against Sony or the Wannacry virus. These aren't the careful planning of a small state actor trying to accomplish specific goals. These are the actions of an actor that supports hacker groups, and lets them loose without a lot of oversight and direction. Wannacry in particular is an example of an undirected cyber attack. We know from our experience with network worms that its effects were impossible to predict. Somebody just stuck the newly discovered NSA EternalBlue payload into an existing virus framework and let it run to see what happens. As we worm experts know, nobody could have predicted the results of doing so, not even its creators.

Another example is the DNC election hacks. The reason we can attribute them to Russia is because it wasn't their narrow goal. Instead, by looking at things like their URL shortener, we can see that they flailed around broadly all over cyberspace. The DNC was just one of their few successes, among a large number of failures. We then watched their incompetent bungling of that opportunity, such as inadvertently leaving their identity behind in Word metadata.

In contrast to these broad, opportunistic hacking from Russia, China, North Korea, and Iran we have the narrow, focused hacking from the U.S. and its allies Britain and Israel. Stuxnet is really the only example we have of a narrow, focused attack being successful. The U.S. can succeed at such an improbable attack because of its enormous investment in the best cyber warriors in the world. But still, we struggle against our cyber adversaries because they are willing to do undirected, opportunistic hacking while we insist on doing narrow, well-defined hacking. Despite our skill, we can't overcome the compelling odds of the Birthday Attack.

What's interesting about the cyber guerillas we face is their comparative lack of skill. The DNC hackers were based primarily on things like phishing, which unsophisticated teenagers can do. They were nothing like the sophisticated code found in Stuxnet. Rather than a small number of talented cyberwarriors, they are more accurately using the infinite monkeys approach of banging away on keyboards until they come up with the works of Shakespear.

I don't know about the real policy makers and what they decide in secret, but in public, our politicians struggle to comprehend this paradox. They insist on seeing things like the DNC hack or Wannacry as the careful plans of our adversaries. This hinders our response to cyber insurgencies.

I'm a hacker and not a student of history, but I suspect those famous real-world insurgencies relied upon much the same odds, that their success is the same illusion as hacker successes. Sure, Che Guevara participated in the successful Cuban revolution, but was a failure in other revolutions in Africa and South America. Mao Zedong wasn't the leader of China's communist revolution so much as one of many leaders. He's just the one of many who ended up with all the marbles at the end.

It's been fashionable lately to quote Sun Tzu or von Clausewitz on cyberwar, but it's just pretentious nonsense. Cyber needs to be understand as something in its own terms, not as an extension of traditional warfare or revolution. We need to focus on the realities of asymmetric cyber attacks, like the nation states mentioned above, or the actions of Anonymous, or the successes of cybercriminals. The reason they are successful is because of the Birthday Paradox: they aren't trying to achieve specific, narrowly defined goals, but are are opportunistically exploiting any achievement that comes their way. This informs our own offensive efforts, which should be less centrally directed. This informs our defenses, which should anticipate attacks based not on their desired effect, but what our vulnerabilities make possible.

Thursday, October 04, 2018

Notes on the Bloomberg Supermicro supply chain hack story

Bloomberg has a story how Chinese intelligence inserted secret chips into servers bound for America. There are a couple issues with the story I wanted to address.

Friday, September 28, 2018

Mini pwning with GL-iNet AR150

Seven years ago, before the $35 Raspberry Pi, hackers used commercial WiFi routers for their projects. They'd replace the stock firmware with Linux. The $22 TP-Link WR703N was extremely popular for these projects, being half the price and half the size of the Raspberry Pi.

Monday, September 10, 2018

California's bad IoT law

California has passed an IoT security bill, awaiting the governor's signature/veto. It’s a typically bad bill based on a superficial understanding of cybersecurity/hacking that will do little improve security, while doing a lot to impose costs and harm innovation.

Wednesday, August 29, 2018

Debunking Trump's claim of Google's SOTU bias

Today, Trump posted this video proving Google promoted all of Obama "State of the Union" (SotU) speeches but none of his own. In this post, I debunk this claim. The short answer is this: it's not Google's fault but Trump's for not having a sophisticated social media team.


The evidence still exists at the Internet Archive (aka. "Wayback Machine") that archives copies of websites. That was probably how that Trump video was created, by using that website. We can indeed see that for Obama's SotU speeches, Google promoted them, such as this example of his January 12, 2016 speech:


And indeed, if we check for Trump's January 30, 2018 speech, there's no such promotion on Google's homepage:
But wait a minute, Google claims they did promote it, and there's even a screenshot on Reddit proving Google is telling the truth. Doesn't this disprove Trump?

No, it actually doesn't, at least not yet. It's comparing two different things. In the Obama example, Google promoted hours ahead of time that there was an upcoming event. In the Trump example, they didn't do that. Only once the event went live did they mention it.

I failed to notice this in my examples above because the Wayback Machine uses GMT timestamps. At 9pm EST when Trump gave his speech, it was 2am the next day in GMT. So picking the Wayback page from January 31st we do indeed see the promotion of the live event.


Thus, Trump still seems to have a point: Google promoted Obama's speech better. They promoted his speeches hours ahead of time, but Trump's only after they went live.

But hold on a moment, there's another layer to this whole thing. Let's look at those YouTube URLs. For the Obama speech, we have this URL:


For the Trump speech, we have this URL:


I show you the complete URLs to show you the difference. The first video is from the White House itself, whereas the second isn't (it's from the NBC livestream).

So here's the thing, and I can't stress this enough Google can't promote a link that doesn't exist. They can't say "Click Here" if there is no "here" there. Somebody has to create a link ahead of time. And that "somebody" isn't YouTube: they don't have cameras to create videos, they simply publish videos created by others.

So what happened here is simply that Obama had a savvy media that knew how to create YouTube live events, and make sure they get promoted, while Trump doesn't have such a team. Trump relied upon the media (which he hates so much) to show the video live, making no effort himself to do so. We can see this for ourselves: while the above link clearly shows the Obama White House having created his live video, the current White House channel has no such video for Trump.

So clearly the fault is Trump's, not Google's.

But wait, there's more to the saga. After Trump's speech, Google promoted the Democrat response:


Casually looking  back through the Obama years, I don't see any equivalent Republican response. Is this evidence of bias?

Maybe. Or again, maybe it's still the Democrats are more media savvy than the Republicans. Indeed, what came after Obama's speech on YouTube in some years was a question-and-answer session with Obama himself, which of course is vastly more desirable for YouTube (personal interaction!!) and is going to push any competing item into obscurity.

If Trump wants Google's attention next January, it's quite clear what he has to do. First, set up a live event the day before so that Google can link to it. Second, setup a second post-speech interactive question event that will, of course, smother the heck out of any Democrat response -- and probably crash YouTube in the process.

Buzzfeed quotes Google PR saying:
On January 30 2018, we highlighted the livestream of President Trump’s State of the Union on the google.com homepage. We have historically not promoted the first address to Congress by a new President, which is technically not a State of the Union address. As a result, we didn’t include a promotion on google.com for this address in either 2009 or 2017.
This is also bunk. It ignores the difference between promoting upcoming and live events. I can't see that they promoted any of Bush's speeches (like in 2008) or even Obama's first SotU in 2010, though it did promote a question/answer session with Obama after the 2010 speech. Thus, the 2017 trend has only a single data point.

My explanation is better: Obama had a media savvy team that reached out to them, whereas Trump didn't. But you see the problem for a PR flack: while they know they have no corporate policy to be biased against Trump, at the same time, they don't necessarily have an explanation, either. They can point to data, such as the live promotion page, but they can't necessarily explain why. An explanation like mine is harder for them to reach.










Sunday, August 26, 2018

Provisioning a headless Raspberry Pi

The typical way of installing a fresh Raspberry Pi is to attach power, keyboard, mouse, and an HDMI monitor. This is a pain, especially for the diminutive RPi Zero. This blogpost describes a number of options for doing headless setup. There are several options for this, including Ethernet, Ethernet gadget, WiFi, and serial connection. These examples use a Macbook as an example, maybe I'll get around to a blogpost describing this from Windows.