Friday, August 15, 2014

Grow up, you babies

When I came home crying to my mommy because somebody at school called me "grahamcracker", my mother told me to just say "sticks and stones may break my bones but names will never hurt me". This frustrated me as a kid, because I wanted my mommy to make it stop, but of course, it's good advice. It was the only good advice back then, and it's the only solution now to stop Internet trolls.

In its quest to ban free speech, this NYTimes article can't even get the definition of the word "troll" right. Here's the correct definition:
"somebody who tries to provoke an emotional reaction"
The way to stop trolls is to grow up and stop giving them that emotional reaction. That's going to be difficult, because we have a nation of whiners and babies who don't want to grow up, who instead want the nanny-state to stop mean people from saying mean things. This leads to a police-state, where the powerful exploit anti-trolling laws to crack down on free-speech.

That NYTimes article claims that trolling leads to incivility. The opposite is true. Incivility doesn't come from me calling you a jerk. Instead, incivility comes from your inability to ignore it. It's your emotional response that is the problem, and your desire to sic the police-state on me to make me stop.

Let's work together and make our society more civil, and get people to stop responding to trolls. Let's tell the whining babies to grow the fuck up, and just repeat "sticks and stones may break my bones but names will never hurt me".

Saturday, August 02, 2014

C10M: The coming DDR4 revolution

Computer memory has been based on the same DRAM technology since the 1970s. Recent developments have been versions of the DDR technology, DDR2, DDR2, and now DDR4. The capacity and transfer speed have been doubling every couple years according to Moore's Law, but the latency has been stuck at ~70 nanoseconds for decades. The recent DDR4 standard won't fix this latency, but will give us a lot more tools to mitigate its effects.

Latency is bad. If a thread needs data from main memory, it must stop and wait for around 1000 instructions before the data is returned from memory. CPU caches mitigate most of this latency by keeping a copy of frequently used data in local, high-speed memory. This allows the processor to continue at full speed without having to wait.

The problem with Internet scale is that it can't be cached. If you have 10 million concurrent connections, each requiring 10-kilobytes of data, you'll need 100-gigabytes of memory. However, processors have only 20-megabytes of cache -- 50 thousand times too small to cache everything. That means whenever a packet arrives, the memory associated with that packet will not be in cache. The CPU will have to stop and wait while the data is retrieved from memory.

There are some ways around this. Specialty network processors solve this by having 8 threads per CPU core (whereas Intel has only 2 or even 1 thread per core). At any point in time, 7 threads can be blocked waiting for data to arrive from main memory, while the 8th thread continues at full speed with data from the cache.

On Intel processors, we have only 2 threads per core. Instead, our primary strategy for solving this problem is prefetching: telling the CPU to read memory into the cache that we'll need in the future.

For these strategies to work, however, the CPU needs to be able to read memory in parallel. To understand this, we need to look into details about how DRAM works.

As you know, DRAM consists of a bunch of capacitors arranged in large arrays. To read memory, you first select a row, and then rid each bit a column at a time. The problem is that it takes a long time to open the row before a read can take place. Also, before reading another row, the current row much be closed, which takes even more time. Most of memory latency is the time that it takes to close the current row and open the next row we want to read.

In order to allow parallel memory access, a chip will split the memory arrays into multiple banks, currently 4 banks. This now allows memory requests in parallel. The CPU issues a command to memory to open a row on bank #1. While it's waiting for the results, it can also issue a command to open a different row on bank #3.

Thus, with 4 banks, and random memory accesses, we can often have 4 memory requests happening in parallel at any point in time. The actual reads must happen sequentially, but most of the time, we'll be reading from one bank while waiting for another bank to open a row.

There is another way to increase parallel access, using multiple sets or ranks of chips. You'll often see that in DIMMs, where sometimes only one side is populated with chips (one rank), but other times both sides are populated (two ranks). In high density server memory, they'll double the size of the DIMMs, putting two ranks on each side.

There is yet another way to increase parallel access, using multiple channels. These are completely separate subsystems: not only can there be multiple commands outstanding to open a row on a given bank/rank, they can be streaming data from the chips simultaneously too. Thus, adding channels adds both to the maximum throughput as well as to the number of outstanding transactions.

A typical low-end system will have two channels, two ranks, and four banks giving a total of eight requests outstanding at any point in time.

Given a single thread, that means a C10M program with a custom TCP/IP stack can do creative things with prefetch. It can pull eight packets at a time from the incoming queue, hash them all, then do a prefetch on each one's TCP connection data. It can then process each packet as normal, being assured that all the data is now going to be in the cache instead of waiting on memory.

The problem here is that low-end desktop processors have four-cores with two-threads each, or eight threads total. Since the memory only allows eight concurrent transactions, we have a budget of only a single outstanding transaction per core. Prefetching will still help a little here, because it parallel access only works when they are on different channels/ranks/banks. The more outstanding requests, the more the CPU can choose from to work in parallel.

Now, here's where DDR4 comes into play: it dramatically increases the number of outstanding requests. It increases the number of banks from the standard 4 to 16. It also increases ranks from 4 to 8. By itself, this is an 8 fold increase in outstanding commands.

But it goes even further. A hot new technology is stacked chips. You see that in devices like the Raspberry Pi, where the 512-megabyte DDR3 DRAM chip is stacked right on top of the ARM CPU, looking from the outside world as a single chip.

For DDR4, designers plan on up to eight stacked DRAM chips. They've added chip select bits to select which chip in the stack is being accessed. Thus, this gives us a 256-fold theoretical increase in the number of outstanding transactions.

Intel has announced their Haswell-E processors with 8 hyperthreaded cores (16 threads total). This chip has 4 channels of DDR4 memory. Even a low-end configuration with only 32-gigs of RAM will still give you 16 banks times 2 ranks times 4 channels, or 128 outstanding transactions for 16 threads, or 8 outstanding transactions per thread.

But that's only with unstacked, normal memory. Vendors are talking about stacked packages that will increase this even further -- though it may take a couple years for these to come down in price.

This means that whereas in the past, prefetch has made little difference to code that was already limited by the number of outstanding memory transactions, it can make a big difference in future code with DDR4 memory.


This post is about getting Internet scale out of desktop hardware. An important limitation for current systems is the number of outstanding memory transactions possible with existing DRAM technology. New DDR4 memory will dramatically increase the number of outstanding transactions. This means that techniques like prefetch, which had limited utility in the past, may become much more useful in the future.

That Apache 0day was troll

Last week, many people saw what they thought was an Apache 0day. They say logs with lots of suggestive strings that looked like this:

[28/Jul/2014:20:04:07 +0000] “GET /?x0a/x04/x0a/x02/x06/x08/x09/cDDOSSdns-STAGE2;; HTTP/1.0″ 301 178 “-” “chroot-apach0day-HIDDEN BINDSHELL-ESTAB” “-”
Somebody has come forward and taken credit for this, admitting it was troll.

This is sort of a personality test. Many of us immediately assumed this was a troll, but that's because we are apt to disbelieve any hype. Others saw this as some new attack, but that's because they are apt to see attacks out of innocuous traffic. If your organization panicked at this "0day attack", which I'm sure some did, then you failed this personality test.

I don't know what tool the troll used, but I assume it was masscan, because that'd be the easiest way to do it. To do this with masscan, get a Debian/Ubuntu VPS and do the following:

apt-get install libpcap-dev dos2unix
git clone
cd masscan
echo "GET /my0dayexploit.php?a=\x0acat+/etc/password HTTP/1.0" >header.txt
echo "Referer:" >>header.txt
echo "" >>header.txt
unix2dos header.txt
iptables -A INPUT -p tcp --destination-port 4321 -j DROP

bin/masscan -p80 --banners --hello-file header.txt --source-port 4321 --rate 1500000

Depending on the rate your VPS can transmit, you'll cover the entire Internet in one to ten hours.

The response output from servers will be printed to the screen. You probably don't want that, so you should add the "-oX troll.xml" to save the responses to an XML file.

The above example uses "echo" to append lines of text to a file since HTTP is conveniently a text-based protocol. Its uses "unix2dos" to convert the line-feeds into the cr-lf combination that HTTP wants.

Masscan has it's own TCP/IP stack. Thus, on Linux, it can't establish a TCP connection, because when it tries, the existing TCP stacks sees something wrong and sends a RST to kill the connection. One way to prevent this is to configure a firewall rule to tell the built-in Linux TCP/IP stack to ignore the port that masscan uses. Another way is to tell masscan to use a --source-ip that isn't assigned to any existing machine on the network.

The rates at which you can transmit vary widely by hosting provider. In theory, you should get a rate of 1.5-million packets/second, and that's easily obtained in a lab on slow machines. Yet, in real hosting environments, things slow down, and I haven't been able to figure out why. In my experience, 300,000 is more of what you'd expect to get.

Thursday, July 31, 2014

No, the CIA didn't spy on other computers

The computer's the CIA spied on were owned and operated by the CIA.

I thought I'd mention this detail that is usually missing from today's news about the CIA spying on Senate staffers. The Senate staffers were investigating the CIA's torture program, reviewing classified documents. The CIA didn't trust the staffers, so they setup a special computer network just for the staffers to use -- a network secured and run by the CIA itself.

The CIA, though, spied on what the staffers did on the system. This allowed the CIA to manipulate investigation. When the staffers found some particularly juicy bit of information, the CIA was able to yank it from the system and re-classify it so that the staffers couldn't use it. Before the final report was ready, the CIA was already able to set the political machine in motion to defend itself from the report.

Thus, what the CIA did was clearly corrupt and wrong. It's just that it isn't what most people understand when they read today's headlines. It wasn't a case of the CIA hacking into other people's computers.

Many stories quote CIA director Brennan who said earlier this year:
I think a lot of people who are claiming that there has been this tremendous sort of spying and monitoring and hacking will be proved wrong
Many stories (like this one) claim that it's Brennan who was proven wrong, but instead, he was proven right. The investigation showed that at no time did the CIA hack anybody else's computer.

In pointing out the truth many people assume that I'm defending the CIA. I'm not. The torture program was morally wrong and beneath us as a country. Surreptitiously spying on the investigators into the program is clearly corrupt, and all involved need to be fired -- even if it turns out no law was broken (since it was the CIA's own computers).

I'm outraged, but believe we should be outraged by the right things, not the distorted story in the news. Seriously, I can't be more outraged at how the CIA revoked the declassification of things the staffers found useful to their investigation. It's not the spying (of their own computer) that angers me so much as their corrupt actions the spying enabled.

Update: @grayrisk points to this Lawfare blogpost with specifics. As you can see, CIA sysadmins had access to the system to administer it, but otherwise the system was supposed to be segregated from the rest of the CIA.

Monday, July 28, 2014

Cliché: open-source is secure

Some in cybersec keep claiming that open-source is inherently more secure or trustworthy than closed-source. This is demonstrably false.

Firstly, there is the problem of usability. Unusable crypto isn't a valid option for most users. Most would rather just not communicate at all, or risk going to jail, rather than deal with the typical dependency hell of trying to get open-source to compile. Moreover, open-source apps are notoriously user-hostile, which is why the Linux desktop still hasn't made headway against Windows or Macintosh. The reason is that developers blame users for being stupid for not appreciating how easy their apps are, whereas Microsoft and Apple spend $billions in usability studies actually listening to users. Desktops like Ubuntu are pretty good -- but only when they exactly copy Windows/Macintosh. Ubuntu still doesn't invest in the usability studies that Microsoft/Apple do.

The second problem is deterministic builds. If I want to install an app on my iPhone or Android, the only usable way is through their app stores. This means downloading the binary, not the source. Without deterministic builds, there is no way to verify the downloaded binary matches the public source. The binary may, in fact, be compiled from different source containing a backdoor. This means a malicious company (or an FBI NSL letter) can backdoor open-source binaries as easily as closed-source binaries.

The third problem is code-review. People trust open-source because they can see for themselves if it has any bugs. Or, if not themselves, they have faith that others are looking at the code ("many eyes makes bugs shallow"). Yet, this rarely happens. We repeatedly see bugs giving backdoor access ('vulns') that remain undetected in open-source projects for years, such as the OpenSSL Heartbleed bug. The simple fact is that people aren't looking at open-source. Those qualified to review code would rather be writing their own code. The opposite is true for closed-source, where they pay people to review code. While engineers won't review code for fame/glory, they will for money. Given two products, one open and the other closed, it's impossible to guess which has had more "eyes" looking at the source -- in many case, it's the closed-source that has been better reviewed.

What's funny about this open-source bigotry is that it leads to very bad solutions. A lot of people I know use the libpurple open-source library and the server (run by CCC hacking club). People have reviewed the libpurple source and have found it extremely buggy, and chat apps don't pin SSL certificates, meaning any SSL encryption to the CCC server can easily be intercepted. In other words, the open-source alternative is known to be incredibly insecure, yet people still use it, because "everyone knows" that open-source is more secure than closed-source.

Wickr and SilentCircle are two secure messaging/phone apps that I use, for the simple fact that they work both on Android and iPhone, and both are easy to use. I've read their crypto algorithms, so I have some assurance that they are doing things right. SilentCircle has open-sourced part of their code, which looks horrible, so it's probable they have some 0day lurking in there somewhere, but it's really no worse than equivalent code. I do know that both companies have spent considerable resources on code review, so I know at least as many "eyes" have reviewed their code as open-source. Even if they showed me their source, I'm not going to read it all -- I've got more important things to do, like write my own source.

Thus, I see no benefit to open-source in this case. Except for Cryptocat, all the open-source messaging apps I've used have been buggy and hard to use. But, you can easily change my mind: just demonstrate an open-source app where more eyes have reviewed the code, or a project that has deterministic builds, or a project that is easier to use, or some other measurable benefit.

Of course, I write this as if the argument was about the benefits of open-source. We all know this doesn't matter. As the EFF teaches us, it's not about benefits, but which is ideologically pure; that open-source is inherently more ethical than closed-source.

Wednesday, July 23, 2014

Everything can be a bomb

This last week, pranksters replaced the US flag on top the Brooklyn Bridge with a white-flag. Nobody knows who or why. Many in the press have linked this to terrorism, pointing out that it could've been a bomb. Not only local New York newspapers have said this, but also CNN.

Such irrational fears demonstrate how deeply we've fallen for police-state fears, where every action is perceived as a potential terrorist threat.

It could've been a bomb, of course. But what could also have been a bomb is a van full of C4 explosives driven across the bridge. There are no checkpoints at either end inspecting vehicles with bomb sniffing dogs. What also could've been a bomb is a ship full of fertilizer that, when ignited, would act as a small nuke. The point is that everything can be a bomb. Instead of using this as justification for an ever increasing police-state, we just need to accept this and live with the danger -- because this danger is, in the end, tiny. A thousand 9/11 events would still not equal cancer, for example.

I mention this because the former 9/11 commission released a new report yesterday stoking the fears of cyber-terrorism, calling for an increase in the cyber-police-state. This is nonsense. If government wants to fix cybersecurity, their first effort should focus on fixing their own computers rather than violating our rights. Enable SSL on all government computers, disable SQL string pasting, and get rid of all default/backdoor passwords.

Government is trying to ratchet up the fear of cyber-terror. It's not that their scenarios aren't possible, it's that they use this fear to drive the police-state. In exactly the same way we shouldn't have bomb sniffing dogs checking every car before it crosses the Brooklyn Bridge, we should not have the current proposals for cybersecurity that violate rights.

Tuesday, July 22, 2014

Um, talks are frequently canceled at hacker cons

Talks are frequently canceled at hacker conventions. It's the norm. I had to cancel once because, on the flight into Vegas, a part fell off the plane forcing an emergency landing. Last weekend, I filled in at HopeX with a talk, replacing somebody else who had to cancel.

I point this out because of this stories like this one hyping the canceled Tor talk at BlackHat. It's titled says the talk was "Suddenly Canceled". The adverb "suddenly" is clearly an attempt to hype the story, since there is no way to slowly cancel a talk.

The researchers are academics at Carnegie-Mellon University (CMU). There are good reasons why CMU might have to cancel the talk. The leading theory is that it might violate prohibitions against experiments on unwilling human subjects. There also may be violations of wiretap laws. In other words, the most plausible reasons why CMU might cancel the talk have nothing to do with trying to suppress research.

Suppressing research, because somebody powerful doesn't want it to be published, is the only reason cancelations are important. It's why the Boston MTA talk was canceled, because they didn't want it revealed how to hack transit cards. It's why the Michael Lynn talk was (almost) canceled, because Cisco didn't want things revealed.  It's why I (almost) had a talk canceled, because TippingPoint convinced the FBI to come by my offices to threaten me (I gave the talk because I don't take threats well). These are all newsworthy things.

The reporting on the Tor cancelation talk, however, is just hype, trying to imply something nefarious when there is no evidence.

Monday, July 21, 2014

More fun with #TSA

That's Julian in the center waving at me to stop taking pictures.
That's Michael faced away on his right
Coming back through JFK, my bag was stopped in the x-ray. The examiner shouted "bag checked", and sat and waited. And waited. Nobody came. Finally, he shunted it aside to the special bag check area. Where it sat, and sat.

There was as TSA agent standing around doing nothing, except flirting with a cute passenger standing right next to me bag. Finally, I pointed out that my bag needed to be checked, at which point he talked to the x-ray examiner, pulled it out, and checked it (I had a spray can of foot powder I bought because omg I wore my workout shoes that stink to the convention).

So, of course, I asked to see his badge, which was turned away from me, and to talk to his manager. He refused to even tell me his name, but he did get the supervisor, who confirmed his name was "Michael Vails". The manager was quite rude, looking at me in disbelief as I pointed out the guy was standing around flirting with girls instead of checking my bag. He wouldn't let me see his badge either, but claimed his name was "Julian something". I forget the something because I'm not good with names and forgot it by the time I was able to jot down notes.

So, I stood out of the way of traffic and started taking pictures. At this point, Julian came up to me and threatened me with arrest. I pointed out I'd read the TSA rules, which say it's legal (here). He said he knew the rules too, and that it wasn't illegal.

Unfortunately, I couldn't press the point, because I was at 1% battery, and had an electronic checking, needing a charged phone to get on the plane. Otherwise, I woulda popped out live video then and there. So I sat down and charged for a while.

I went back shortly after 3pm. I say a bunch of non-TSA non-police "security" people. I asked them about it, and they said that the actual police guy was just around the corner -- the only guy authorized to arrest me and carry a gun.

I was "streaming" to UStream at this point ( -- or so I thought. You guys missed the conversation. I was very polite, saying sir, ma'am (as appropriate), please, thank you, and "have a nice day".

I found the cop, and talked to him. He confirmed that he'd read the TSA guidelines and knew it was legal. He further confirmed that when they called for an arrest, it'd be him, and no, he wouldn't arrest me. I shook his hand and left.

I headed back to the TSA to take UStream video, and after checking it, realized I hadn't been streaming. So I took new video, this time getting it right, but it's quite boring. Unfortunately, the 3pm shift change happened, and there were new TSA agents, and the new ones didn't care that I was filming.

Anyway, that's today's adventures going through TSA. By the way, always remember when pissing off the TSA: be polite, calm, nice, and make sure you aren't standing in the way impeding traffic.

This was an awesome tweet somebody sent me:

The case involved a guy going through security charged with all sorts of things. It shows how the police can charge me with things no matter what I do. Luckily, the jury found this guy innocent, but there's a good chance an otherwise identical case might find somebody guilty.

Update: Some people on twitter asked "what's the goal" of this, or "what I'm trying to point out". The answer is "violation of rights". We have the right to hold the police accountable. When they threaten us to stop taking pictures, if we are afraid to take pictures, then we live in a police-state. Whipping out your cell phone and filming the TSA is something all passengers should do every time the TSA displeases them for any reason -- as long as they are doing so in a non-disruptive manner out of the way of traffic.

Also, remember to always have phone charged before going through TSA :).