Monday, August 25, 2014

Masscan does STARTTLS

Just a quick note: I've updated my port-scanner masscan to support STARTTLS, including Heartbleed checks. Thus, if you scan:

masscan -p0-65535 --banners --heartbleed

...then it'll find not only all vulnerable SSL servers, but also vulnerable SMTP/POP3/IMAP4/FTP servers using STARTTLS.

The issue is that there are two ways unencrypted protocols can support SSL. One is to assign a new port number (like 443 instead of 80), establish the SSL connection first, then the normal protocol second within the encrypted tunnel. The second way is the method SMTP uses: it starts the normal unencrypted SMTP session, then issues the "STARTTLS" command to convert the connection to SSL, then continue with SMTP encrypted.

Here's what a scan will look like:

Banner on port 143/tcp on [ssl] cipher:0x39 ,  
Banner on port 143/tcp on [vuln] SSL[heartbeat] SSL[HEARTBLEED] 

Because of the --banners option, we see the normal IMAP4 banners. Because the banner advertises STARTTLS, masscan will attempt to execute that feature. The SSL banner shows the name of the system "", and the vulnerability banner shows that it has heartbeats enabled, and that the software is vulnerable to Heartbleed.

I suggest you run this on all your outward facing sites on all ports -p0-65535 to find lots of Heartbleed vulnerable services that your normal vulnerability scanner might've missed.

I've probably introduced a bug doing this, so please update your code and try this out, and notify me of any bugs.

Note: if you want to also grab the full certificate with the SSL connection, use the option --capture cert to dump the BASE64 X.509 certificates as part of the scan.

Friday, August 15, 2014

Grow up, you babies

When I came home crying to my mommy because somebody at school called me "grahamcracker", my mother told me to just say "sticks and stones may break my bones but names will never hurt me". This frustrated me as a kid, because I wanted my mommy to make it stop, but of course, it's good advice. It was the only good advice back then, and it's the only solution now to stop Internet trolls.

In its quest to ban free speech, this NYTimes article can't even get the definition of the word "troll" right. Here's the correct definition:
"somebody who tries to provoke an emotional reaction"
The way to stop trolls is to grow up and stop giving them that emotional reaction. That's going to be difficult, because we have a nation of whiners and babies who don't want to grow up, who instead want the nanny-state to stop mean people from saying mean things. This leads to a police-state, where the powerful exploit anti-trolling laws to crack down on free-speech.

That NYTimes article claims that trolling leads to incivility. The opposite is true. Incivility doesn't come from me calling you a jerk. Instead, incivility comes from your inability to ignore it. It's your emotional response that is the problem, and your desire to sic the police-state on me to make me stop.

Let's work together and make our society more civil, and get people to stop responding to trolls. Let's tell the whining babies to grow the fuck up, and just repeat "sticks and stones may break my bones but names will never hurt me".

Saturday, August 02, 2014

C10M: The coming DDR4 revolution

Computer memory has been based on the same DRAM technology since the 1970s. Recent developments have been versions of the DDR technology, DDR2, DDR2, and now DDR4. The capacity and transfer speed have been doubling every couple years according to Moore's Law, but the latency has been stuck at ~70 nanoseconds for decades. The recent DDR4 standard won't fix this latency, but will give us a lot more tools to mitigate its effects.

Latency is bad. If a thread needs data from main memory, it must stop and wait for around 1000 instructions before the data is returned from memory. CPU caches mitigate most of this latency by keeping a copy of frequently used data in local, high-speed memory. This allows the processor to continue at full speed without having to wait.

The problem with Internet scale is that it can't be cached. If you have 10 million concurrent connections, each requiring 10-kilobytes of data, you'll need 100-gigabytes of memory. However, processors have only 20-megabytes of cache -- 50 thousand times too small to cache everything. That means whenever a packet arrives, the memory associated with that packet will not be in cache. The CPU will have to stop and wait while the data is retrieved from memory.

There are some ways around this. Specialty network processors solve this by having 8 threads per CPU core (whereas Intel has only 2 or even 1 thread per core). At any point in time, 7 threads can be blocked waiting for data to arrive from main memory, while the 8th thread continues at full speed with data from the cache.

On Intel processors, we have only 2 threads per core. Instead, our primary strategy for solving this problem is prefetching: telling the CPU to read memory into the cache that we'll need in the future.

For these strategies to work, however, the CPU needs to be able to read memory in parallel. To understand this, we need to look into details about how DRAM works.

As you know, DRAM consists of a bunch of capacitors arranged in large arrays. To read memory, you first select a row, and then rid each bit a column at a time. The problem is that it takes a long time to open the row before a read can take place. Also, before reading another row, the current row much be closed, which takes even more time. Most of memory latency is the time that it takes to close the current row and open the next row we want to read.

In order to allow parallel memory access, a chip will split the memory arrays into multiple banks, currently 4 banks. This now allows memory requests in parallel. The CPU issues a command to memory to open a row on bank #1. While it's waiting for the results, it can also issue a command to open a different row on bank #3.

Thus, with 4 banks, and random memory accesses, we can often have 4 memory requests happening in parallel at any point in time. The actual reads must happen sequentially, but most of the time, we'll be reading from one bank while waiting for another bank to open a row.

There is another way to increase parallel access, using multiple sets or ranks of chips. You'll often see that in DIMMs, where sometimes only one side is populated with chips (one rank), but other times both sides are populated (two ranks). In high density server memory, they'll double the size of the DIMMs, putting two ranks on each side.

There is yet another way to increase parallel access, using multiple channels. These are completely separate subsystems: not only can there be multiple commands outstanding to open a row on a given bank/rank, they can be streaming data from the chips simultaneously too. Thus, adding channels adds both to the maximum throughput as well as to the number of outstanding transactions.

A typical low-end system will have two channels, two ranks, and four banks giving a total of eight requests outstanding at any point in time.

Given a single thread, that means a C10M program with a custom TCP/IP stack can do creative things with prefetch. It can pull eight packets at a time from the incoming queue, hash them all, then do a prefetch on each one's TCP connection data. It can then process each packet as normal, being assured that all the data is now going to be in the cache instead of waiting on memory.

The problem here is that low-end desktop processors have four-cores with two-threads each, or eight threads total. Since the memory only allows eight concurrent transactions, we have a budget of only a single outstanding transaction per core. Prefetching will still help a little here, because it parallel access only works when they are on different channels/ranks/banks. The more outstanding requests, the more the CPU can choose from to work in parallel.

Now, here's where DDR4 comes into play: it dramatically increases the number of outstanding requests. It increases the number of banks from the standard 4 to 16. It also increases ranks from 4 to 8. By itself, this is an 8 fold increase in outstanding commands.

But it goes even further. A hot new technology is stacked chips. You see that in devices like the Raspberry Pi, where the 512-megabyte DDR3 DRAM chip is stacked right on top of the ARM CPU, looking from the outside world as a single chip.

For DDR4, designers plan on up to eight stacked DRAM chips. They've added chip select bits to select which chip in the stack is being accessed. Thus, this gives us a 256-fold theoretical increase in the number of outstanding transactions.

Intel has announced their Haswell-E processors with 8 hyperthreaded cores (16 threads total). This chip has 4 channels of DDR4 memory. Even a low-end configuration with only 32-gigs of RAM will still give you 16 banks times 2 ranks times 4 channels, or 128 outstanding transactions for 16 threads, or 8 outstanding transactions per thread.

But that's only with unstacked, normal memory. Vendors are talking about stacked packages that will increase this even further -- though it may take a couple years for these to come down in price.

This means that whereas in the past, prefetch has made little difference to code that was already limited by the number of outstanding memory transactions, it can make a big difference in future code with DDR4 memory.


This post is about getting Internet scale out of desktop hardware. An important limitation for current systems is the number of outstanding memory transactions possible with existing DRAM technology. New DDR4 memory will dramatically increase the number of outstanding transactions. This means that techniques like prefetch, which had limited utility in the past, may become much more useful in the future.

That Apache 0day was troll

Last week, many people saw what they thought was an Apache 0day. They say logs with lots of suggestive strings that looked like this:

[28/Jul/2014:20:04:07 +0000] “GET /?x0a/x04/x0a/x02/x06/x08/x09/cDDOSSdns-STAGE2;; HTTP/1.0″ 301 178 “-” “chroot-apach0day-HIDDEN BINDSHELL-ESTAB” “-”
Somebody has come forward and taken credit for this, admitting it was troll.

This is sort of a personality test. Many of us immediately assumed this was a troll, but that's because we are apt to disbelieve any hype. Others saw this as some new attack, but that's because they are apt to see attacks out of innocuous traffic. If your organization panicked at this "0day attack", which I'm sure some did, then you failed this personality test.

I don't know what tool the troll used, but I assume it was masscan, because that'd be the easiest way to do it. To do this with masscan, get a Debian/Ubuntu VPS and do the following:

apt-get install libpcap-dev dos2unix
git clone
cd masscan
echo "GET /my0dayexploit.php?a=\x0acat+/etc/password HTTP/1.0" >header.txt
echo "Referer:" >>header.txt
echo "" >>header.txt
unix2dos header.txt
iptables -A INPUT -p tcp --destination-port 4321 -j DROP

bin/masscan -p80 --banners --hello-file header.txt --source-port 4321 --rate 1500000

Depending on the rate your VPS can transmit, you'll cover the entire Internet in one to ten hours.

The response output from servers will be printed to the screen. You probably don't want that, so you should add the "-oX troll.xml" to save the responses to an XML file.

The above example uses "echo" to append lines of text to a file since HTTP is conveniently a text-based protocol. Its uses "unix2dos" to convert the line-feeds into the cr-lf combination that HTTP wants.

Masscan has it's own TCP/IP stack. Thus, on Linux, it can't establish a TCP connection, because when it tries, the existing TCP stacks sees something wrong and sends a RST to kill the connection. One way to prevent this is to configure a firewall rule to tell the built-in Linux TCP/IP stack to ignore the port that masscan uses. Another way is to tell masscan to use a --source-ip that isn't assigned to any existing machine on the network.

The rates at which you can transmit vary widely by hosting provider. In theory, you should get a rate of 1.5-million packets/second, and that's easily obtained in a lab on slow machines. Yet, in real hosting environments, things slow down, and I haven't been able to figure out why. In my experience, 300,000 is more of what you'd expect to get.

Thursday, July 31, 2014

No, the CIA didn't spy on other computers

The computer's the CIA spied on were owned and operated by the CIA.

I thought I'd mention this detail that is usually missing from today's news about the CIA spying on Senate staffers. The Senate staffers were investigating the CIA's torture program, reviewing classified documents. The CIA didn't trust the staffers, so they setup a special computer network just for the staffers to use -- a network secured and run by the CIA itself.

The CIA, though, spied on what the staffers did on the system. This allowed the CIA to manipulate investigation. When the staffers found some particularly juicy bit of information, the CIA was able to yank it from the system and re-classify it so that the staffers couldn't use it. Before the final report was ready, the CIA was already able to set the political machine in motion to defend itself from the report.

Thus, what the CIA did was clearly corrupt and wrong. It's just that it isn't what most people understand when they read today's headlines. It wasn't a case of the CIA hacking into other people's computers.

Many stories quote CIA director Brennan who said earlier this year:
I think a lot of people who are claiming that there has been this tremendous sort of spying and monitoring and hacking will be proved wrong
Many stories (like this one) claim that it's Brennan who was proven wrong, but instead, he was proven right. The investigation showed that at no time did the CIA hack anybody else's computer.

In pointing out the truth many people assume that I'm defending the CIA. I'm not. The torture program was morally wrong and beneath us as a country. Surreptitiously spying on the investigators into the program is clearly corrupt, and all involved need to be fired -- even if it turns out no law was broken (since it was the CIA's own computers).

I'm outraged, but believe we should be outraged by the right things, not the distorted story in the news. Seriously, I can't be more outraged at how the CIA revoked the declassification of things the staffers found useful to their investigation. It's not the spying (of their own computer) that angers me so much as their corrupt actions the spying enabled.

Update: @grayrisk points to this Lawfare blogpost with specifics. As you can see, CIA sysadmins had access to the system to administer it, but otherwise the system was supposed to be segregated from the rest of the CIA.

Monday, July 28, 2014

Cliché: open-source is secure

Some in cybersec keep claiming that open-source is inherently more secure or trustworthy than closed-source. This is demonstrably false.

Firstly, there is the problem of usability. Unusable crypto isn't a valid option for most users. Most would rather just not communicate at all, or risk going to jail, rather than deal with the typical dependency hell of trying to get open-source to compile. Moreover, open-source apps are notoriously user-hostile, which is why the Linux desktop still hasn't made headway against Windows or Macintosh. The reason is that developers blame users for being stupid for not appreciating how easy their apps are, whereas Microsoft and Apple spend $billions in usability studies actually listening to users. Desktops like Ubuntu are pretty good -- but only when they exactly copy Windows/Macintosh. Ubuntu still doesn't invest in the usability studies that Microsoft/Apple do.

The second problem is deterministic builds. If I want to install an app on my iPhone or Android, the only usable way is through their app stores. This means downloading the binary, not the source. Without deterministic builds, there is no way to verify the downloaded binary matches the public source. The binary may, in fact, be compiled from different source containing a backdoor. This means a malicious company (or an FBI NSL letter) can backdoor open-source binaries as easily as closed-source binaries.

The third problem is code-review. People trust open-source because they can see for themselves if it has any bugs. Or, if not themselves, they have faith that others are looking at the code ("many eyes makes bugs shallow"). Yet, this rarely happens. We repeatedly see bugs giving backdoor access ('vulns') that remain undetected in open-source projects for years, such as the OpenSSL Heartbleed bug. The simple fact is that people aren't looking at open-source. Those qualified to review code would rather be writing their own code. The opposite is true for closed-source, where they pay people to review code. While engineers won't review code for fame/glory, they will for money. Given two products, one open and the other closed, it's impossible to guess which has had more "eyes" looking at the source -- in many case, it's the closed-source that has been better reviewed.

What's funny about this open-source bigotry is that it leads to very bad solutions. A lot of people I know use the libpurple open-source library and the server (run by CCC hacking club). People have reviewed the libpurple source and have found it extremely buggy, and chat apps don't pin SSL certificates, meaning any SSL encryption to the CCC server can easily be intercepted. In other words, the open-source alternative is known to be incredibly insecure, yet people still use it, because "everyone knows" that open-source is more secure than closed-source.

Wickr and SilentCircle are two secure messaging/phone apps that I use, for the simple fact that they work both on Android and iPhone, and both are easy to use. I've read their crypto algorithms, so I have some assurance that they are doing things right. SilentCircle has open-sourced part of their code, which looks horrible, so it's probable they have some 0day lurking in there somewhere, but it's really no worse than equivalent code. I do know that both companies have spent considerable resources on code review, so I know at least as many "eyes" have reviewed their code as open-source. Even if they showed me their source, I'm not going to read it all -- I've got more important things to do, like write my own source.

Thus, I see no benefit to open-source in this case. Except for Cryptocat, all the open-source messaging apps I've used have been buggy and hard to use. But, you can easily change my mind: just demonstrate an open-source app where more eyes have reviewed the code, or a project that has deterministic builds, or a project that is easier to use, or some other measurable benefit.

Of course, I write this as if the argument was about the benefits of open-source. We all know this doesn't matter. As the EFF teaches us, it's not about benefits, but which is ideologically pure; that open-source is inherently more ethical than closed-source.

Wednesday, July 23, 2014

Everything can be a bomb

This last week, pranksters replaced the US flag on top the Brooklyn Bridge with a white-flag. Nobody knows who or why. Many in the press have linked this to terrorism, pointing out that it could've been a bomb. Not only local New York newspapers have said this, but also CNN.

Such irrational fears demonstrate how deeply we've fallen for police-state fears, where every action is perceived as a potential terrorist threat.

It could've been a bomb, of course. But what could also have been a bomb is a van full of C4 explosives driven across the bridge. There are no checkpoints at either end inspecting vehicles with bomb sniffing dogs. What also could've been a bomb is a ship full of fertilizer that, when ignited, would act as a small nuke. The point is that everything can be a bomb. Instead of using this as justification for an ever increasing police-state, we just need to accept this and live with the danger -- because this danger is, in the end, tiny. A thousand 9/11 events would still not equal cancer, for example.

I mention this because the former 9/11 commission released a new report yesterday stoking the fears of cyber-terrorism, calling for an increase in the cyber-police-state. This is nonsense. If government wants to fix cybersecurity, their first effort should focus on fixing their own computers rather than violating our rights. Enable SSL on all government computers, disable SQL string pasting, and get rid of all default/backdoor passwords.

Government is trying to ratchet up the fear of cyber-terror. It's not that their scenarios aren't possible, it's that they use this fear to drive the police-state. In exactly the same way we shouldn't have bomb sniffing dogs checking every car before it crosses the Brooklyn Bridge, we should not have the current proposals for cybersecurity that violate rights.

Tuesday, July 22, 2014

Um, talks are frequently canceled at hacker cons

Talks are frequently canceled at hacker conventions. It's the norm. I had to cancel once because, on the flight into Vegas, a part fell off the plane forcing an emergency landing. Last weekend, I filled in at HopeX with a talk, replacing somebody else who had to cancel.

I point this out because of this stories like this one hyping the canceled Tor talk at BlackHat. It's titled says the talk was "Suddenly Canceled". The adverb "suddenly" is clearly an attempt to hype the story, since there is no way to slowly cancel a talk.

The researchers are academics at Carnegie-Mellon University (CMU). There are good reasons why CMU might have to cancel the talk. The leading theory is that it might violate prohibitions against experiments on unwilling human subjects. There also may be violations of wiretap laws. In other words, the most plausible reasons why CMU might cancel the talk have nothing to do with trying to suppress research.

Suppressing research, because somebody powerful doesn't want it to be published, is the only reason cancelations are important. It's why the Boston MTA talk was canceled, because they didn't want it revealed how to hack transit cards. It's why the Michael Lynn talk was (almost) canceled, because Cisco didn't want things revealed.  It's why I (almost) had a talk canceled, because TippingPoint convinced the FBI to come by my offices to threaten me (I gave the talk because I don't take threats well). These are all newsworthy things.

The reporting on the Tor cancelation talk, however, is just hype, trying to imply something nefarious when there is no evidence.