Monday, December 16, 2013

CCC, 100-gbps, and your own private Shodan

One of the oldest/biggest "hacker" conventions is the CCC congress every December in Germany. This year, they are promising 100-gbps connectivity to the Internet. That's 'g' as in 'giga', and as in 'omfg that's a lot of bandwidth'.

So, what shall we do with all this bandwidth? The answer is masscan: scan the entire Internet and create your own, private, Shodan-style database.

Here is an example:
masscan -p80,21,22,23,25,110,143,443,3389,U:161 --banners --rate 1500000 -oB ccc-shodan.scan --exclude

This scans the entire Internet at one gigabit speed saving the results into a binary file. When you get home, you can extract the contents of that file to build a searchable database, such as sticking everything into an Elastisearch engine. You'll have everything: HTTP headers, SSL certificates, SNMP device descriptions, and so on.

In the above example, I'm grabbing HTTP, FTP, SSH, Telnet, SMTP, POP, IMAP, SSL, RDP, and SNMP, including their banners. That's 10 ports, which at only 1-gbps, will take about 7 hours to complete.

But that's doing things the normal way, using the Ethernet port on your laptop. If you've got two laptops, you can spread the scan across them. Add the "--shard 1/2" on one laptop, and "--shard 2/2" on the other. Any number of shards are supported, so if you have 10 friends, you can combine your efforts and complete this scan in 40 minutes. Remember: the CCC promises enough bandwidth to satisfy all your friends (well, up to 100, but seriously, nobody has that many people they can put up with).

Better yet, bring 10-gbps Ethernet cards. CCC said they'll have a place where you can hookup at the faster speed. You can bring a small portable desktop (like a Shuttle box computer). Or, you can use MacBooks with their Thunderbolt connectors, along with a cage holding a 10gig card. I show this last option in the screenshot below:

What you see in this picture is my 11-inch MacBook Air, with a cage attached via Thunderbolt to the right. Inside the cage is an Intel 10gig Ethernet adapter. The notebook is running Ubuntu. I'm using Linux instead of Mac OS X here so that I can use the "PF_RING" drivers, which transfer packets directly to the Ethernet card bypassing the Linux kernel. (Also note I'm using copper -- CCC implies their switches use fiber, so bring SPF+).

On the screen you see me running masscan. Yes, the screen is physically damaged, which is why it's my lab machine. What you see here is that I've launched masscan with parameters similar to those above, with only 5 ports, and with 10-million packets/second as the desired rate.

It's not working like it should. In theory, it should be doing 10 million packets/second, but as you can see, it's only achieving 3.7 million packets/second. I've tried debugging this to see why, but I can't figure it out. I suspect that it's the Thunderbolt connection that's the problem (e.g. it's just as fast using one core as two). It's still a good result: no other port scanner, on any machine, even high-end servers, can transmit 3.7 million packets/second. It's a pretty good result for a tiny 11-inch notebook computer.

Newer MacBooks might perform better. They support Thunderbolt2, which is twice as fast as the original Thunderbolt. MacBook Pros also have two Thunderbolt ports. MacBook Pros also have twice the cores, at twice the speed, so in theory could achieve 20-gbps, or 30 million packets/second. That should be enough to get your entire Shodan clone created in a half hour. I doubt anybody will go to this effort, but those of you with 15-inch MacBook Pros should at least go to the store to get a second Thunderbolt Ethernet adapter. You can get at twice the speed of gigabit, splitting the scan across two adapters (--adapter[0] eth0 --adapter[1] eth1) (which is a separate feature from sharding, so you are still 1/3 when scanning with friends).

Remember that masscan is insanely easy to build for whatever computer you have. For Ubuntu, the process is:
apt-get install git gcc gmake libpcap-dev
git clone
cd masscan
For Mac OS X, you can just run 'git' and XCode to compile it (using the XCode project). Or, you can type 'make' from the command-line. This assumes you have the compiler installed.

On Windows, you have to get WinPcap, and you have to get some sort of 'git' program. I use the "msysgit" environment. It's nice because it includes gcc/gmake, so one could build masscan with no further software. Personally, I don't use gcc on Windows, but VisualStudio 2010 instead, but both work fine.

On Windows, there are a few tweeks you have to be aware of. Using your own IP address for the scan really slows things down, a lot. Therefore, you have to spoof an otherwise unused IP address (e.g. --source-address Also, you have to use the --sendq parameter to make it go faster. Without these options, it'll run at an abysmal 30k packets/second. With these options, you should get over 1-million packets/second.

If you go to this year's CCC and don't scan the entire Internet, you are doing it wrong. I expect all you kids to come back from Christmas CCC Congress with your own, personal Shodan databases. 

Note: The 10gbps NICs from Intel are the best to use with masscan. That's because masscan can use PF_RING, a custom driver. Using the built-in Linux driver, masscan maxes out at 2.5 million packets/second. Using PF_RING, it can do 25 million packets/second -- according to my benchmarks. The single port Intel NIC using copper costs $350, dual port $550. NICs that support fiber will cost more, you need to get the SPF+ NICs, then buy the SPF+ fiber modules.

Note: I'm not sure how big a full, Shodan-style scan of the Internet will be, but the binary format of Masscan is pretty small. I'd estimate maybe 10-gigabytes for the entire scan. You can use "--nocapture cert" in order to avoid storing the SSL certificates, which is the bulkiest data it has. You'll can use the "--rotate" feature to split the results across smaller files. You'll use the "--readscan" feature later in order to parse the binary results into something like XML, JSON, or a Redis database.

Note: Some targets don't like getting scan. I mean, they really don't like getting scanned to the point of sending men in suites to politely suggest that you stop. This leaves an embarrassing hole in the scan data. This is a good time to fill that hole -- by the time they get their act together and send agents the conference to shutdown the network, the conference will be over. I'm just saying...

Note: For extra special fun, you can use masscan as a "load generator" instead of a port scanner. Masscan gets its speed by being a "user-mode TCP stack". That speed can be repurposed. Instead of a one-to-many configuration, you go do a many-to-one, spoofing a lot of local IP addresses in order to send lots of requests at the remote target(s).

Configuring source addresses/ports in masscan is limited. The number of sources it'll spoof must be a power of 2 (that is, 1, 2, 4, 8, 16, etc.). That applies to both the port and IP address.

Running masscan spoofing these, against one target, would look like this:

masscan -p80 --banners --rate 10000000 --repeat --source-address --source-port 0-65535 --output-format null

The "--repeat" command means to repeat and start the scan over again in an endless loop. Thus,  even though the scan actually sends only one probe, we can repeat the scan a million times a second. The actual limiting factor here is the source. At only 8 IP addresses and 64k ports per address, you'll be duplicating TCP sessions (stuck in TIME_WAIT) long before filling the 1-gbps pipe. To avoid duplication, you need either to spoof more source IP addresses, or target more destination ports/addresses.

In principle, this should run at the rate of millions of HTTP requests/second using a 10-gbps card. In other words, you should be able to generate the same load on a website with a single machine as would be produced by a 10,000 node botnet -- given a connection like that provided by CCC.


Unknown said...

Are you coming to CCC to do a scan?

Robert Graham said...

Sadly, my travel schedule won't allow me to make CCC. But maybe somebody will let me log into their machine on the last day to do a scan.