Fun with IDS funtime #3: heartbleed

I don't like the EmergingThreat rules, not so much because of the rules themselves but because of the mentality of the people who use them. I scan the entire Internet. When I get clueless abuse complaints, such as "stop scanning me but I won't tell you my IP address", they usually come from people using the EmergingThreat rules. This encourages me to evade those rules.

The latest example is the Heartbleed attack. Rules that detect the exploit trigger on the pattern |18 03| being the first bytes of TCP packet payload. However, TCP is a streaming protocol: patterns can therefore appear anywhere in the payload, not just the first two bytes.

I therefore changed  masscan to push that pattern deeper into the payload, and re-scanned the Internet. Sure enough, not a single user of EmergingThreats complained. The only complaints from IDS users were from IBM sensors with the "TLS_Heartbeat_Short_Request" signature, and from some other unknown IDS with a signature name of "Heartbleed_Scan".

That the rules can so easily be evaded is an important consideration. Recently, the FBI and ICS-CERT published advisories, recommending IDS signatures to detect exploits of the bug. None of their recommended signatures can detect masscan. I doubt these organizations have the competency to understand why, so I thought I'd explain it in simple terms.

How TCP and Sockets work

Network software uses the "Sockets API", specifically the "send()" function. Typically, the packets we see on the wire match the contents of sends. If code calls these two functions back-to-back...

    send(fd, "abc", 3, 0);
    send(fd, "xyz", 3, 0);

...then we'll see two TCP packets, one containing a payload of "abc" and the other containing a payload of "xyz".

The send() function will only transmit immediately like this when the receiver is keeping up with the transmitter. If the receiver is falling behind, then the send() function writes the data into a kernel buffer instead of transmitting, waiting for the receiver to catch back up. Multiple sends will be combined into the same buffer. In this example, in the case of slow connections, the kernel will combine "abc" and "xyz" into a single buffer "abcxyz". Later, the kernel may transmit "a" in one packet (if the receiver continues to be slow) followed by "bcxyz", or maybe "abcx" followed by "yz", or the entire buffer at once with "abcxyz". TCP makes sure these bytes are sent in order, but doesn't guarantee how they might be fragmented.

Masscan exploits this feature of TCP. When transmitting packets to the server, it combines data so that the Heartbeat pattern occurs later in the packet. It then pretends to be a slow connection, causing the server to combine data in the responses. Thus, in neither the to or from direction do any packets start with the triggering pattern |18 03|.

Viewing the packets

You can see this in action within this packet-capture using Wireshark. Frame #10 is where masscan transmits the Heartbeat request. It combines this with an Alert request, thus pushing the |18 03| pattern seven bytes deeper into the packet. This evades detection when the packets are sent to the server.

In order to cause the response to evade detection, masscan must appear as a slow receiver. It can do this in a number of ways. In this example, it does this by advertising a small "window". This is a parameter in TCP packets that tells the other side how many bytes are available in the receive buffers. This is shown in frame #6, where masscan claims to have a window size of only 600 bytes.

This causes the server side to buffer the send() requests in the kernel. Normally, we should be seeing some separate packets at this point, such as a "Server Certificate", "Server Key Exchange", and "Server Hello Done" message, followed by the "Heartbeat" response. Because of the small window, the server has combined these all together and streamed them across the connection. In frame #16, we see the result, with the pattern |18 03| at offset 0x009F in packet.

This trick is, by itself, detectable. Notice that Wireshark is unhappy with this, and helpfully prints the message [TCP Window Full] several times. An IDS might use the same logic to detect a problem. But there's other ways to cause the same effect that an IDS might not detect. For example, masscan might choose to not acknowledge sent data, forcing data to build up in the remote buffers. That an IDS might trigger on one trick doesn't mean there aren't other easy tricks to evade detection of fixed-offset TCP payloads.

Verifying with Security Onion

The "Security Onion" is a Linux distro with many IDS packages built in. I installed the latest version and ran it, including running the "pulledpork" tool to download the latest signatures. I also copied the Fox-IT rules linked by the FBI into the "local.rules".

To test that things were running correctly, I ran the "ssltest.py" script -- one of the first tools used to test for the Heartbleed problem. This is what running the script looked like:

As expected, the IDS had no problem detecting the exploit. All the appropriate rules from the FBI/ICS-CERT guidance fired, including the "Snort Community Rules", the "Emerging Threat" rules, and the "Fox-IT" rules.

I then ran masscan, as shown below. Nothing triggered on the IDS, as I expected. No other events fired either, such as complaining about the "window" being full. The underlying Snort engine may have logic designed to detect things like small TCP windows, but if it did, Security Onion didn't have them enabled.

I also ran my heartleech tool to automatically extract the private-key from the target. In theory, it should be detectable. It uses the same trick to evade detect on the packet sent to the server, but doesn't have the same control over packets coming back from the server (heartleech goes through the Sockets API whereas masscan doesn't, so can't control the "window"). It appears that the signatures, in order to avoid false-positives, won't always trigger on a response unless they have also seen a request -- thus allowing heartleech to evade detection sometimes. I could only get the Security Onion device to trigger about one out of every three times I ran heartleech.

How IDSs can detect Heartbleed

The above tricks masscan uses only evades "pattern-matching" signatures. It doesn't work against "protocol-analysis". In the first picture above, you'll notice how Wireshark (a protocol-analyzer) isn't fooled by the evasions. It correct decodes the SSL protocol and knows which bytes are the Heartbeat.

The "Bro" open-source IDS works off the same principle. It decodes SSL (with a recent upgrade) can detect the Heartbleed vulnerability, regardless of what masscan and heartleech do to the packets. Some commercial IDSs use protocol-analysis as well, such as the IBM sensor** that trigger the "TLS_Heartbeat_Short_Request". Furthermore, it's likely that Snort will be upgraded in the near future to implement Heartbleed detection in their "SSL Preprocessor", which similarly does protocol-analysis.

That doesn't mean any of these are completely correct, either. SSL can run on all ports, but for performance reasons, preprocessors might be configured to examine traffic on a limited number of ports. This will miss many attacks. The following is an example attacking a server on port 444 [pcap]:

Another problem is that while protocol analysis can't be evaded by simple TCP tricks, the person who wrote the signature may have written an incomplete one. The IBM protocol-analysis is able to measure the length of the request, but the threshold for triggering may be too low. They may have put a threshold of only 3 bytes for the request, which is the standard exploit, but I could easily increase that to 61 bytes to make an incoming HeartBEAT look indistinguishable from a HeartBLEED attempt.

Lastly, there is the problem of response time. IBM uses "state-machines" to decode protocols, and thus was able to release it's protocol-analysis signature for this problem in roughly the same timeframe as others released pattern-matching signatures (April 9). Bro has a scripting language that likewise makes fast turn arounds easier, but there's no way for such fast updates to be automatically imported into a sensor. Snort and Suricata have said they are updating their SSL preprocessors, but as of April 28, they haven't been released yet.

Conclusion

Just because somebody has released a pattern-matching signature for the latest threat doesn't mean it's adequate. We have organizations like the FBI and ICS-CERT that blinding repeat such recommendations because they don't fully grasp how TCP works. The consequence is that if you followed the FBI/ICS-CERT recommendations, you didn't detect the masscan Heartbleed scan I did this last weekend.

---

Disclaimer: I created the technology in the IBM products. I've had zero involvement with them for the last 7 years, so I know little about what they've been doing lately, but the fact that they detected masscan implies they are still using the protocol-analysis state-machines that I put into the product those many years ago. By the way, masscan itself uses those same state-machines to parse responses from the server: if you want to see how they work, just read the "proto-ssl.c" source code in masscan.

---

Victor Julien, lead Suricata developer, detected my scan with the new protocol-decode added to Suricata:

1. 04/26/2014-13:20:05.515184  [**] [1:2230012:1] SURICATA TLS overflow heartbeat encountered, possible exploit attempt (heartbleed) [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 209.126.230.74:20193 -> 192.168.47.27:443

Seth Hall, a Bro developer, detected my scan with their new protocol-decode logic:
1398506591.523781 CsXIjO1BWbvfZpbnha 209.126.230.74 17193 x.x.x.x 443 - -- tcp Heartbleed::SSL_Heartbeat_Attack An TLS heartbleed attack was detected! Record length 3, payload length 16384 - 209.126.230.74 x.x.x.x 443 - worker1-10 Notice::ACTION_LOG 3600.000000 F - - - - -

Both these cases demonstrate the point that to correctly detect such threats, you need protocol-analysis, not pattern-matching.