Early in 2003, I went to the New York chapter meeting of the ISSA. I forget what my presentation was about, but Dave Aitel was in the audience. At the time, Dave was famous for things like his "spike" fuzzer of Microsoft protocols. So I went into detail on my IPS code, how I had created a Microsoft protocol stack that did fragment reassembly at both the SMB (Named Pipes) layer and the MS-RPC layer (on top of existing TCP/IP reassembly, of course). Moreover, my code was extremely fast, doing this at a full gigabit/second.
I could tell at the time Dave didn't believe me. He was the well-known expert hacker at SMB/MS-RPC, and if he didn't know about it, then it probably didn't exist. But in fact, he was only the expert in the "open source" world. Plenty of closed-source programmers like myself had been doing far more with Microsoft protocols for over a decade. Lots of clever stuff exists in closed-source products that the open-source world doesn't know about.
But also his incredulousness came from the fact that such an immense of code was worthless: there was no big threat for MS-RPC at the time.
I point this out to demonstrate that Blaster was predictable. People like Dave and I had put a lot of effort into writing code to address this issue we knew was coming. Microsoft's MS-RPC was an enormous attack surface, enabled by default in not only their servers but also their desktops. Once hackers cracked this open, there was going to be rush of bad stuff flowing out. People like us were just waiting for that crack to appear, and for the bad stuff to drive sales.
That crack appeared on July 8, 2003. I opened my browser at 1:15pm EST (9:15am PST) in order to check out the latest Microsoft "Super Tuesday" vulnerability announcements. When I read the MS03-026 announcement, a chill went through my spine. This was it, this was what I'd spent the last three years building my product to defend against.
By the way, this annoyed me. I had friends at Microsoft. They'd often warn me when big things were coming, like the CodeRed and Nimda vulns. But they didn't warn me about this one.
The first thing to do was call up the CEO of the company. Internet Security Systems (ISS) had bought my company two years prior, and I was merely the "Chief Scientist", hanging around waiting for my golden handcuffs to expire. I had no official authority in the company. But I called the CEO and said "This is an emergency. I'm taking over. I'm going to bypass process, tell people what to do (including you), and make a lot of people unhappy". He said "ok". This annoyed me -- I was prepared for an argument protesting my little corporate coup, but he gave into me without a fight.
The first thing was to go to the in-house hackers, the "X-Force researchers", and get them onto this bug to figure out the details. Unlike most vulnerabilities, both the discoverers and Microsoft hid the details of MS03-026. They knew it was bad -- they wanted to forestall the inevitable. Therefore, I needed the (brilliant) X-Force guys to look into this.
While they went off and worked, I did my own investigation. About an hour later, one of the X-Force guys, let's call him "Neil", came into my office with his solution: he had diffed the patch and found a stack buffer that had been increased from 32-bytes to 512-bytes. That matched up with what I'd found in the last hour by working form the top down with the API -- a field for the machine name in the MS-RPC IRemoteActivation protocol. Microsoft machine names are 16 Unicode (2-byte) characters, while full filenames are 256 Unicode characters.
It was now 2:15pm. We had fully discovered in an hour all the details Microsoft had wanted to hide from us, using two independent means. This is a good lesson about hacking: knowledgeable people attempted to hide the details, in an attempt to save the Internet from disaster, and they only delayed us by 1 hour.
About 3 hours later, "Neil" had a working exploit. It was rough, but it confirmed that this thing was robust and "wormable". In other words, about 4 hours after Microsoft's public announcement, we knew a worm was coming.
Raising the AlertCon wasn't simply a matter of changing the GIF on our website. There were things tied to this. It entailed calling our major customers and warning them, and working with various agencies like the FBI and DHS to warn them as well. Throughout the next couple days I spent a lot of my time on conference calls.
In addition, we got an IPS signature out by the next morning. This is what I had prepared for: because we had a full protocol decode of the MS-RPC protocol, we could build a signature that measured the length of the buffer (the machine name), and trigger an event when that buffer exceeded 16 characters. It's actually kind of cool: when the event triggers, it doesn't tell you merely that it detected the problem, it also tells you the length of the buffer that was sent. Because it triggered on the length of this field, the signature caught the later exploits that were developed. It also, from the very first day, handled SMB fragmentation and MS-RPC fragmentation.
Then I had to wait.
This was the hardest part. A worm was coming. It was obviously coming. Moreover, it was going to be the worst worm ever -- infecting more machines than all previous worms combined. Whereas the most active worm prior to this had only infected a few hundred thousand machines, the coming worm was going to infect tens of millions of machines. That's because this was the first "remote root in the default install" of Windows NT desktop. Previous Windows worms, like CodeRed, Nimbda, and Slammer, had infected only servers. Sure, these servers could optionally be installed on desktop computers, but it wasn't common. This was the first worm that threatened all Windows desktops, in their default configuration, with no special software installed.
I warned customers, mailing lists, anybody who would listen. But my predictions weren't coming true. It had taken us hours to write an exploit, I thought at the outside one would appear in the public within a week. It took several weeks, until July 25, for a group of Chinese hackers known as XFocus to publish an exploit. I had predicted a worm would appear the next day after a public exploit, in fact, it took until August 11.
My "friends" at ISS kept mocking me for the entire month, asking me were this "imminent" worm was.
So I was a wrong on the timing. But I was right on the spread. Blaster infected well over 10 million machines. I've seen estimates as low as 100,000, but they are nonsense. The DoD alone experienced more than 100,000 infections. Every Windows NT desktop, more than 100-million of them, was vulnerable to the bug, and only a small percentage had been patched when the worm hit.
But I had other failures in my predictions. I predicted that the worm would disable the Internet. This didn't happen -- because the guy who wrote the worm was incredibly inept. It was full of bugs that, accidentally, guaranteed the Internet wouldn't fail.
After the worm hit, I pulled out my trusty IDApro and disassembled the entire thing all the way back to the original C code. You can get a copy of this C source at http://www.robertgraham.com/journal/030815-blaster.c. Read the comments, they are amazing, as I describe what the writer of the worm was probably thinking as he wrote the code -- and the stupid mistakes he made.
Most of the code, including variable names, is accurate and "original". That's because the worm creator copied bits and piece from around the Internet. Once I reverse engineered a bit, I was able to google the bits I had to find the parts I hadn't. For example, the code he used to calculate the TCP checksums was wrong. Even though it was wrong, Blaster worked, because by 2003, network chips would do the checksum calculations for you, so it didn't matter that the code was wrong. But, because the code was wrong, it was distinctive and googlable, so I was able to find the original source that he had copy/pasted from.
The biggest bug was in the TFTP transfer of the worm itself. After infection, the worm used the TFTP protocol to upload a copy of itself to the victim. Unfortunately, the TFTP code didn't retransmit lost packets, and it took 7 packets to fully upload the worm. What this meant was that once the worm started causing network congestion, and packets started being lost, it would fail to continue to be spread. Thus, instead of spreading to a point the entire Internet failed, it spread to the point that the Internet was dropping 1 in 7 packets.
I had been telling people the Internet was going down hard, and that nothing could stop it. I was wrong: inept worm writers who can't figure out how to retransmit lost packets would prevent the Internet from going down.
This made Blaster the best thing that could've happened to the Internet, because it quickly inoculated the Internet against better worms. New, and better, worms appeared after Blaster exploiting MS03-026, but they didn't have much impact because people had cleaned up their systems after Blaster.
After Blaster came a flood of MS-RPC vulnerabilities, as predicted. The next year the MS04-011 bug led to the Sasser worm. For most people, Sasser was worse than Blaster, because it was competently written. But at the same time, it infected fewer systems. In the year since Blaster, people changed how they connected to the Internet, putting more stuff behind firewalls, and blocking MS-RPC. Had Blaster not happened before Sasser, then Sasser would've been the worm that killed the Internet.
The lesson
The lesson here isn't that I was clever and predicted things (though damnit, I was clever), but how little that mattered. Every cybersec company came out with great marketing materials about their great handling of Blaster, no matter how lame their actual response. It was impossible for customers to tell the difference. On one end, everyone else would exaggerate their skill. On the other end, they would exaggerate our faults. This made it impossible for almost anybody to tell the difference. Sure, I had lots of gratifying experiences with customers were our product rocked when everything else failed, but I completely failed at communicating this to the market.Thus, the lesson here is: don't try to be too clever, it won't work. Sure, try to stay slightly ahead of your competitors, but doing things like putting a reassembling SMB/MS-RPC stack in your code three years before it was needed it just pointless.
No comments:
Post a Comment