Thursday, August 09, 2012

A bit of IDS history

I saw this tweet go by, and as a creator of a popular IDS (BlackICE aka. Proventia), I thought I'd discuss a bit of the history, as things are a bit more complicated than just that:

From talking to fellow network IDS creators (Marty, Ron, Marcus, Vern), there was never any basis for our work other than "packets". We each built a packet analyzer to look for stuff we were interested in, each building a wildly different architecture in pursuit of wildly differing goals.

IDS is how everyone else described our work, but this was not necessarily the goal we had when starting out. I had code running for years (starting in 1990) before I ever heard the term "intrusion detection system". In much the same way, I called my inline version "inline IDS" before others decided to call it "IPS".

This is not say that the above papers don't deserve credit for coming up with brilliant ideas. They probably do. It's just that I've never read them, and they were not the "basis" for my work.

Likewise, while not the "basis", they probably influenced my work in some ways. Other people read them, creating a mishmash of ideas that probably influenced the direction of my work without me realizing it. Simply the fact that a "syslog" demon exists and that I send events to it means that I'm undoubtedly influenced by whoever invented "syslog".

Consider Marcus's IDS called "NFR". If it's based on anything, it's based on MUDs (multi user dungeons, a type of early text based "game"). He had bits of code lying around for which he repurposed into some totally unrelated bit of code. I use this as the most amusing example, but pretty much all early Internet code was developed with the same sort of model: lurching in unexpected directions rather than being based on a plan.

My own start was in 1990 with my first job out of college working as a generic software programmer for a company that made a product called the "Protolyzer", a protocol analyzer competing against the better known Network Genral "Sniffer". The Protolyzer's unique features was that it had a graphical user interface (based on OS/2) when all competing tools were text based.

Among my duties was to create "addons" for that plugged into the Protolyzer to look for interesting stuff other than simply decoding packets. A lot of these addons were to diagnose network faults. Others were related to security. Since these addons could act as filters, the Protolyzer had primitive "IPS" capability, in that it could capture, filter out security events (namely ARP spoofing), and retransmit.

The company was then bought by Network General and the technology folded into the Sniffer, and largely died. I tried to interest the company in resurrecting it in a Version 2.0 form that I had come up with using "streaming state-machines". You see, all previous protocol analysis was done at the level of "packets" or "packet payloads". My design was to use the TCP "stream" as the atomic unit, and instead of buffering packets to reassemble them into buffers, to parse the stream using a state-machine.

This is actually a pretty common technology now, and the basis for most network IDS/IPS, but back then it was unique.

But Network General wasn't interested, so in 1998 when Network General merged with McAfee Associates to form Network Associates, I left to found my own company and implement my technology. The result was "BlackICE", the first intrusion prevention system, now sold as IBM Proventia (ISS acquired my company in 2001, then IBM acquired ISS in 2006).

The point of this story is to describe what my stuff was "based on". It was based upon my experience with hacking computers, and based upon my experience in analyzing packets. Indeed, I probably would have done a better job had I read more academic papers. For example, I "invented" my own multi-pattern matching system that, as it turns it, is just the Aho-Corasick algorithm. I would have saved a lot of time and effort had I just studied a bit better.

Conclusion

I'm not trying to diminish earlier works, or praise us early IDS inventors for inventing everything ourselves. Instead, I'm trying to describe how things were messy, how we didn't really follow a plan, didn't really "base" our work on anything, and how we are as surprised as everyone else how things turned out.

Also, I don't want to put words in the mouths of Vern, Marcus, Marty, or Ron (which is why I only use first names so that it may be hard for you to identify them), so much as tell my own story. They will undoubtedly have very different stories if you talk to them. (I'm eagerly awaiting a comment from Marcus correcting my interpretation).

So in short, while IDS may indeed be "based" on earlier things, the path to get there is "complicated".


Updates:

1 comment:

Unknown said...

I think you don't give the path you took enough credit- it's by NOT seeing what everyone else has built their perspective on that can often lead to a new disruptive technology. Sometimes it does lead you to invent what was already being invented, but many times it can also lead to a new, better way to do things. And in network security, being able to produce the same results with an entirely different infrastructure from the norm is a security enhancement in and of itself. And how many times have we found in network gear the same tired but accepted methods for accomplishing tasks that are somehow blown away later by a new technology that shows how imperfect it always really was? ATM or Frame Relay, anyone? Meet "Ethernet everywhere". They seemed so right for the WAN, and Ethernet so, so wrong… and yet…

Could you have read those papers and been well on your way to creating some things sooner? Sure, but those things may have just looked like "another XYZ IDS".

Entrepreneurial spirit is most vividly displayed in times of crisis- the entrepreneur views the crisis as an opportunity and seizes the moment. Less glamorized but often just as useful is the opportunity created by simply being able to do something differently than the rest of the pack. Something better, something cheaper, something more secure. As for reading up all those papers others wrote, Sonny once said, "Save it for the library!"