A bit of IDS history

I saw this tweet go by, and as a creator of a popular IDS (BlackICE aka. Proventia), I thought I'd discuss a bit of the history, as things are a bit more complicated than just that:

Have you read any of these early computer security papers archived by NIST? csrc.nist.gov/publications/h… The Anderson work was the basis for IDS.

— Richard Bejtlich (@taosecurity) August 10, 2012

From talking to fellow network IDS creators (Marty, Ron, Marcus, Vern), there was never any basis for our work other than "packets". We each built a packet analyzer to look for stuff we were interested in, each building a wildly different architecture in pursuit of wildly differing goals.

IDS is how everyone else described our work, but this was not necessarily the goal we had when starting out. I had code running for years (starting in 1990) before I ever heard the term "intrusion detection system". In much the same way, I called my inline version "inline IDS" before others decided to call it "IPS".

This is not say that the above papers don't deserve credit for coming up with brilliant ideas. They probably do. It's just that I've never read them, and they were not the "basis" for my work.

Likewise, while not the "basis", they probably influenced my work in some ways. Other people read them, creating a mishmash of ideas that probably influenced the direction of my work without me realizing it. Simply the fact that a "syslog" demon exists and that I send events to it means that I'm undoubtedly influenced by whoever invented "syslog".

Consider Marcus's IDS called "NFR". If it's based on anything, it's based on MUDs (multi user dungeons, a type of early text based "game"). He had bits of code lying around for which he repurposed into some totally unrelated bit of code. I use this as the most amusing example, but pretty much all early Internet code was developed with the same sort of model: lurching in unexpected directions rather than being based on a plan.

My own start was in 1990 with my first job out of college working as a generic software programmer for a company that made a product called the "Protolyzer", a protocol analyzer competing against the better known Network Genral "Sniffer". The Protolyzer's unique features was that it had a graphical user interface (based on OS/2) when all competing tools were text based.

Among my duties was to create "addons" for that plugged into the Protolyzer to look for interesting stuff other than simply decoding packets. A lot of these addons were to diagnose network faults. Others were related to security. Since these addons could act as filters, the Protolyzer had primitive "IPS" capability, in that it could capture, filter out security events (namely ARP spoofing), and retransmit.

The company was then bought by Network General and the technology folded into the Sniffer, and largely died. I tried to interest the company in resurrecting it in a Version 2.0 form that I had come up with using "streaming state-machines". You see, all previous protocol analysis was done at the level of "packets" or "packet payloads". My design was to use the TCP "stream" as the atomic unit, and instead of buffering packets to reassemble them into buffers, to parse the stream using a state-machine.

This is actually a pretty common technology now, and the basis for most network IDS/IPS, but back then it was unique.

But Network General wasn't interested, so in 1998 when Network General merged with McAfee Associates to form Network Associates, I left to found my own company and implement my technology. The result was "BlackICE", the first intrusion prevention system, now sold as IBM Proventia (ISS acquired my company in 2001, then IBM acquired ISS in 2006).

The point of this story is to describe what my stuff was "based on". It was based upon my experience with hacking computers, and based upon my experience in analyzing packets. Indeed, I probably would have done a better job had I read more academic papers. For example, I "invented" my own multi-pattern matching system that, as it turns it, is just the Aho-Corasick algorithm. I would have saved a lot of time and effort had I just studied a bit better.

Conclusion

I'm not trying to diminish earlier works, or praise us early IDS inventors for inventing everything ourselves. Instead, I'm trying to describe how things were messy, how we didn't really follow a plan, didn't really "base" our work on anything, and how we are as surprised as everyone else how things turned out.

Also, I don't want to put words in the mouths of Vern, Marcus, Marty, or Ron (which is why I only use first names so that it may be hard for you to identify them), so much as tell my own story. They will undoubtedly have very different stories if you talk to them. (I'm eagerly awaiting a comment from Marcus correcting my interpretation).

So in short, while IDS may indeed be "based" on earlier things, the path to get there is "complicated".

---

Updates:

@securityskeptic @erratarob That paper and the Tweets/posts following miss @toddheberlein and his NSM taosecurity.blogspot.com/2007/04/networ…

— Richard Bejtlich (@taosecurity) August 10, 2012