Tuesday, February 19, 2013

Unlearning College


I’m writing up my Shmoocon talk as a series of blog posts. In this post, I’m going to talk about the pernicious problem of college indoctrination. What colleges teach about networking is bad. A lot of material is out of date. A lot is just plane inaccurate (having never been “in date”). A lot is aspirational: networks don’t work that way, but your professor wishes they would. As a consequence, students leaving college are helpless, failing at real-world problems like portability, reliability, cybersecurity, and scalability.

A good example of this problem is the college textbook “Unix Network Programming” by Richard Stevens. It’s the most common textbook on the subject. In my Shmoocon talk, over half the audience raised their hands in response to my question “have you read this book?”.

Students love this book. Stevens is considered almost a saint. But he’s not a saint, he’s the devil. Stevens is the AOL of network programming. AOL was a huge company during the dot-com boom that aggressively marketed dialup access to newbies. It flooded the Internet with people steeped in the AOL version of the Internet that was out of sync with the real Internet. Both teach that AOL/Unix is the primary provider of the Internet, and that if it’s not something AOL/Unix provides, then it’s not something you need to worry about.

But in much the same way AOL sucked at networking, so does Unix.


Consider the issue of byte-order or “endianness”. When computers store bytes in memory, they can do so either left-to-right or right-to-left. It’s a simple idea made simpler by the fact that you almost NEVER need to worry about it.

Consider the lowly IPv4 address. There are three “canonical” forms of the IP address. These are:

char *ip_string = "10.2.3.4";
unsigned char ip_bytes[] = {10, 1, 2, 3};
int ip_integer = 0x0a010203;

The first is the string humans read. The second is how the address appears in a packet on the network. The third is an internal representation of the address as an integer (written in hex, so the decimal number “10” becomes “0a”).

Converting “ip_string” or “ip_bytes” into an integer means parsing the bytes one at a time, such as in in this example:

int ip_integer = ip_bytes[0]<<24 | ip_bytes[1]<<16 | ip_bytes[2]<<8 | ip_bytes[3]

The problem is that back in 1983 when Bill Joy and BBN put the first TCP/IP stack into Unix, they added a fourth incorrect form where they used C to “cast” the raw form of the packet into an integer:

int ip_something = *(int*)ip_bytes;

While conceptually the same, “ip_integer” and “ip_something” are different, for two reasons.

The first is that on RISC processors, this will sometimes crash. RISC demands that integers be aligned on natural boundaries. Compilers guarantee that all “internal” integers will be aligned with padding and such. But they can’t guarantee that for “external” integers. When the coder casts a packet pointer into an integer pointer, the program may crash.

The second issue is the byte-order/endian problem. On little-endian processors like Intel x86 and most ARM, the bytes will be reversed, so the value of “ip_something” will be:

int ip_something = 0x0302010a;

Having first made the wrong decision to cast integers, Unix provides functions to fix the situation with “byte-swapping” macros of “htons()”, “ntohs()”, “htonl()”, and “ntohl()”. You might use them like:

int ip_integer = ntohl(*(int*)ip_bytes);

Students think Unix’s byte-swapping macros are a good solution to the problem. But this is a problem of Unix’s own making. Had you never mixed integer points with byte pointers in the first place, you never would’ve needed to swap the bytes.

By the way, if you learned networking using some language other than C/C++, you are probably wondering what the big deal is. Your language doesn’t allow you to cast integers. You’ve always been forced to do things the right way. Therefore, you’ve never been confused by byte-order. This is purely a problem for people who use C and who cast pointers between integers and bytes.


I use byte-order because while it’s a largely unimportant issue, it’s easy to understand. The more important issues are harder to understand.

My Shmoocon talk was about scalability. The Unix kernel has enormous scalability problems because back in the day, AT&T designed Unix to control the flow of traffic, but not carry the network traffic itself. Unix replaced the switch board operators, but didn't replace the circuits that carried telephone traffic. In modern time, engineers talk about the "control plane" and "data plane", reflecting this ancient distinction.

That's the crux of my talk. Unix has been good enough for many "data" functions, like being a web server. But its limits are becoming increasingly apparent, which is why Nginx is rapidly displacing Apache. Apache relied heavily on the kernel to maintain abstractions like threads; Nginx does raw programming getting rid of such abstractions. My talk was about taking that idea to the logical conclusion, getting rid of the kernel completely, splitting a typical Linux server into "control plane" and "data plane" sections, where the "data plane" never touches the operating system kernel, and thus gets 100x the scalability.

The Unix solution to byte-order is wrong. So is the Unix solution to network stacks, multi-core programming, and memory management. These things are holding you back, so if you learn more non-Unix network programming, you can achieve things that otherwise seem impossible today.



Update: You still have to use the ntohs() solution when interacting with the sockets API, you just shouldn't use it when parsing data from packets.

6 comments:

Anonymous said...

WTF? This does not make any sense. It's not "raw" programming, it's just a different programming paradigm.

What does "Unix" even mean in your blog post? select() has been part of POSIX for more than ten years.

Anonymous said...

Also, even a "core contributor" of NodeJS disagrees: https://speakerdeck.com/felixge/the-nodejs-scalability-myth?slide=20

Robert Graham said...

What "unix" means in the blogpost is the underlying unix-like operating system and it's APIs. I suppose you could be using POSIX APIs on top of Windows or VxWorks, but the same comments apply.

Morgaine Dinova said...

This article was severely lacking in logic.

First of all it makes an incredibly stretched analogy between AOL's networking and Stevens' book, and somehow turns that into a reason to call Stevens "the devil". It really makes no sense at all. Books are written to educate and for clarity, and Stevens achieved that very well. The lack of logic in this article is in stark contrast with Stevens' writing.

Secondly, the article takes a (poor) swipe at C's casting. C was designed to be a very low level systems implementation language, the kind of language you need to access individual bits in device hardware registers when writing an operating system. It achieved its goals admirably, partly as a result of aliasing evils such as casting and unions which gave it much flexibility. Such features are dangerous in the wrong hands, but they are a valid part of the low-level O/S programmer's toolkit. Only experts should use them.

And finally, the article tries to make some sort of argument in which "Unix == bad, Nginx == good", which is more hysterical than reasoned because Nginx uses the most efficient event mechanism available on each O/S. This is also true when running on Linux or any Unix, so the article is making no sense at all. What's more, Nginx still uses the native TCP/IP stack, and gains its efficiency by coupling a good event mechanism with an avoidance of unnecessary context switching and good caching. So what exactly is the point of the article? None that I can see, at least none that has been described logically and factually.

To try to salvage something from what is a rather empty diatribe, perhaps one could validly suggest that Unix systems could do with some more advanced networking frameworks so that applications can more easily structure themselves along the lines of Nginx instead of having to architect themselves with insight from scratch. (Think of the OTP framework in Erlang which provides directly implemented design patterns called "behaviors" that applications can use immediately to enjoy best-in-class designs). That would make such effective architectures more easily acquired by applications, regardless of the language in which they happen to be written.

There is always more to do, and there are always better ways to do it. Poorly reasoned rants and attacks on books/authors that provide background reading and education do not help at all.

Morgaine.

danskan said...
This comment has been removed by the author.
Anonymous said...

I have no idea what you were talking about in your post, so I guess I need to just take it one paragraph at a time and start learning. LOL