The Importance of Being Canonical

Wikipedia defines "canonical" as "reduced to the simplest and most significant form possible without loss of generality. It is "often used when describing … those books that all educated people have supposedly read, or are advised to read".

I would advise you to read the play-by-play action of the guy who hacked phpbb.com at http://hackedphpbb.blogspot.com. It is a simple tale of the specifics of what he did in that attack, yet it is a good representation of the sorts of things that all hackers go through when breaking into a website.

The description starts with the standard "script-kiddy" formula. The hacker keeps up-to-date with the latest exploits from sites like milw0rm. Specifically, a package with the name "phplist" has a newly announced vulnerability, he remembers that "phpbb.com" uses that package, so he tries out the new exploit and find it works.

A typical script-kiddy would, at that point, simply overwrite the homepage and move on. This hacker, however, goes deeper.

These days, behind every website is a database. It's the database that's the prize for hackers. Hackers are looking for e-mail addresses (for spam/phishing), passwords, and credit-card numbers. The 'phpbb.com' site has no credit card numbers, but it has 400,000 e-mail addresses, which is worth a fair amount of money on the underground market.

One protection against this is to encrypt passwords. This is done with a "one-way hash" against the password, he hash is stored in the database and the original password discarded. This allows the server to check whether the user enters the correct password when logging in, but does not store the password in a way that hackers can easily read. Thus, the phpbb hacker was able to get password hashes, but not the original passwords themselves.

Hackers can get around this with "rainbow tables". This is a method of creating a big database of password-to-hash combinations, so that you can simply look up the hash in the database to find the original password that generated it. There is more to it than that: rainbow tables compress the database hugely, so such lookups can take a while.

A lot of websites in the hacker underground provide rainbow cracking services. You submit files of hashes to them, they run the database lookups in the background, then return any passwords they find. One example is milw0rm's "cracker. All the ones I know about only let people submit a few hashes at a time, this hacker found one that allowed him to submit 28k hashes.

This is an obvious weakness, so the phpbb software has been upgraded to "salt" password hashes. A salt is a set of random characters added to a password before it's hashed. Then, the salt+hash is stored in the database. Let's assume that my password is "trustno1". The system then adds a 'salt', making the password '78dh37trustno1". It remembers the salt, so whenever I log back in, it automatically adds '78dh37' to my password to see if the hash matches.

This breaks rainbow tables. Combining the salt with the password makes it too big to precompute into a database. Now, when hackers steal passwords, the only option is running password cracking software that makes millions of guesses a second by combining dictionary words with the salt to see if any come up with the same result. They also might try brute-force attacks of trying every possible combination.

Rainbow tables, dictionary attacks, and brute-force attacks can be defeated by choosing a sufficiently long and complex password. One of the weaknesses of phpbb is that it doesn't require users to select complex/harder-to-hack passwords, but instead, let's them choose anything, including "password". As a result, few of their users have any passwords of adequate complexity.

These are the most popular passwords of phpbb users, which is roughly the same passwords you'll find on any system:
3.03% "123456"
2.13% "password"
1.45% "phpbb"
0.91% "qwerty"
0.82% "12345"
0.59% "12345678"
0.58% "letmein"
0.53% "1234"
0.50% "test"
0.43% "123"
0.36% "trustno1"
0.33% "dragon"

Except for system administrators, hackers aren't really concerned with any particular user. They do the simplest and fastest cracking to get as many passwords as possible. They aren't going to spend two weeks brute-forcing your password if they can instead spend one day each on fourteen other passwords.

It appears that phpbb.com upgraded to the newer salted scheme a while back. Anybody who created an account or changed their password since that date has the newer, salted hashes. All the old accounts still have the unsalted passwords, which are the ones the hacker cracked. Presumably, the hacker can also get the newer passwords by using something like John the Ripper. Curiously, Solar Designer wrote both John the Ripper AND the new password hashing scheme for phpbb.

There is not a lot that phpbb.com can do to mitigate such attacks. They are "free" software, so they don't have a lot of money to spend dealing with security threats. The problem here was that they used a free "phplist" software, and the patch came out at the same time as the exploits. They didn't have enough manpower to get the patch installed before they were hacked.

On the other hand, there are some strategies they could use.

The first is to create "canary" accounts. Create accounts that have e-mail addresses, like "something-really-long-xyz-123@gmail.com". This account is not going to get any spam e-mail. When it does get its first spam, you'll know that it came from your database. When I create recommendations for clients, this is always one of the first things I suggest. (Likewise, if you are an e-commerce site, you should get dummy credit cards that only exist in your database). This won't stop you from getting hacked, but it will at least tell you when a hack has happened. (I suspect that this isn't the first time phpbb has been hacked - just the first time it's been made public).

The second thing is to fix the password problem. They should force users to create more complex passwords when they log in. A six-character password consisting of numbers, upper/lower case, and symbols can be brute-force cracked in about a half-hour on a modern computer. I would suggest at least 8 character passwords, containing at least one symbol.

They should upgrade the unsalted hashes to newer salted hashes. Remember that since they don't know the original password, they can't in theory upgrade them. One way around this is to simply run a cracking tool, like this hacker did. Extract the password from the database, do the rainbow crack on it, and if it's found, hash it the new way and put it back in the database. For the remainder of the passwords, add something to the login script: when the user logs in, you have their plaintext password for a few moments, so you can upgrade it right then. My guess is that 90% of user's passwords can quickly be upgraded this way.

Lastly, I would re-architect their software to put the details hackers want (e-mail, password hashes, credit cards) within a different administrative domain. Put that information on a different server. Harden that server so only a single port is open. Make an API that allows you to do things like check passwords, but not ever read passwords. Thus, when a hacker gets into their front-end, they won't be able to compromise that backend accounts machine. It's the same principle as "shadow passwords". The /etc/passwd file contains all the normal account information, but the "shadow" password file contains the account information and cannot be read by normal users.

Update: PHBB3 already automatically upgrades passwords to the new hash on login (I assumed it only upgraded users when they changed passwords). A reader named "Erik" suggests something even smarter for the old passwords: to re-hash the hash using a salt, ie. MD5(salt+MD5(password)) or BLOWFISH(salt+MD5(password)). This works because while they don't know the original password, they do know MD5(password).