Saturday, February 07, 2009

The Importance of Being Canonical

Wikipedia defines "canonical" as "reduced to the simplest and most significant form possible without loss of generality. It is "often used when describing … those books that all educated people have supposedly read, or are advised to read".

I would advise you to read the play-by-play action of the guy who hacked at It is a simple tale of the specifics of what he did in that attack, yet it is a good representation of the sorts of things that all hackers go through when breaking into a website.

The description starts with the standard "script-kiddy" formula. The hacker keeps up-to-date with the latest exploits from sites like milw0rm. Specifically, a package with the name "phplist" has a newly announced vulnerability, he remembers that "" uses that package, so he tries out the new exploit and find it works.

A typical script-kiddy would, at that point, simply overwrite the homepage and move on. This hacker, however, goes deeper.

These days, behind every website is a database. It's the database that's the prize for hackers. Hackers are looking for e-mail addresses (for spam/phishing), passwords, and credit-card numbers. The '' site has no credit card numbers, but it has 400,000 e-mail addresses, which is worth a fair amount of money on the underground market.

One protection against this is to encrypt passwords. This is done with a "one-way hash" against the password, he hash is stored in the database and the original password discarded. This allows the server to check whether the user enters the correct password when logging in, but does not store the password in a way that hackers can easily read. Thus, the phpbb hacker was able to get password hashes, but not the original passwords themselves.

Hackers can get around this with "rainbow tables". This is a method of creating a big database of password-to-hash combinations, so that you can simply look up the hash in the database to find the original password that generated it. There is more to it than that: rainbow tables compress the database hugely, so such lookups can take a while.

A lot of websites in the hacker underground provide rainbow cracking services. You submit files of hashes to them, they run the database lookups in the background, then return any passwords they find. One example is milw0rm's "cracker. All the ones I know about only let people submit a few hashes at a time, this hacker found one that allowed him to submit 28k hashes.

This is an obvious weakness, so the phpbb software has been upgraded to "salt" password hashes. A salt is a set of random characters added to a password before it's hashed. Then, the salt+hash is stored in the database. Let's assume that my password is "trustno1". The system then adds a 'salt', making the password '78dh37trustno1". It remembers the salt, so whenever I log back in, it automatically adds '78dh37' to my password to see if the hash matches.

This breaks rainbow tables. Combining the salt with the password makes it too big to precompute into a database. Now, when hackers steal passwords, the only option is running password cracking software that makes millions of guesses a second by combining dictionary words with the salt to see if any come up with the same result. They also might try brute-force attacks of trying every possible combination.

Rainbow tables, dictionary attacks, and brute-force attacks can be defeated by choosing a sufficiently long and complex password. One of the weaknesses of phpbb is that it doesn't require users to select complex/harder-to-hack passwords, but instead, let's them choose anything, including "password". As a result, few of their users have any passwords of adequate complexity.

These are the most popular passwords of phpbb users, which is roughly the same passwords you'll find on any system:
3.03% "123456"
2.13% "password"
1.45% "phpbb"
0.91% "qwerty"
0.82% "12345"
0.59% "12345678"
0.58% "letmein"
0.53% "1234"
0.50% "test"
0.43% "123"
0.36% "trustno1"
0.33% "dragon"

Except for system administrators, hackers aren't really concerned with any particular user. They do the simplest and fastest cracking to get as many passwords as possible. They aren't going to spend two weeks brute-forcing your password if they can instead spend one day each on fourteen other passwords.

It appears that upgraded to the newer salted scheme a while back. Anybody who created an account or changed their password since that date has the newer, salted hashes. All the old accounts still have the unsalted passwords, which are the ones the hacker cracked. Presumably, the hacker can also get the newer passwords by using something like John the Ripper. Curiously, Solar Designer wrote both John the Ripper AND the new password hashing scheme for phpbb.

There is not a lot that can do to mitigate such attacks. They are "free" software, so they don't have a lot of money to spend dealing with security threats. The problem here was that they used a free "phplist" software, and the patch came out at the same time as the exploits. They didn't have enough manpower to get the patch installed before they were hacked.

On the other hand, there are some strategies they could use.

The first is to create "canary" accounts. Create accounts that have e-mail addresses, like "". This account is not going to get any spam e-mail. When it does get its first spam, you'll know that it came from your database. When I create recommendations for clients, this is always one of the first things I suggest. (Likewise, if you are an e-commerce site, you should get dummy credit cards that only exist in your database). This won't stop you from getting hacked, but it will at least tell you when a hack has happened. (I suspect that this isn't the first time phpbb has been hacked - just the first time it's been made public).

The second thing is to fix the password problem. They should force users to create more complex passwords when they log in. A six-character password consisting of numbers, upper/lower case, and symbols can be brute-force cracked in about a half-hour on a modern computer. I would suggest at least 8 character passwords, containing at least one symbol.

They should upgrade the unsalted hashes to newer salted hashes. Remember that since they don't know the original password, they can't in theory upgrade them. One way around this is to simply run a cracking tool, like this hacker did. Extract the password from the database, do the rainbow crack on it, and if it's found, hash it the new way and put it back in the database. For the remainder of the passwords, add something to the login script: when the user logs in, you have their plaintext password for a few moments, so you can upgrade it right then. My guess is that 90% of user's passwords can quickly be upgraded this way.

Lastly, I would re-architect their software to put the details hackers want (e-mail, password hashes, credit cards) within a different administrative domain. Put that information on a different server. Harden that server so only a single port is open. Make an API that allows you to do things like check passwords, but not ever read passwords. Thus, when a hacker gets into their front-end, they won't be able to compromise that backend accounts machine. It's the same principle as "shadow passwords". The /etc/passwd file contains all the normal account information, but the "shadow" password file contains the account information and cannot be read by normal users.

Update: PHBB3 already automatically upgrades passwords to the new hash on login (I assumed it only upgraded users when they changed passwords). A reader named "Erik" suggests something even smarter for the old passwords: to re-hash the hash using a salt, ie. MD5(salt+MD5(password)) or BLOWFISH(salt+MD5(password)). This works because while they don't know the original password, they do know MD5(password).


Matt Weir said...

One thing I want to point out is that the plaintext passwords the hacker posted, (about 28 thousand of them), are only the passwords that he was able to crack. There were significantly more in the MD5 database that were strong enough that he wasn't able to break.

I read your article on darkreading which implied that the passwords used on phpBB were weaker than the myspace ones. In reality it would be hard to compare the two lists until a majority of the phpBB ones are cracked. I will admit though that without a password creation policy I would expect the phpbb ones to be easier.

All that being said, I fully agree with your recommendations and conclusions.

Kellanved said...

Just a minor thing: was hacked two weeks before a patch became available.

Good rundown otherwise, especially about the reason for the passwords being as they are.

Jessica said...

The problem here was that they used a free "phplist" software, and the patch came out at the same time as the exploits. They didn't have enough manpower to get the patch installed before they were hacked.

This is simply untrue. The milworm exploit was out on the 14th Jan, while phplist didn't put out a patch until the 29th Jan. Manpower or not, no one can apply a patch that does not exist.

Robert Graham said...

The milworm exploit was out on the 14th Jan, while phplist didn't put out a patch until the 29th Jan. Manpower or not, no one can apply a patch that does not exist.

This is a good point, but wrong.

Organizations that spend a lot on security subscribe to forums like milw0rm. As soon as they see something appear on milw0rm, full-disclosure, etc. that affects them, they fix their software EVEN IF NO OFFICIAL PATCH EXISTS. If given no other choice, at least you can disable the offending software. In the case of phplist, it was an easy fix the customer can do themselves. Indeed, that's the point of open-source: you don't have to wait for the vendor to release a patch, you can simply change the source yourself.

Jessica said...

The milw0rm post has no suggested fix. Patching software without an official patch normally requires regression testing and the like to make sure your 'fix' not only works against the original problem, but also doesn't miss out any edge cases of the original problem and doesn't introduce security problems of its own.

Your original paragraph implied that there was a patch to be applied but no one did in time, which is my objection.

It's very easy for people to miss things. After all, everyone's human. To insist that phpBB was at fault for not fixing the issue is a stretch at best.

I'd prefer to say the real problem is the fact that phplist emulates register globals. There's a damn good reason why it's being removed.

Robert Graham said...

To insist that phpBB was at fault for not fixing the issue is a stretch at best.

I said nothing implying phpBB was at "fault" for anything. This issue of "fault" is something you read into the post that wasn't there. This insistence of finding "fault" is a disease in the cybersecurity community that disgusts me, and hinders rational discussion of cybersecurity.

Instead, I'm trying to figure out what I could suggest to them to help them mitigate such threats in the future. Normally, this recommendation would be to keep up with Bugtraq, Full-Disclosure, Milw0rm, and similar forums. This exploit was on all three forums on Jan. 14. Moreover, this exploit shows how to patch the vuln (filter the _SERVER variable in the url somehow).

However, I doubt a site like phpBB has the manpower to either keep up with the forums, or to quickly implement mitigation strategies while keeping the system up and running. I'm not saying they are at fault for the lack of manpower, I'm saying the reality of the situation is that as a free site, they can't get the manpower, and that they have to look for other strategies that don't require manpower. I suggest a few things that can be done without spending money or manpower. For example, canary, accounts, are a cheap and easy intrusion detection system that doesn't require manpower to operate.

Jessica said...

An even better suggestion would be IDS/firewall options; like mod_security, suphp to use separate users for each application the server runs, proper db permissions to segregate applications, and even the use of Suhosin.

Canary accounts would only tell you that you've been hacked, while the above would prevent it (mod_security and Suhosin alone in this instance would've sufficed to prevent the attack between them).

Manpower doesn't really factor in to it. There's at least 50 phpBB team members, so if you think between them they can't sort out a server due to manpower issues, then you're overlooking more fundamental problems...

Erik said...

Upgrading to salted passwords by running cracking tool doesn't sound like very ethical. Besides, complex passwords will most likely not be regenerated with rainbow tables.

What I would have done is to take the non-salted hashes and simply hash them again, but with a salt this time. I.e what will be stored in the database would be something like:


Robert Graham said...


Smart idea. I'm jealous I didn't think of that.

website design nyc said...

nice post