DEF CON drink-off -- for science!

The DEF CON hacking conference is a mixture of techies and drinkers. I propose we exploit this for science. Specifically, we should take a look at vodka. Vodka is just ethanol and water with all taste removed by distillation and filtering. We can answer two important questions.

1. Poorly made, cheap vodka lets too much of the (bad) flavor through. Can this be improved by running it through a filter? (Such as a cheap Brita water filter).
2. Well-made vodka should be indistinguishable from each other. Can people really taste the difference? Or are they influenced by brands?

We need to science the shit out of these questions with a double-blind taste test. DEF CON is a perfect venue for getting a statistically relevant number of samples. We should setup a table in a high-traffic area. We'll ask passersby to taste a flight of several vodkas and to rate them.

I suggest the following as the set of vodkas to test.

1. Smirnoff, by far the market leading vodka in America, a "mid-shelf" vodka at $22 for a 1.75 liter bottle.
2. Grey Goose, the third most popular vodka in America, a "top-shelf" vodka for $58 a 1.75 liter bottle.
3. A randomly chosen "bottom shelf" vodka, chosen at the local liquor store, for the cheapest price (around $10 for a 1.75 liter bottle).
4. That same "bottom shelf" vodka, but this time filtered through a Brita system.
5. Costco vodka, which costs about $14 for a 1.75 liter bottle. Costco notoriously sells high-quality products for a low price.

These are all 80 proof (40% ethanol). I suggest half-shots (20 milliliters) in little paper cups, which is about 80 samples per 1.75 liter bottle. All five bottles would cost $114 combined. We'd want a large number of subjects, around 500, so it'd be about $700 worth of vodka. We couldn't possibly sell the vodka, but we could ask for donations, asking tasters to contribute $1.50 for the flight they've tasted. This would help defray the costs.

The test should be blind, so that the subjects have no idea which vodka is which. Each subject should taste the five in a different random order, so that ordering doesn't affect results. Furthermore, the test should be double-blind: one tester first decants the original vodka into numbered bottles, so that the other testers manning the table do not themselves know which is which, and therefore cannot influence the subject's judgement. We'll need to get a set of identical bottles, which means everyone will need to pitch in helping empty some bottles the night before.

Tasting should be done "neat" (not mixed with anything else) and at room temperature. Mixed ingredients kill the taste of vodka, making it harder to tell the difference. Chilled temperatures likewise kill the taste, so the warmer the better.

---

Alternate protocol

There are a number of alternate experiments we can run. I chose the above example for ease-of-testing. But, as @paulm has pointed out, it'd probably be better just to test two vodkas at a time, side-by-side.

In this alternate experiment, we'd have the same 5 vodkas to start from. We'd pick one out of random of the five, and give it to the subject. We'd then pic a second at random from all give, and give it to the subject. We'd then ask the subject to determine which is better, or if they taste the same. (In one-in-five cases, they actually will be the same).

The biggest problem is logistics. It'll take about 5 minutes per person, or 12 people per hour, or 80 people per day. I'm guessing it would be roughly the same amount of time per subject whether they take 2 samples or 5.

Another problem is number of bottles. You need two groups of five to choose from, so that they subject can't tell when they've been given two of the same vodka.

The biggest problem is statistics. Assuming we can get 500 samples, any particular combination only occurs 10 times. That makes answering narrow questions like "Is Grey Goose better than Smirnoff?" difficult. This can be solved by narrowing the choices, but I don't like that. There may be some weirdness about that particular combination that you wouldn't see given a different combination. If we are testing "Can rotgut be improved by filtering?" I'd rather have a variety of different vodkas to test against.