Why cybersecurity tests fail

The Christian Science Monitor, a newspaper, recently had an online quiz to test how well you know cybersecurity. This is a good demonstration of the sorts of problem all such tests (like the CISSP certification test) have.


The first question asks what term William Gibson coined in "Neuromancer”.

The correct answer is "none of the above”.

This is a sort of trick question. While Neuromancer is credited with popularizing the term cyberspace, Gibson "coined” the term in an earlier work.

But the answer given in the test isn’t correct, either. The term "Internet” was coined in the 1970s. By the time the "Internet Protocol” (the basis for today’s Internet) was specified in RFC791 (three years before Neuromancer), the term "Internet” was already in widespread use.

The cop-out from test designers is that when given a bad set of choices, you are supposed to choose the "best” one. Clearly the "best” answer is "cyberspace”, not "Internet”. At least, it’s clear to me, since I was on the Internet before Neuromancer was published.


Question #8 is probably the most amusing, because it demonstrates a lack of knowledge of the English language rather than cybersecurity:

If you parse the English language, the question asks "Who invaded Georgia?", not "Who cyber-attacked Georgia?". We don’t know who was responsible for the cyber-attacks against Georgia, but we do know that Russia invaded Georgia with ground troops.

There is also a confusion as to what "its” means: does it mean the invasion was preceded by attacks on Georgia’s computers, or on Russia’s computers?


The reason I point this out is not to beat up on the Christian Science Monitor, but to use this as an analogy. Certification tests (like the CISSP) are hardly better. They are written by generalists who know a little about everything, but aren't an expert in any one thing. They make themselves immune to criticism (the first rule of the CISSP is that you will not criticize the CISSP), so it’s hard to debate those questions openly.