Wednesday, July 27, 2011

The ethical problems of the CISSP and (ISC)2

This article from Attrition.org and InfoSecIsland.com is a good discussion about the ethical problems of CISSP/(ISC)². I thought I'd add my own 2 cents, since the ethics problem with the CISSP certification are pretty grave.

The article above only briefly mentions the more common complaint about the (ISC)², that they are technically incompetent, and thus not qualified to certify people. I would suggest that the two are intertwined: the cause of (ISC)²'s poor ethics is the need to cover up their lack of technical competence.

Other certifications, such as passing the bar to become a lawyer, publish past tests, or otherwise provide ways to judge their competence. In that way, those in the profession can judge for themselves whether passing the bar is an adequate measure of competence in the legal profession. If the bar itself proves incompetent, this will quickly become apparent in the published test questions.

No such accountability exists for (ISC)².There is no transparency in their tests. They keep the questions secret, which prevents outsiders from judging the quality of their tests. Moreover, their tests contain trial questions, which are not graded, but are being evaluated for official inclusion in future tests. When test takers point out obviously bogus questions, those are just dismissed as being trial questions.

This issue is doubly troublesome because of the roll "transparency" plays in cybersecurity. We would never trust an encryption algorithm that was not completely transparent. Yet, we are expected to trust a certification program shrouded in obscurity.

Like other professions, one of the chief concerns of professional ethics is protecting the profession. Whether you are a doctor, clergyman, lawyer, or journalist, you are expected to keep confidential information secret. This protects the profession: your patients won't confide their medical secrets with you if you blab about them, preventing you from being able to treat the patient. Your flock won't confess to you if you blab about it on your next sermon. Your client won't confide their legal secrets to you if you then confess them in court. Anonymous sources won't give you juicy stories if you reveal your source.

Curiously, though, that item isn't among the CISSP ethics. I guess it's implied ("respect their trust"), but there is nothing as explicit as in these other professions that says "you will not divulge the secrets your client confides in you". In particular, there is no guidance for real-world problems, such as "should I keep illegal activity by the customer secret?".

But, there is a lot of things in the CISSP ethics that aren't in those other professions. For example, according to the (ISC)², you should not professionally associate with or recognize criminals, amateurs, or the non-certified. This changes cybersecurity into a corrupt cartel, who shuns those who do not pay into the cartel. It's a lot like the licensing boards in states, where professionals have lobbied the government to enact barriers to competition, all in the name of "certification".

The model for certification in our industry shouldn't be the closed, obscure processes of the corrupt and incompetent, but the open processes we see in academia. The competency of physicists is judged by the papers they write. Of course, if you are new to cyberscurity, you aren't going to have a track record of published papers, but you will have blog posts, or at least posts to sites like http://security.stackexchange.com.

An example of this is the SANS GIAC certification. I know little about it, maybe it has even worse problems than (ISC)²'s CISSP. But, I can judge it in an open source way. Take the board of directors. The (ISC)² board is a bunch of people I've never heard of before, but I know everyone on the SANS board, not personally, but through the work they've published. Likewise, as part of certification, people publish papers on a narrow topic. My search results of cybersecurity topics is littered with GIAC papers. Finally, and most important, the professionals who carry GIAC are usually competent, whereas CISSP certified professionals rarely are. I can't find a record of past exams, but GIAC sample test questions display competence, whereas the sample CISSP test questions do not.

To be fair, the academics on the SANS board are different than industry professionals on ISC(2) board. But here's the thing: being a corporate executive only validates your leadership (or political) skills, not your technical skills. Like law and medicine, our field has a heavy academic component. You wouldn't put hospital administrators in charge of evaluating the skills of doctors, nor should you put managers in charge of certifying cybersec professionals.

The SANS ethics contain a lot less nonsense than the CISSP ethics. They are clearly about protecting the reputation of professionals, such as explicitly saying "don't divulge secrets", and there's none of the corrupt cartel-supporting ethics of the CISSP like "valuing the certificate". The SANS list provides guidance for the ethical questions cybersec professionals actually face, whereas the CISSP ethics appears to be written by somebody who has never experienced real world ethical questions.

Of particular interest is this one case where they disagree on ethics:
CISSP ethics: "Discourage unsafe practice" (translation: be more a security advocate)
SANS ethics: "I distinguish between advocacy and engineering" (translation: be less a security advocate)

The CISSP ethic is wrong. Outsiders distrust cybersec professionals because of our tireless advocacy against everything that is deemed unsafe. The reality is that cybersecurity is a tradeoff between costs and benefits. Our job is to accurately and dispassionately communicate the risks, but recognize that customers may choose risk over costs. This unrelenting advocacy is corrosive to our industry, and is the biggest reason for lack of trust in our profession, not the other ethical reasons (like divulging secrets) that one expects.

Now, I don't like all the items on the SANS ethics list. The last item forbidding "discrimination" is pointless. Yes, I suppose it's a good thing, but you could also include "be nice to children" and "don't kick puppies". They don't relate to cybersec. However, this nonsense just reflects the general stupidity of political correctness rather than calling into question their technical competency.

I could endlessly dump on the nonsense that is CISSP, but I leave you with one last ethical problem. Their ethics discourages "Professional recognition of or association with amateurs". The (ISC)² is clearly a bunch of amateurs. Therefore, by definition, you should neither recognize the CISSP certificate nor associate with (ISC)² people. I know this is a snide comment, but it reflects the real ethical quandary of whether it's ethical to recognize a certification that we know certifies the unqualified.

10 comments:

StephenRayNorthcutt said...

After competing ( in some sense ) with ISC2 for 11 years, I wish I could jump for joy with this post. I can't. GIAC will implement non-scoring questions by the end of the year. It is the standard for high stakes testing. The SAT, lSAT and every other certified testing agency does it in order to ensure the questions that count perform properly. We tried to find another way to do it, but this seems to be the standard.

Respectfully,

Stephen Northcutt, Outgoing Chairman of the GIAC Board of Directors ( 3 more weeks )

Robert Graham said...

Non-scoring questions isn't the problem. Accountability is the problem. I could equally have said "they hide behind the secrecy of the questions" -- all tests have secret questions, that's how they work, but the CISSP uses the secret questions to obscure their competency problems.

Here's why I brought GIAC into the post: I was googling for the "CISSP grievance" process, to find out how, precisely, I could challenge a test question I didn't like. I could find no results for CISSP, but all the results I did find pointed to the GIAC grievance procedures.

All tests have problems. The test of a test is if you are accountable to those problems.

infosecman said...

To some degree I agree with you on this post. But look at Microsoft certifications and even Cisco and the rest. Most of these certification lost their value due to the fact that their question are floating all over the internet that it wouldn't take a grade 3 student to memorize the question and go pass the exams. If ISC2 should make their questions available, wouldn't you think the value of the certification will be like those mentioned above? Isn't this the transparency you're talking about? The context in which you compare encryption algorithm to certification is not right.
Comparing ISC2 certifications to GIAC clearly shows that you're being bias. We all fail exams and admittedly I have failed once and passed on my second attempt. I can see the fustration many people go through upon failing an exam. I've been there and even emailed ISC2 to complain. If it's the cost involved just as written in other articles then check GIAC certifications and compare the two. I've met Cisco CCIE certified inviduals that lack *real* skills. But how much does their practical exams cost? I will challenge you to sit for CISSP and tell me it's was like sitting for certifications like the *others*. Be careful on your comparison of security certification with that of Doctors etc. Should those organization put their pool of questions in the public domain, wouldn't you be a Doctor too? Yes, there are *fake* security professionals with numerous certifications just as we have *fake* professionals in other disciplines. What do you want ISC2 to do - publish their questions in the public domain or make the exams easy so everyone pass? It's the industry as many lack the passion to study new skills etc. One will argue that most Pentester can't even code tools etc. My opinion, if they understand the technology that underpins the tools and can practically use the tools to achieve thier purpose whilst delivering satisfactory services why should they code their own tools? Here, Core Impact, Metasploit Pro etc. Do we need anymore tools? How many of these tools do we even use practically without having to certify it to comply with regulations etc. You see my point...? I'd have given 100% on this good post if you'd address the issue of individuals with various certifications from ISC2, GIAC, EC-Council etc who do not know their stuff. That I'll sing you praises. Until then you only earned 70%. All the best in your fight!

Robert Graham said...

I think you missed the point.

The issue is competency, and their exploitation of secrecy in order to avoid accountability.

There are two doctors. One was certified by other doctors. The other was certified by hospital administrators. Pick which one you want to operate on you.

a. said...

During the CISSP exam, I was given a "comment form" and advise that I could use this to comment on or communicate problems with a questions. I don't remember very clearly, but I think I used at least three of them. One to note that they forgot the word "not" when translating one question from English to German (I used both version of the questionnaire, just for fun). Only problem was that one had to fill in a ton of information on the top of the comment form, which took quite some time. I left the exam with a feeling that the questions mostly made kind of sense, and that I tried to point out those, which were not.

Later I did the ISACA CISM exam. There was no method of feedback available. And there were not few questions which felt very "fluffy", with more than one "correct" answer. And don't get me started on asking how the CISM exam scoring works (200 points minimum score?!?).

Please don't get me wrong. I am not trying to defend the CISSP exam or certifications in general.

But regarding the CISSP questions: I think there are others out there, doing it worse.

Regarding the recognition of amateurs: This section has a disclaimer of being informational only ("the professional is not required or expected to
agree with [it]"). Nevertheless, I think it is doing more harm than good and should be removed.

Regards,

a.

P.S.: I did not see you at the "CISSP Certificate Burning" panel this weekend, though you were listed as speaker. Or do you wear a long white beard these days?

Nunya Bidness said...

I appreciated this article on the CISSP/ISC2. I personally held my CISSP for over 7 years but did so thinking it was the most absurd standard and certification I have ever held.

I ended my relationship with ISC2 after they "lost" my information for the second time and I did not get my recert notice until I happen to look up at my certificate and realize it was out of date. I say second time, because the first time I re-certified, they did the same thing - lost my information (had my old information from my original application). At that time (the first time) they admitted that their entire certification management was being done in excel spreadsheets.

Everything about the CISSP and ISC2 is dubious. The questions/answers they use to certify are antiquated, if not in some cases wrong and possibly dangerous. The process they use, as discussed in this article, is not transparent and borderlines on unethical. The infrastructure they use to manage certifications, and the people who manage them have lied and do not live up to their own mistakes.

I have distanced myself fully from this certification and this organization.

As for the background I have to make such statements - I have lived all aspects of security my entire adult life. I spent over a decade in special operations, was a former police officer, and went on to a very deep technical security career.

Jamy said...

Another blatant issue I have seen is the endorsement process to become a CISSP. I have personally been asked by several people that have passed the exam to endorse them, I have refused two of them as I did not see the four years of experience in IT Security required by ISC2, I told these individuals to apply for the associate of ISC2 program. In one case the person was given the full CISSP after ISC2 reviewed the same information. In the second case the person found another CISSP that was willing to endorse them even though they could not show the proper experience level.

This illustrates that ISC2 is willing to sacrifice their requirements and rubber stamp anyone to get test fees and annual dues. It also perpetuates keeping unethical CISSP's in the field by allowing them to continue to endorse those who are unqualified. This further illustrates your conclusion that ISC2 themselves are not qualified.

Anonymous said...

I have held cryptology, cryptography and Information security roles in the military, federal and contractor industries since 1996 and suddenly I became unqualified overnight in 2006 because I did not have various newly required certifications. I managed to get approval for the government to pay for my Security+ certification and now I am required to complete the CISSP certification. The government created certification requirements for access to their networks so the government should oversee the ISC2 certification process to make sure it is fair, accurate and reliable. Once escalating cyber wars revel how worthless these so called CISSP experts are the government will most likely come up with new certification requirements which will invalidate the entire CISSP industry.

Anonymous said...

I have held cryptology, cryptography and Information security roles in the military, federal and contractor industries since 1996 and suddenly I became unqualified overnight in 2006 because I did not have various newly required certifications. I managed to get approval for the government to pay for my Security+ certification and now I am required to complete the CISSP certification. The government created certification requirements for access to their networks so the government should oversee the ISC2 certification process to make sure it is fair, accurate and reliable. Once escalating cyber wars revel how worthless these so called CISSP experts are the government will most likely come up with new certification requirements which will invalidate the entire CISSP industry.

Kristian Erik Hermansen said...

Excellent post! I proposed a talk for cansecwest a while back with Josh Abraham (Jabra) regarding certification organization deficiencies. It included ISC2, eccouncil, and IACRB. The talk as going to feature a nice twist, which was how the test questions had become compromised and leaked due to security issues at the organizations. Being unable to secure such information properly should have been enough to cast doubt on all these organization, of which I am a member. Sadly, Dragos and the board did not accept our talk stating that such issues were already well known!
--
http://www.linkedin.com/in/kristianhermansen