But that's not what the question is asking. Instead, it's asking superfluous information about the default behavior, namely about Linux defaults. It's a trivia test, not a knowledge test. If you've recently studied the subject, your course book probably tells you that Linux traceroute defaults to UDP packets on transmit. So, those who study for the test will do well on the question.
But those with either a lot of deep knowledge or practical experience will find this question harder. Windows and Linux use different defaults (Windows uses ICMP ECHOs, Linux uses UDP). Personally, I'm not sure which is which (well, I am now, 'cause I looked it up, but I'm likely to forget it again soon, because it's a relatively unimportant detail).
Those with deep learning have another problem with the word "protocol". This question uses "protocol" in one sense, where only UDP, TCP, and ICMP are valid "protocols".
But the word can be used in another sense, where "Echo" and "TTL" are also valid "protocols". A protocol is a set of rules that govern things. Thus we say phrases like "slow start protocol" for how TCP handles initial congestion, even though this "protocol" has no protocol header or particular fields. In much the same way, TTL is a "protocol" or "set of rules" for handling routing loops that traceroute exploits. That Linux uses the TTL protocol when transmitting packets is a perfectly valid answer to this question, albeit not the conventional one.
Exams suck because those writing the exams themselves often lack experience and deep knowledge. They are only one short step ahead of their students.
This leaves such test prejudiced toward those who have recent read (and who are likely soon to forget) a textbook. The tests are prejudiced against those who the tests are intended to highlight, those with experience and deep knowledge.
I'm not really trying to beat up on the GIAC tests here. I'm simply demonstrating the problem in our industry. We want to be able to certify people like doctors and lawyers, real "professions" where if things go wrong, people's lives can be ruined. We are far from that. All certification tests are entry-level only. Our trade has not existed long enough to become a full trustworthy "profession".
Unfortunately, certifications are what employers are looking for. Most of them are oblivious to the fact that they are useless. Some of the ones that actually matter, they have never even heard of before. Thank you for putting together the article.
ReplyDeletehttps://www.scag.co.za/
The CREST CPSA exam was like a survey of 30 years of computer trivia.
ReplyDelete(Disclaimer: I am a current member of the GIAC Advisory Board (this is just the people who average 90% or more on GIAC exams)
ReplyDeleteBased on my experience so far (two tests), this question is probably not representative of the (relatively) straightforward questions that are sufficient to pass the test.
Instead, this is probably a "separate wheat from chaff" question - testing for something not directly related to the test's target competence. Getting this question right would help to reach the next threshold (the 90% required to join the GIAC Advisory Board). Most other certifications provide no additional benefit from going beyond just passing.
Put another way: Real life is also full of misleading, confusing, misclassified, and contradictory information. If you can get a good percentage of gotcha questions right, that may be an indicator of a related skill that is useful in the problem domain.
Regardless of the reasons for such a question, I don't agree with a professional certification being based on multiple choice questions. An exam based on open-ended questions are where you will really separate the "wheat from chaff". Candidates may have exceptionally good memories, but not a whole lot of sense with respect to application.
ReplyDeleteA well reasoned answer to an open-ended question will quickly determine who deserves the certification, and who does not.
I actually disagree with your assessment. Understanding how trace route works in this case definitely helps a pen tester perform better. For example; imagine you are performing a pen test and attempt a trace route to a windows device and a linux device on network, the trace route completes for linux, but not for windows. If you know how trace route works on both platforms you can determine traffic that is not allowed vs traffic that is. I personally have met many pentesters that are essentially script kitties, running pre-built scripts and tools without actually understanding how those tools work.
ReplyDeleteYou can argue the value of multiple choice vs. fill in the blank etc. However, if the individual doesn't actually understand how Linux trace route is working, they have at best a 25% chance of answering correctly.
No one has ever argued that multiple choice questions are superior to determining someones ability. The fact is they are the simplest method of testing that can be graded on a large scale, making them cost effective. Certifications in IT are decent for assessing a baseline of knowledge, in-person interviews are needed to asses the mastery of a subject to the hiring managers standards.
ReplyDeleteAs far as your analysis of this question, understanding the difference in which protocols are used is essential for someone who works either as an analyst, pentester, or even looks at the traffic in a network. They need to know what protocols are doing what and why. It's far from a pointless question. Also, in reference to your rant on the use of the word "protocol" (since you want to get in to semantics), the question asks which protocol is used to transmit traceroute, not which ones are involved in the process. Attention to detail, understanding the question, and the topic are required.
I agree with your sentiment. Any IT certifications is no assurance of professional competence, I also agree they will never be able to provide the kind of certainly that titles in more mature professions such as medicine, or mechanical engineering. Part of this is lack of maturity of our profession, but part of it lies in the fact that the laws that govern "cyber" space are made up by humans, generally fungible, and subject to change based on the whims of designers. The laws that govern biology and physics don't really change (although our understanding of them may).
ReplyDeleteI disagree with the way your extrapolating this based on the semantics of single sample question. I hold several GIAC certifications including the GPEN. My understanding is that none of the sample questions or practice test questions actually count towards toward the exam score. They are provided by GIAC to allow a potential test taker to assess their preparedness for the exam, which in the case of the open book GIAC exams is most critically- prior access to the associated ~$5000 SANS course material. I'm not GIAC fanboy - Overall I've gotten a lot of value out of SANS security training, although I find the requirements of certain GIAC certification under DoDD 8570 to be questionable considering the amount of taxpayer money being directed at SANS because of the lack of viable training alternatives for GIAC preparation. Yes they are testing to the book, there is a huge revenue stream to be had making sure that exam takers take the prerequisite SANS courses even if they are already a subject matter experts.
Despite these feelings: I've interviewed a number of candidates and certifications do give the interviewer a window into what topics the candidate has been exposed to. A GPEN is an ok for an entry level pen tester, OSCP or GXPN is something I might look for in a more experienced candidate, if I were looking for a rock star I'd ask for a git-hub or other portfolio of prior work. This isn't much difference than another profession in that an experienced surgeon with a proven track record commands more value then someone strait out of med school. After hiring a candidate the SANS training curriculum is a powerful tool for filling in gaps in the candidates prior experience. Certifications clearly have limitations, but your statement that "All certification tests are entry-level only" is either driven by an inflated sense of what it means to be a non-entry level candidate, or ignorance to the managerial challenges of maintaining a competent team with less then an unlimited budget. It's not perfect but I still reject kings, presidents, and voting... Like it or not certification bodies like GIAC, E-Council, Offsec represent rough consensus and running code.
I think example this perfectly sums up certification. Someone who has no knowledge or experience can cram and get a high enough score to pass themselves off as knowledgeable in the area they've tested on.
ReplyDeleteI'm anti-certifications. In a previous position, a well-respected co-worker almost had me convinced to get my CISSP. Then a new InfoSec Manager started. He was so horrible (my peers were all in agreement), I wrote a script to mine the cert site to verify he even possessed it.
Certifications say to me "These people have similar qualifications". Maybe I have had more bad experiences than most, but the majority of the time I just don't want to be considered similar to "those people".
Great post and the write up is interesting. I love the exam question considering that ttl and echo are not "protocols" but options/knobs used in a protocol. Most old school engineers know the UDP vs. ICMP use for Traceroute from OS packet traces of traceroute. Actually for a bit of "trivia" that is how early security os detection freeware utilities determined the difference between a windows and linux OS. They used the difference as part of the OS's signature.
ReplyDelete