Friday, October 16, 2020

No, that's not how warrantee expiration works

The NYPost Hunter Biden story has triggered a lot of sleuths obsessing on technical details trying to prove it's a hoax. So far, these claims are wrong. The story is certainly bad journalism aiming to misinform readers, but it has not yet been shown to be a hoax.

In this post, we look at claim the timelines don't match up with the manufacturing dates of the drives. Sleuths claim to prove the drives were manufactured after the events in question, based on serial numbers.

What this post will show is that the theory is wrong. Manufacturers pad warrantee periods. Thus, you can't assume a date of manufacture based upon the end of a warrantee period.


The story starts with Hunter Biden (or associates) dropping off a laptop at a repair shop because of water damage. The repair shop made a copy of the laptop's hard drive, stored on an external drive. Later, the FBI swooped in and confiscated both the laptop and that external drive.

The serial numbers of both devices are listed in the subpoena published by the NYPost:


You can enter these serial numbers in the support pages at Apple (FVFXC2MMHV29) and Western Digital (WX21A19ATFF3) to discover precisely what hardware this is, and when the warrantee periods expire -- and presumably, when they started.

In the case of that external drive, the 3-year warrantee expires May 17, 2022 -- meaning the drive was manufactured on May 17, 2019 (or so they claim). This is a full month after the claimed date of April 12, 2019, when the laptop was dropped off at the repair shop.


There are lots of explanations for this. One of which is that the drive subpoenaed by the government (on Dec 9, 2019) was a copy of the original drive.

But a simpler explanation is this: warrant periods are padded by the manufacturer by several months. In other words, if the warrantee ends May 17, it means the drive was probably manufactured in February.

I can prove this. Coincidentally, I purchased a Western Digital drive a few days ago. If we used the same logic as above to work backward from warrantee expiration, then it means the drive was manufactured 7 days in the future.

Here is a screenshot from Amazon.com showing I purchased the drive Oct 12.


Here is a picture of the drive itself, from which you can read the serial number:


The Date of Manufacture (DOM) is printed right on the device as July 31, 2020.

But let's see what Western Digital reports as the end of warrantee period:


We can see that the warrantee ends on Oct 25, 2025. According to Amazon where I purchased the drive, the warrantee period is 5 years:


Thus, if we were to insist on working back from the expiration date precisely 5 years, then that means this drive was manufactured 7 days in the future. Today's date is Oct 16, the warrantee starts Oct 23. 

The reality is that Western Digital has no idea when the drive arrives, and hence when I (as the consumer) expect the warrantee period to start. Thus, they pad the period by a few months to account for how long they expect the device to be in the sales channel, the period between manufacture and when they are likely to arrive at the customer. Computer devices rapidly depreciate so are unlikely to be in the channel more than a few months.

Thus, instead of proving the timeline wrong, the serial number and warrantee expiration shows the timeline right. This is exactly the sort of thing you'd expect if the repair shop recovered the files onto a new external drive.

Another issue in the thread is about the "recovery" of files, which the author claims is improbable. In Apple's latest MacBooks, if the motherboard is damaged, then it's impractical to recover the data from the drive. These days, in the year 2020, the SSD drive inside notebooks are soldered right on the motherboard, and besides, encrypted with a TPM chip on the motherboard.

But here we are talking about a 2017 MacBook Pro which apparently had a removeable SSD. Other notebooks by Apple have had special connectors for reading SSDs from dead motherboards. Thus, recovery of files for notebooks of that era is not as impossible as a it sounds.

Moreover, maybe the repair shop fixed the notebook. "Water damage" varies in extent. It may have been possible to repair the damage and boot the device, at least in some sort of recovery mode.


Conclusion

Grabbing serial numbers and looking them is exactly what hackers should be doing in stories like this. Challenging the narrative is great -- especially with regards to the NYPost story, which is clearly bad journalism.

On the other hand, it goes both ways. We should be even more concerned about challenging those things that agree with us. This is a great example -- it appears we've found conclusive evidence that the NYPost story was a hoax. We need to carefully challenge that, too.


No, font errors mean nothing in that NYPost article

The NYPost has an article on Hunter Biden emails. Critics claim that these don't look like emails, and that there are errors with the fonts, thus showing they are forgeries. This is false. This is how Apple's "Mail" app prints emails to a PDF file. The font errors are due to viewing PDF files within a web browser -- you don't see them in a PDF app.

In this blogpost, I prove this.

I'm going to do this by creating forged email. The point isn't to prove the email wasn't forged, it could easily have been -- the NYPost didn't do due diligence to prove they weren't forged. The point is simply that that these inexplicable problems aren't evidence of forgery. All emails printed by the Mail app to a PDF, then displayed with Scribd, will look the same way.

To start with, we are going to create a simple text file on the computer called "erratarob-conspire.eml". That's what email messages are at the core -- text files. I use Apple's "TextEdit" app on my MacBook to create the file.

The structure of an email is simple. It has a block of "metadata" consisting of fields separated by a colon ":" character. This block ends with a blank line, after which we have the contents of the email.

Clicking on the file launches Apple's "Mail" app. It opens the email and renders it on the screen like this:
Notice how the "Mail" app has reformatted the metadata. In addition to displaying the email, it's making it simple to click on the names to add them to your address book. That's why there is a (VP) to the right on the screen -- it creates a placeholder icon for every account in your address book. I note this because in my version of Mail, the (VP) doesn't get printed to the PDF, but it does appear in the PDF on the NYPost site. I assume this is because their Mail app is 3 years older than mine.

One thing I can do with emails is to save them as a PDF document.

This creates a PDF file on the disk that we can view like any other PDF file. Note that yet again, the app has reformatted the metadata, different from both how it displayed it on the screen and how it appears in the original email text.

Sometimes web pages, such as this one, wants to display the PDF within the web page. The Scribd website can be used for this purpose, causing PDFs to appear like below:

Erratarob Conspire by asdfasdf




How this shows up on your screen will change depending on a lot of factors. For most people, though, they'll see slight font problems, especially in the name "Hunter Biden". Below is a screenshot of how it appears in my browser. You can clearly see how the 'n' and 't' characters crowd each other in the name "Hunter".


Again, while this is a fake email message, any real email message would show the same problems. It's a consequence of the process of generating a PDF and using Scribd. You can just click through on Scribd to download the original PDF (either mine or the one on the NYPost site), and then use your favorite PDF viewing app. This gets rid of Scribd's rendering errors.

Others have claimed that this isn't how email works, that email clients always show brackets around email message, using the < and > characters. Usually, yes, but not in all cases. Here, Apple's "Mail" app is clearly doing a lot more work to make things look pretty, not showing them.

There are some slight difference between what my 2020 MacBook produces and what the original NYPost article shows. As we can see from the metadata on their PDF, it was produced by a 2017 MacBook. My reproduction isn't exact, but it's pretty darn close that we don't need to doubt it.

We would just apply Occam's Razor here. Let's assume that the emails were forged. Then the easiest way would be to create a text document like I've shown above and open it in an email client to print out the message. It took me less than a minute, including carefully typing an unfamiliar Russian name. The hardest way would be to use Photoshop or some other technique to manipulate pixels, causing those font errors. Therefore, if you see font problems, the most likely explanation is simply "something I don't understand" and not "evidence of the conspiracy".

Conclusion

The problem with conspiracy theories is that everything not explained is used to "prove" the conspiracy.

We see that happening here. If there are unexplained formatting errors in the information the NYPost published, and the only known theory that explains them is a conspiracy, then they prove the conspiracy.

That's stupid. Unknown things may simply be unknown, that while you can't explain them doesn't mean they are unexplainable. That's what we see here: people are have convinced themselves they have "proof" because of unexplainable formatting errors, when in fact, such formatting can be explained.

The NYPost story has many problems. It is data taken out of context in an attempt to misinform the reader. We know it's a garbage story, even if all the emails are authentic. We don't need to invent conspiracy theories to explain it.

Wednesday, October 14, 2020

Yes, we can validate leaked emails

When emails leak, we can know whether they are authenticate or forged. It's the first question we should ask of today's leak of emails of Hunter Biden. It has a definitive answer.

Today's emails have "cryptographic signatures" inside the metadata. Such signatures have been common for the past decade as one way of controlling spam, to verify the sender is who they claim to be. These signatures verify not only the sender, but also that the contents have not been altered. In other words, it authenticates the document, who sent it, and when it was sent.

Crypto works. The only way to bypass these signatures is to hack into the servers. In other words, when we see a 6 year old message with a valid Gmail signature, we know either (a) it's valid or (b) they hacked into Gmail to steal the signing key. Since (b) is extremely unlikely, and if they could hack Google, they could a ton more important stuff with the information, we have to assume (a).

Your email client normally hides this metadata from you, because it's boring and humans rarely want to see it. But it's still there in the original email document. An email message is simply a text document consisting of metadata followed by the message contents.

It takes no special skills to see metadata. If the person has enough skill to export the email to a PDF document, they have enough skill to export the email source. If they can upload the PDF to Scribd (as in the story), they can upload the email source. I show how to below.

To show how this works, I send an email using Gmail to my private email server (from gmail.com to robertgraham.com).

The NYPost story shows the email printed as a PDF document. Thus, I do the same thing when the email arrives on my MacBook, using the Apple "Mail" app. It looks like the following:

The "raw" form originally sent from my Gmail account is simply a text document that looked like the following:

This is rather simple. Client's insert details like a "Message-ID" that humans don't care about. There's also internal formatting details, like the fact that this is a "plain text" message rather than an "HTML" email.

But this raw document was the one sent by the Gmail web client. It then passed through Gmail's servers, then was passed across the Internet to my private server, where I finally retrieved it using my MacBook.

As email messages pass through servers, the servers add their own metadata.

When it arrived, the "raw" document looked like the following. None of the important bits changed, but a lot more metadata was added:
The bit you care about here is the "DKIM-Signature:" metadata.
This is added by Gmail's servers, for anything sent from gmail.com. It "authenticates" or "verifies" that this email actually did come from those servers, and that the essential content hasn't been altered. The long strings of random-looking characters are the "cryptographic signature". That's what all crypto is based upon -- long chunks of random-looking data.

To extract this document, I used Apple's "Mail" client program and selected "Save As..." from the "File" menu, saving as "Raw Message Source".




I uploaded this this document to Scrib so that anybody can download and play with it, such as verifying the signature.

To verify the email signature, I simply open the email document using Thunderbird (Firefox's email client) with the "DKIM Verifier" extension, which validates that the signature is indeed correct. Thus we see it's a valid email sent by Gmail and that the key headers have not been changed:
The same could be done with those emails from the purported Hunter Biden laptop. If they can be printed as a PDF (as in the news story) then they can also be saved in raw form and have their DKIM signatures verified.

This sort of thing is extraordinarily easy, something anybody with minimal computer expertise can accomplish. It would go a long way to establishing the credibility of the story, proving that the emails were not forged. The lack leads me to believe that nobody with minimal computer expertise was involved in the story.

The story contains the following paragraph about one of the emails recovered from the drive (the smoking gun claiming Pozharskyi met Joe Biden), claiming how it was "allegedly sent". Who alleges this? If they have the email with a verifiable DKIM signature, no "alleging" is needed -- it's confirmed. Since Pozharskyi used Gmail, we know the original would have had a valid signature.


The lack of unconfirmed allegations that could be confirmed seems odd for a story of this magnitude.

Note that the NYPost claims to have a copy of the original, so they should be able to do this sort of verification:

However, while they could in theory, it appears they didn't in practice. The PDF displayed in the story is up on Scribd, allowing anybody to download it. PDF's, like email, also have metadata, which most PDF viewers will show you. It appears this PDF was not created after Sunday when the NYPost got the hard drive, but back in September when Trump's allies got the hard drive.





Conclusion

It takes no special skills to do any of this. If the person has enough skill to export the email to a PDF document, they have enough skill to export the email source. Instead of "Export to PDF", select "Save As ... Raw Message Source". Instead of uploading the .pdf file, upload the resulting .txt to Scribd.

At this point, a journalist wouldn't need to verify DKIM, or consult an expert: anybody could verify it. There a ton of tools out there that can simply load that raw source email and verify it, such as the Thunderbird example I did above.










Thursday, October 08, 2020

Factcheck: Regeneron's use of embryonic stem cells

This week, Trump's opponents misunderstood a Regeneron press release to conclude that the REG-COV2 treatment (which may have saved his life) was created from stem cells. When that was proven false, his opponents nonetheless deliberately misinterpreted events to conclude there was still an ethical paradox. I've read the scientific papers and it seems like this is an issue that can be understood with basic high-school science, so I thought I'd write up a detailed discussion.

The short answer is this:

  • The drug is not manufactured in any way from human embryonic tissues.
  • The drug was tested using fetal/embryonic cells, but ones almost 50 years old, not new ones.
  • Republicans want to stop using new embryos, the ethical issue here is the continued use of old embryos, which Republican have consistently agreed to.
  • Yes, the drug is still tainted by the "embryonic stem cell" issue -- just not in any of the ways that people claim it is, and not in a way that makes Republicans inconsistent.
  • Almost all medical advances of the last few decades are similarly tainted.
Now let's do the long, complicated answer. This starts with a discussion of the science of Regeneron's REG-COV2 treatment.

A well-known treatment that goes back decades is to take blood plasma from a recently recovered patient and give it to a recently infected patient. Blood plasma is where the blood cells are removed, leaving behind water, salts, other particles, and most importantly, "antibodies". This is the technical concept behind the movie "Outbreak", though of course they completely distort the science.

Antibodies are produced by the immune system to recognize and latch onto foreign things, including viruses (the rest of this discussion assumes "viruses"). They either deactivate the virus particle, or mark it to be destroyed by other parts of the immune system, or both.

After an initial infection, it takes a while for the body to produce antibodies, allowing the disease to rage unchecked. A massive injection of antibodies during this time allows the disease to be stopped before it gets very far, letting the body's own defenses catch up. That's the premise behind Trump's treatment.

An alternative to harvesting natural antibodies from recently recovered patients is to manufacture artificial antibodies using modern science. That's what Regeneron did.


An antibody is just another "protein", the building blocks of the body. The protein is in the shape of a Y with the two upper tips formed to lock onto the corresponding parts of a virus ("antigens"). Every new virus requires a new antibody with different tips.

The SARS-COV-2 virus has these "spike" proteins on it's surface that allow it to invade the cells in our lungs. They act like a crowbar, jamming themselves into the cell wall, then opening up a hole to allow the rest of the virus inside. Since this is the important and unique protein of the virus, it's what most antibodies will want to lock onto.

Proteins are created from genes. A new protein, like an antibody with tips identifying a new virus, needs a new gene to create it. In other words, we need new DNA.

This happens in "white blood cells". Inside these cells, the gene that makes antibodies can easily mutate. When the white blood cell encounters a new object, it mutates that gene slightly, makes some new antibodies, and tests them against the foreign object. The cell then divides, and the child cells do the same thing. Each generation gets better and better and better at creating antibodies. Those tips of the antibody become better and better at locking onto the infecting virus.

Before we go down into Lamarck genetics, we should point out that these genes are not passed down to children. Only a few white blood cells change their DNA, but this doesn't affect any other cells, especially not the ones in your gonads.

The way Regeneron makes its treatment is to harvest the white blood cells, extract the gene that makes the antibody, then sticks that gene inside some hamster cells to produce copious amounts of the antibody. (Yes, hamsters, but we'll get to that).

Sometimes human subjects aren't available as a source of white blood cells. For example, let's consider a disease that hasn't infected humans yet, but which has a potential to do so. In that case, you need a factory for white blood cells that isn't human.

Regeneron has a solution for this: transgenic mice that have the important parts of the human immune system grafted in. This allows them to inject things into the mice to cause this hypermutation of the antibody gene, which they can then harvest.

In the case of their REG-COV2 treatment, Regeneron used both mice and men. They gathered about 200 candidate antibody genes from both sources.

Remember: each time white blood cells mutate to create an antibody, they'll do it differently. That means everybody's antibodies are different even though the disease is the same. Even a single patient will have multiple strains of white blood cells mutating in different directions, creating different antibodies, for the same thing.

Thus, from 32 mice and a few human patients, Regeneron got around 200 candidate antibody genes. They then reduced this number down to 4 really good candidates, and then 2 (one from a human, one from a mouse) that they decided to use for manufacturing. These were sent to the hamster factory.

It's at this point we need to talk about hamsters and immortalized cell lines.

You can keep tissues alive outside the body by bathing them in a nutrient bath, but they won't grow on their own. But in some cases, you can cause them to grow without end, in which.case, you'll have an endless supply of those cells. The cell line has then become immortal. This is especially true if the original cells came from a cancer -- that's what cancer is, when the thing that prevents cells from dividing has been broken, and they grow out of control.

Of the many immortalized cell lines used by researchers, some come from adults who consented, some from adults who were never asked (such as the famous HeLa line), some from animals, and of course, some from embryos/fetuses.

One important cell line comes from a Chinese hamster ovary (CHO) that was smuggled out of China. It's become the preferred source for producing mammal proteins that can't be produced from yeasts or bacteria. In other words, simple proteins like insulin can be produced from yeast, but complex proteins like antibodies can only be produced within mammals. They insert a human gene into the cell, then encourage it to start dividing into billions of cells containing a copy of that gene.

Note that while the CHO cell line is used for about 50% of the time in this sort of case, about 20% of the time, human cell lines are used. The two human cell lines for doing this are known as HEK293 and PER.C6. Once Regeneron decided upon which genes it wanted to manufacture, it inserted those genes into Chinese hamster ovary (CHO) cells to mass produce the drug. The fact that it was CHO and not the human cell lines is pretty important to this story.

Immortalized cell lines appear in other places in our story. When selecting which of the 200 candidate antibodies it wanted to mass produce, Regeneron tested them for efficacy. It tested against tissues in vitro (a test tube using immortalized cell lines) rather than in vivo (inside a human body). One cell line is "Calu-3", derived from a 25-year-old lung cancer patient in 1975. Another cell line is "Vero", derived from the kidney's of an African green monkey in 1962.

A third test uses proteins made from the "HEK293" cell line from the kidney of a human fetus aborted around 1972-1973 in the Netherlands. This the center of the current controversy.

This test wasn't necessary to the creation of REG-COV2. It was included with the tests because other researchers used the technique, and that's what science does, replicates the work of other researchers.

I mention this because while people have reluctantly agreed that REG-COV2 isn't manufactured from embryos (from the HEK293 or PER.C6 cell lines), they insist that because of this test, it couldn't have been made without embryonic cells. This is not true, the test wasn't necessary. In addition, the test could've been done in different way, using a different source for the proteins involved. Vaccines are tested in similar ways, some using the ethically questionable cell lines, some not.

But the results are still ethically tainted. The point here isn't to excuse this taint, but to simply point out it's different type of taint. There's a range of ethical questions here. The issue is being politicized to make this one ethical question, when it's a different ethical question.

This is a good time to talk about the ethics of embryonic stem cells. There are a lot of different technical issue here.

The major issue that upsets Republicans is the harvesting of new material from blastocysts, embryos, and aborted fetuses. This is a wholly separate question of continuing to use old material from 50 years ago.

When President George Bush was elected in 2000, he instituted rulings forbidding harvesting new material, but which allowed the continued use of old material (within the United States). The continued use of HEK293 was explicitly allowed. Likewise, Trump issued an executive order limiting stem cell research. It explicitly targeted harvesting new embryonic cells, while at the same time, explicitly allowed existing lines like HEK293.

Thus, if you are trying to show that Republicans are hypocrites, that their rules change when their own life is at stake, then the evidence doesn't support your conclusion. Even if the HEK293 cell line was used for manufacturing instead of testing, it still would be consistent with Republican positions. Their concern is to stop the use of exploitation of new embryos.

Now for Catholics, things might be different. The Vatican has repeatedly come down against using old material like HEK293 [a] [b]. They view it along the same lines as using research from Nazi medical experiments on Jews in concentration camps. People ask the ethical question whether the event was so offensive that the medical knowledge can't be used, even if it saves lives. Even here, though, Catholics have a more nuanced view, allowing just things to be used in practice when there is no alternative.

From that perspective, then all medical research is tainted. For example, our knowledge from all vaccines comes from Edward Jenner's testing on an unwitting 8 year old boy. Ethics have been continually changing throughout history, if we reject all knowledge from what we now consider to be unethical sources, then we wouldn't have any medicine. 50 years ago when the HEK293 was acquired, it was under a different understanding of ethics than we have today.

Cell lines like the 50 year old HEK293 are used to test almost every drug. Google those letters and any of the other drugs Trump took in response to his infection, and you'll find they are all tainted. Moreover, many of the upcoming vaccines will also use these such cell lines to test their efficacy. This may still be an ethically important question, but it's not the politicized question at stake here.

This piece has tried to avoid getting into the technical weeds. For example, the HEK293 line aren't "stem cells" but "kidney cells". But HEK293 still comes from an aborted fetus, and thus, has the same ethical issue as what people understand by "embryonic stem cells". Instead, I tried to look at the technical issues I feel do matter, like whether this is researchers using a 50 year old line that Republicans have consistently agreed to, versus newly harvested material which they vehemently oppose. Theoretically, somebody could have an issue with "stem cells" even when they come from bone marrow or cord blood, in which case, this article is not for you. I'm pretty sure no such people exist, except those who misunderstand the science. If you feel I've glossed over a technical issue (or gotten it wrong), please tell me https://twitter.com/ErrataRob.


Conclusion

This piece is not a defense of Trump but of science. Please vote for Biden on November 3. European countries with leaders to the left of Biden are nonetheless still prosperous. Conversely, when otherwise prosperous democracies have failed, it's because of leaders as unfit and corrupt as Trump.

This issue started when people gleefully believed they had caught Trump in a trap. When this was proven a misconception, they went searching for other ties to stem cells, and found them. This is still a gross distortion of science -- every modern medical treatment can be found to be tainted if you look hard enough. Trying to rescue your misconception by jumping through hoops like this makes you look worse, not better.

The MIT Technology Review article cited above is a particularly egregious example of the politicization of science. It cites Trump's order on embryonic stem cells while knowingly avoiding what the order actually said, that it was about new vs. old embryos. They knowingly distorted the information to make it look like the consistent position was inconsistent. They knowingly distorted the science to make political points.

There are important areas where science is entangled with politics (e.g. climate change). But it seems like everyone takes the opportunity to irresponsibly distort science to further their politics, as seen here.

Frankly, I'm freaked out by planting a human immune system into mice into order to drive hypermutation, to extract a gene that you then place into an immortal line of hamster ovary cells to produce a crafted protein. I'm sure when somebody makes a movie based on this, it won't be anything other than dystopic.

Saturday, September 12, 2020

Cliché: Security through obscurity (yet again)

Infosec is a largely non-technical field. People learn a topic only as far as they need to regurgitate the right answer on a certification test. Over time, they start to believe misconceptions about that topic that they never learned. Eventually, these misconceptions displace the original concept in the community.

A good demonstration is this discussion of the "security through obscurity fallacy". The top rated comment makes the claim this fallacy means "if your only security is obscurity, it's bad". Wikipedia substantiates this, claiming experts advise that "obscurity should never be the only security mechanism".

Nope, nope, nope, nope, nope. It's the very opposite of what you suppose to understand. Obscurity has problems, always, even if it's just an additional layer in your "defense in depth". The entire point of the fallacy is to counteract people's instinct to suppress information. The effort has failed. Instead, people have persevered in believing that obscurity is good, and that this entire conversation is only about specific types of obscurity being bad.


Hypothetical: non-standard SSH

The above discussion mentions running SSH on a non-standard port, such as 7837 instead of 22, as a hypothetical example.

Let's continue this hypothetical. You do this. Then an 0day is discovered, and a worm infecting SSH spreads throughout the Internet. This is exactly the sort of thing you were protecting against with your obscurity.

Yet, the outcome isn't what you expect. Instead, you find that the all your systems running SSH on the standard port of 22 remain uninfected, and that the only infections were of systems running SSH on port 7837. How could this happen?

The (hypothetical) reason is that your organization immediately put a filter for port 22 on the firewalls, scanned the network for all SSH servers, and patched the ones they found. At the same time, the worm runs automated Shodan scripts and masscan, and thus was able to nearly instantaneously discover the non-standard ports.

Thus you cleverness made things worse, not better.


Other phrases

This fallacy has become such a cliche that we should no longer use it. Let's use other phrases to communicate the concept. These phrases would be:

  • attackers can discover obscured details far better than you think, meaning, obscurity is not as beneficial as you think
  • defenders are hindered by obscured details, meaning, there's a greater cost to obscurity than you think
  • we can build secure things that don't depend upon obscurity
  • it's bad to suppress information that you think would help attackers
  • just because there's "obscurity" involved doesn't mean this principle can be invoked

Obscurity less beneficial, more harmful than you think

My hypothetical SSH example demonstrates the first two points. Your instinct is to believe that adding obscurity made life harder for the attackers, and that it had no impact on defenders. The reality is that hackers were far better than you anticipated at finding unusual ports. And at the same time, you underestimated how this would impact defenders.

It's true that hiding SSH ports might help. I'm just showing an overly negative hypothetical result to counteract your overly positive result. A robust cost-vs-benefit analysis might show that there is in fact a benefit. But in this case, no such robust argument exists -- people are just in love with obscurity. Maybe hiding SSH on non-standard ports is actually good, it's just that nobody has made an adequate argument for it. Lots of people love the idea, however.


We can secure things

The first two points are themselves based upon a more important idea: we can build secure things. SSH is a secure thing.

The reason people love obscurity is because they have no faith in security. They believe that all security can be broken, and therefore, every little extra bit you can layer on top will help.

In our hypothetical above, SSH is seen as something that will eventually fail due to misconfiguration or an exploitable vulnerability. Thus, adding obscurity helps.

There may be some truth to this, but your solution should be to address this problem specifically. For example, every CISO needs to have an automated script that will cause all the alarms in their home (and mobile) to go off when an SSH CVE happens. Sensitive servers need to have canary accounts that will trigger alarms if they ever get compromised. Planning for an SSH failure is good planning.

But not planning for SSH failure, and instead just doing a bunch of handwaving obscuring things, is a bad strategy.

The fact is that we can rely upon SSH and should rely upon SSH. Yes, an 0day might happen, but that, too, should be addressed with known effective solutions, such as tracking CVEs and vulnerability management, not vague things like adding obscurity.


Transparency good, suppression bad

The real point of this discussion isn't "obscurity" at all, but "transparency". Transparency is good. And it's good for security for exactly the same reason it's good in other areas, such as transparency in government so we can hold politicians accountable. Only through transparency can we improve security.

That was the point of Kerckhoffs's principle from the 1880s til today: the only trustworthy crypto algorithms are open, public algorithms. Private algorithms are insecure.

It's the point behind the full-disclosure debate. Companies like Google who fully disclose in 90 days are trustworthy, companies like Oracle who work hard to suppress vuln information are untrustworthy. Companies who give bounties to vuln researchers to publish bugs are trustworthy, those who sue or arrest researchers are untrustworthy.

It's where security snake oil comes from. Our industry is rife with those who say "trust us ... but we can't reveal details because that would help hackers". We know this statement to be categorically false. If their system were security, then transparency would not help hackers. QED: hiding details means the system is fundamentally insecure.

It's like when an organization claims to store passwords security, but refuses to tell you the algorithm, because that would reveal information hackers could use. We know this to be false, because if passwords were actually stored securely, knowing the algorithm wouldn't help hackers.

Instead of saying the "security through obscurity fallacy" we should instead talk about the "security through suppression fallacy", or simply say "security comes from transparency".


This doesn't apply to all obscurity

This leads to my last point: that just because "obscurity" is happening doesn't mean we can automatically apply this concept.

Closed-source code is a good example. Why won't they company share their source code? If they say "because it helps hackers", then that's a clear violation of this principle. If they say "because trade secrets", then it's not a violation of this principle. They aren't saying obscurity is needed for security, they are saying obscurity is needed because they don't want people copying their ideas.

We can still say that the security of closed-source is worse than open-source, because it usually is. The issues are clearly related. It's simply that the vendor isn't, in this hypothetical, violating the fallacy by claiming closed-source means their code is more secure.

The same is true in the blogpost above of adding decoy cars to a presidential motorcade. I guess you could use the word "obscurity" here, but it has nothing to do with our principle under discussion. For one thing, we aren't talking about "suppressing" information. For another thing, presidential motorcades are inherently insecure -- this isn't a crypto algorithm or service like SSH that can be trusted, it's a real crap system that is inherently insecure. Maybe handwaving with half-assed solutions, like varying travel routes, cellphone jammers to block IEDs, using decoy cars, is the on the whole the best compromise for a bad situation.

Thus, stop invoking this principle every time "obscurity" happens. This just wears out the principle and breeds misunderstanding for the times when we really do need it.


Conclusion

The point of this blogpost is unwinding misconceptions. A couple years from now, I'm likely to write yet another blogpost on this subject, as I discover yet new misconceptions people have developed. I'm rather shocked at this new notion that everyone suddenly believes, that "obscurity" is bad as the only control, but good when added as a layer in a defense-in-depth situation. No, no, no, no ... just no.

These misconceptions happen for good reasons. One of which is that we sometimes forget our underlying assumptions, and that people might not share these assumptions.

For example, when we look at Kerckhoffs' Principle from the 1880s, the underlying assumption is that we can have a crypto algorithm that works, like AES or Salsa20, that can't be broken. Therefore, adding obscurity on top of this adds no security. But when that assumption fails, such as a presidential motorcade that's always inherently insecure (just lob a missile at them), then the argument no longer applies.

When teaching this principle, the problem we have is that a lot of people, especially students new to the field, are working from the assumption that everything is broken and that no security can be relied upon. Thus, adding layers of obscurity always seems like a good idea.

Thus, when I say that "security through obscurity is bad", I'm really using this cliche to express some underlying idea. Am I talking about my political ideas of full-disclosure or open-source? Am I talking about vendor snake-oil? Am I talking about dealing with newbies who prefer unnecessary and ineffective solutions over ones proven to work? It's hard to tell.

The original discussion linked on Hacker News, though, discussed none of these things. Going through the top ranked responses seemed list a list of people who just heard about the thing yesterday and wanted to give their uninformed hot take on what they think these words mean.


Case Study: ASLR (Address Space Layout Randomization) (Update)

After posting, some have discussed on Twitter whether ASLR is just "security through obscurity". Let's discuss this.

The entire point of this post is to raise the level of discussion beyond glibly repeating a cliché. If you have an argument to be made about ASLR, then make that argument without resorting to the cliché. If you think the cost-vs-benefit analysis means ASLR is not worth it, then argue the cost-vs-benefit tradeoff.

The original cliché (from Kerckhoffs principles) wasn't about whether the algorithm added obscurity, but whether the algorithm itself is obscure.

In other words, if Microsoft was claiming Windows is secure because of ASLR, but that they couldn't divulge details how it worked because this would help hackers, then you have a "security through obscurity" argument. Only in this instance can you invoke the cliché and be assured you are doing so correctly.

I suppose you could argue that ASLR is only "obscurity", that it provides no "security". That's certainly true sometimes. But it's false other times. ASLR completely blocks certain classes of attacks on well-randomized 64-bit systems. It's such a compelling advantage that it's now a standard part of all 64-bit operating systems. Whatever ASLR does involving "obscurity", it clearly adds "security".

In short, just because there's "obscurity" involved doesn't mean the cliché "security through obscurity" can be invoked.


Sunday, July 19, 2020

How CEOs think

Recently, Twitter was hacked. CEOs who read about this in the news ask how they can protect themselves from similar threats. The following tweet expresses our frustration with CEOs, that they don't listen to their own people, but instead want to buy a magic pill (a product) or listen to outside consultants (like Gartner). In this post, I describe how CEOs actually think.


Monday, July 13, 2020

In defense of open debate

Recently, Harper's published a Letter on Justice and Open Debate. It's a rather boring defense of liberalism and the norm of tolerating differing points of view. Mike Masnick wrote rebuttal on Techdirt. In this post, I'm going to rebut his rebuttal, writing a counter-counter-argument.

The Letter said that the norms of liberalism tolerate disagreement, and that these norms are under attack by increasing illiberalism on both sides, both the left and the right.

My point is this: Masnick avoids the rebutting the letter. He's recycling his arguments against right-wingers who want their speech coddled, rather than the addressing the concerns of (mostly) left-wingers worried about the fanaticism on their own side.

Tuesday, June 16, 2020

Apple ARM Mac rumors

The latest rumor is that Apple is going to announce Macintoshes based on ARM processors at their developer conference. I thought I'd write up some perspectives on this.