Friday, September 24, 2021

Check: that Republican audit of Maricopa

Author: Robert Graham (@erratarob)

Later today (Friday, September 24, 2021), Republican auditors release their final report on what they found with elections in Maricopa county. Draft copies of the report have already circulated online. In this blogpost, I write up my comments on the cybersecurity portions of their draft.

The three main problems are:

  • They misapply cybersecurity principles that are meaningful for normal networks, but which don’t really apply to the "air gapped" networks we see here.
  • They make some errors about technology, especially networking.
  • They are overstretching themselves to find dirt, claiming the things they don't understand are evidence of something bad.

In the parts below, I pick apart individual pieces from that document to demonstrate these criticisms. I focus on section 7, the cybersecurity section, and ignore the other parts of the document, where others are more qualified than I to opine.

In short, when corrected, section 7 is nearly empty of any content. Software and Patch Management, part 1

They claim Dominion is defective at one of the best-known cyber-security issues: applying patches.

It’s not true. The systems are “air gapped”, disconnected from the typical sort of threat that exploits unpatched systems. The primary security of the system is physical. Frequent patching isn't expected.

This is a standard in other industries with hard reliability constraints, like industrial or medical. Patches in those systems can destabilize computers and kill people, so these industries are risk averse and resist applying them. They prefer to mitigate the threat in other ways, such as with firewalls and air gaps.

Yes, this approach is controversial. There are some in the cybersecurity community who use lack of patches as a bludgeon with which to bully any who don’t apply every patch immediately. But this is because patching is more a political issue than a technical one. In the real, non-political world we live in, most things don’t get immediately patched all the time. Software and Patch Management, part 2

The auditors claim new software executables were applied to the system, despite the rules against new software being applied. This isn’t necessarily true.

There are many reasons why Windows may create new software executables even when no new software is added. One reason is “Features on Demand” or FOD. You’ll see new executables appear in C:\Windows\WinSxS for these. Another reason is their .NET language, which causes binary x86 executables to be created from bytecode. You’ll see this in the C:\Windows\assembly directory.

The auditors simply counted the number of new executables, with no indication which category they fell into. Maybe they are right, maybe new software was installed or old software updated. It’s just that their mere counting of executable files doesn’t show understanding of these differences. Log Management

The auditors claim that a central log management system should be used.

This obviously wouldn’t apply to “air gapped” systems, because it would need a connection to an external network.

Dominion already designates their EMSERVER as the central log repository for their little air gapped network. Important files from C: are copied to D:, a RAID10 drive. This is a perfectly adequate solution, adding yet another computer to their little network would be overkill, and add as many security problems as it solved.

One could argue more Windows logs need to be preserved, but that would simply mean archiving the from the C: drive onto the D: drive, not that you need to connect to the Internet to centrally log files. Credential Management

Like the other sections, this claim is out of place given the airgapped nature of the network.

Dominion simply uses “role based security” instead of normal user accounts. It’s a well known technique, and considered very appropriate for this sort of environment.

The auditors claim account passwords must “be changed every 90 days”. This is a well-know fallacy in cybersecurity. It took years to get NIST to remove it from their recommendations. If CISA still has it in their recommendations for election systems, then CISA is wrong.

Ideally, accounts wouldn’t be created until they were needed. In practice, system administrators aren’t available (again, it’s an airgapped system, so no remote administration). Dominions alternative is to create the accounts ahead of time, suc has “adjuser09”, waiting for the 9th person you hire that might use that account.

They are all given the same default password to start, like “Arizona2019!!!”. Some customers choose to change the default password, but obviously Maricopa did not. This is weak – but not a big deal, since the primary security is from controlling physical access. Lack of Baseline for Host and Network Activity

They claim sort of baselining should be done. This is absurd. Baselines are always problematic, but would be especially so in this case.

The theory of baselines is that a networks traffic is somewhat predictable on a day-to-day basis. This obviously doesn’t apply to elections systems, which are highly variable day-to-day, especially on election day.

Baselining is the sort of thing you do with a dedicated threat hunting team. It’s incredibly inappropriate for a small installation like this. Network Related Data

The auditors asked for an unreasonable access to network data, in the worst way possible, triggering the refusal to hand it over. They didn’t ask for reasonable data. They blame Maricopa Count for the conflict, but it’s really themselves who are to blame.

A reasonable request would take the MAC addresses from the election machines and ask for any matching records the Maricopa might have in their Splunk, DHCP, or ARP logs. Matches shouldn’t be found, but if they were, the auditors should then ask for flow data for the associated IP addresses.

They are correct in identifying this as a very important issue. Dominion security depends upon an airgap. If auditors find a netowrk connection, it’s bad. It’s not catastrophic, and sometimes machines are disconnected from one network and attached to a network during other times than the election. But this would very much be a useful part of a report – if only they had made a reasonable request that didn’t demand Maricopa spend their entire yearly budget to comply. Other Devices Connected to the Election Network

The auditors complain they weren’t given access to the router identified by

It probably doesn’t exist.

Routers aren’t needed by devices that are on the same local Ethernet. They wouldn’t exist on a single-segment air gapped network. But typical operating-system configuration demands one be configured anyway, so it’s common to put in a dummy router address even if it’s unused.

If you see messages like this one in the logs, it means the router wasn’t there:

The auditors are right in identifying this as an important issue. If there were such a router, then this would cast doubt whether the network was “airgapped”.

Note that if such a router did exist, it would almost certainly be a NAT. This would still offer some firewall protection, just not as strong as an air gap.

7.5.4 Anonymous Logins

They see something in the security logs they don’t understand, and blame Maricopa’s lack of network data ("the routers") for their inability to explain it.

This is an extraordinarily inappropriate claim, based not on expert understanding of what they see in the logs, but complete ignorance. There’s no reason to believe that getting access to Maricopa Count network logs would explain what’s going on here.

This demonstrates they are on a phishing expedition, and that everything they see that they can’t explain is used as evidence of a conspiracy, either of Maricopa to withhold data, or of election fraud.

The Dominion suite of applications and services is oddly constructed and will produce anomalies. Comparing against a general Windows server not running Dominion’s suite is meaningless.

7.5.5 Dual Boot System Discovered

The auditors claim something about “dual-homed hosts” or “jump-boxes”. That’s not how these terms are normally used. These terms normally refer to a box with access to two separate networks, not a box with two separate operating systems.

This requires no nefarious explanation. This is commonly seen in corporate networks, either because somebody simply added a new drive to re-install the operating-system, or repurposed an old drive from another system as a data drive, and simply forgot to wipe it. The BIOS points to one they intend to boot from and ignore the fact that the other can also boot.

There are endless non-nefarious explanations for what is seen here that doesn’t require a nefarious one. It’s not even clear its a failure of their build process, which focuses on what’s on the boot drive and not what’s on other drives in the system.

7.5.6 EMS Operating System Logs Not Preserved

It is true the EMS operating-system logs are not preserved (well, generally not preserved). By this I refer to the generic Windows logs, the same logs that your own Windows desktop keeps.

The auditors falsely claim that this violates the law. This is false. The “electron records” laws don’t cover the operating-system. The laws instead are intended to preserve the records of the election software running on top of the operating-system, not those of the operating-system itself.

This issue has long been known. You don’t need an auditor’s report to tell you that these logs aren’t generally preserved – everyone has known this for a long time, including those who certified Dominion.

The subtext of this claim is the continued argument by Republicans that the fact they can’t find evidence for 2020 election fraud is because key data is missing. That’s the argument of Tina Peters, the former clerk of a county in the neighboring state of Colorado, who claims their elections cannot be audited because they don’t have the Windows operating-system logs.

It’s not true. System logs are as likely to cause confusion, as they do above with the “anonymous logins” issue. They are unlikely to provide proof of votes being flipped in a hack. If there was massive fraud, as detected by recounts of paper ballots, I’d certainly want such system logs to search for how it happened. But I wouldn’t use such logs in order to audit the vote.

Note that the description of “deleting” log entries by overfilling the logs is wrong. If it were important to preserve such logs, then they would be copied right after the election. They wouldn’t be left to rot on the boot drive for months afterwards.

As a forensics guy, I would certainly support the idea that Dominion should both enable more logs and preserve them after each election. They don’t require excessive storage and can be saved automatically in the last phase of an election. But their lack really isn’t all that important, they are mostly just full of junk.


We live in a pluralistic democracy, meaning there are many centers of power, each competing with each other. It's inherently valid for one side to question and challenge the other side. But this can go too far, to the point where you are challenging the stability of our republic.

The Republican party is split. Some are upholding that principle of pluralism, wanting to make sure future elections are secure and fair. Others are attacking that principle, challenging the peaceful transfer of power in the last election with baseless accusations of fraud.

This split is seen in Arizona, where Republicans have demanded an audit by highly partisan auditors. An early draft of their report straddles that split, containing some reasonable attempt to create recommendations for future elections, while simultaneous providing fodder for the other side to believe the last election was stolen.

A common problem with auditors is that when they can’t find the clear evidence they were looking for, the fill their reports with things they don’t understand. I think I see that here. The auditors make technical errors in ways that question their competence, but that’s likely not true. Instead, they kept searching past where they were strong into areas where they were weak, looking for as much dirt as possible. Thus, in this report, we see where they are technically weak.

Trumpists, meaning those attacking the peaceful transfer of power with baseless accusations of fraud, will certainly use this report to champion their cause, despite the headline portion that confirms the vote count. But for the rest of us, we should welcome this report. Elections do need to be fixed, and while it’s unlikely we’ll fix them in the ways suggested in this report, it will add visibility into the process which we can use to debate improvements.

This blogpost is only a first draft. While the technical bits in section 7 look fairly straightforward to me, I'm guessing that people who don't understand them will come up with weird conspiracy-theories about them. Thus, I'm guessing I'll have to write another blogpost in a week debunking some of the crazier ideas.


Allen Dekorsey said...

This whole thing should have been over months ago after the hand recount was completed. The election uses paper ballots which are entered into the system by capturing a high resolution digital image, the paper ballots are then securely stored and no longer used. Th digital images are what the system uses in the vote tallies. You count the votes on the paper ballots and compare them against the digital counts, if they are essentially the same then it doesn’t matter even if the election system was hacked or Dominion changed the ‘digital’ votes because the original paper ballots cannot be hacked or changed.

The election wasn’t stolen from Trump. He lost for a very simple reason, he was a terrible candidate seeking re-election, history has several candidates seeking re-election who lost because they were just as terrible (Bush 41 and Carter come to mind).

ez2Talk2TX said...

Thanks. I look forward to your next report on this and other future posts from you.

Steve newman said...

Brilliant and clear. This is heroic work.

JD said...

This is your quote: "
The auditors are right in identifying this as an important issue. If there were such a router, then this would cast doubt whether the network was “airgapped”.

So, there is evidence, even beyond this one "important issue" casting doubt that the system was actually air gapped. Should this " important issue", as you call it, be dropped or investigated.

Elrod16 said...

No, the router DIDN'T exist, hence why there were no records to review from it. He clearly states that the OS expects a value there even if there is no physical router, so a dummy value was used instead. Just how on Linux you can redirect traffic to dummy addresses to give the program the impression that it was granted network access without actually granting network access to block it without crashing it.

Allen Dekorsey said... is the default address for the gateway to a WAN. From what I read, it looks like Windows expects that gateway to be present. So on a private network with no gateway to the internet (WAN) still needs that address be present (even as a dummy).

JD said...

I guess we'll find out if the router exists, when the "compromise" apparently met on the subject occurs. I'm also very curious about the evidence files were deleted---why? And is the timing just coincidental, as opposed to connected to the audit? Will the AG investigate the possible "deleters" identified by Mr Cotton? He is, incidentally, the guy who discovered OPM had been hacked by the Chinese--the most damaging hack in US history. His bone fides are good.

Nos said...

Disclaimer: I do not make a determination of fraud for or against, only reviewing this fact check on the technical merits. Software and Patch Management, part 1

A considerable amount of advisories require physical access to accomplish and lack of physical access is the predominant method of mitigation. Here we have the reverse of a standard corporate network. Physical access, no remote access. The security of the device is completely reliant upon the physical security employed. This is a very poor argument to make.

However, you mention the air gap as the primary reason not to patch in health/industrial, with serviceability/uptime as being the prime factor. Election systems do not have this problem with long periods of downtime. Further, there are arguably more motivational reasons to attack voting systems over health/industrial, which moves the needle more towards security than operability. The analogy to health/industrial does not follow along the air gap lines. Air gaps do not supplant security rigor.

Overall I find this argument to be shallow. Software and Patch Management, part 2

Agreed. The auditors should have detailed out what .EXE were created and where. As you say, this could be a nothing burger or worthy of investigation. Log Management

More networks equal difficult hardening and breaks paradigm from air gap. Though logging to the central server onsite, as mentioned is happening, especially to reconcile logs, would help.

However, there are solutions that provide one way communications while maintaining functional air gaps. Consider classified solutions where SECRET can be sent to TOP SECRET. They use a unidirectional fiber cable that can only send in one direction, towards the more classified. A similar solution could be employed to allow log messages, or any information outbound, but no possibility of inbound. Though this would complicate and possibly put the solution at risk unless adequate physical measures are taken care of, such as physically damaging the receiver to ensure no accidental connections are made.

Adequate solutions exist so the request is not unreasonable. Credential Management

I find the hand waiving of not changing default passwords very damaging to your overall message. Functional accounts are only as strong as being able to ensure a 1-to-1 user match. Not changing default passwords fundamentally breaks authentication and non-repudiation. This does not mean any bad acting occurred, but this is very much a black flag for a solution that relies upon integrity. Lack of Baseline for Host and Network Activity

Baselining during peak is a common practice. It's good to detect outlier behavior, but you would also need to capture data and analyze it, either live or after the fact. It could show anomalous behavior, but also might be another attack vector. I don't disagree with the finding, but I also don't see much value without addressing many other issues. Crawl, walk, run. This is run material. Network Related Data

If you're running an audit, you get the raw data. Asking them to turn over specific logs based upon hardware addresses can lead to human error or allow malfeasance leading to improper conclusions. It's not unreasonable to ask for the raw data to eliminate the human element. That's standard forensics.

Again, I find this to be more hand waving the argument rather than addressing the actual technical merits. You are arguing Schrödinger's logs are irrelevant. It's an open item, which should be easy to close. I cannot fathom a reason not to divulge logs as it only plays into the conspiracy theorists hand.

Nos said... Other Devices Connected to the Election Network

Making the case for windows logs with this argument alone.

You're right to call out it might not have existed, but it could have. We'd need the windows logs to know for certain.

It does also imply a lack of configuration management if this was entered improperly. Configuration management is also paramount to integrity.

Your NAT statement is just wrong on many levels, though. NAT provides zero firewall protection. Just because they are typically done on the same box does not mean NAT is a firewall. If this was truly NAT and not PAT, then there would be zero protection as NAT is a 1-to-1 of IP. PAT, which would be most likely the case, would only offer obfuscation protection. It's harder to attack, make no mistake, but it's not firewall level protection.

I really hope you were just trying to simplify this for laypersons, otherwise, this is really showing some ignorance on the technologies at play and seriously calls into question your credibility to make this fact check.

7.5.4 Anonymous Logins

Are you debunking the anonymous logins? Could the two not be linked? They don't have to be, but it would be great to rule it out. As it stands, it's possible, is it not? Also, any anonymous login needs to be rectified in a strongly enforced AAA solution. This would not fly in a PCI audit and I would put election integrity in the same realm of PCI.

How does this exactly demonstrate a phishing expedition? Is it a logical conclusion where there were anonymous logins along with a potential router connection? It stands to reason it potentially is, but no one knows for certain either direction. Again, I find this hand waving to be not in line with standard forensic procedures. There's an anomaly that might correlate two events. Instead of looking into whether they are correlated, wave them off.

I find the comment about Dominion system creating anomalies as extremely poor. 1) How do you know this? 2) Should these anomalies you admit happen be researched and found to actually be anomalies or part of a proper functioning system? Why should any anomaly exist? It should be known and determined to be part of the system or be investigated.

7.5.5 Dual Boot System Discovered

You simultaneously agree and disagree with the dual images on the machines. Given this is a solution designed for integrity, at a minimum we have poor configuration management, again. It does not mean anything nefarious, but it does mean integrity is not strong and we should be using this finding to correct future elections.

7.5.6 EMS Operating System Logs Not Preserved

In a system designed for integrity, with solid AAA that can enforce 1-to-1 authentication and non-repudiation, system logs are integral to that audit. The system is included in the audit. As you say, this is a glaring issue and with very little price to include. There is no argument to not include them. It is simply more data and can rule out plenty of arguments.

A problem does lie in people assuming because data is missing. However, excusing missing data is also extremely poor form.


Your fact check might do well with the uneducated in security and forensics, but there was a lot of hand waving or even misinformation in your descriptions. I'll give you the benefit of the doubt about the NAT misinformation and over simplified for a general audience, but as for a fact check, most of your arguments are subjective, not based on technical merit and rigor. Waving off missing components using the same logic those claiming there is evidence in the null. All we have is a null in these areas, it could mean something, it could mean nothing. To make any determination of its veracity absent the information is wrong and is itself misinformation.

Allen Dekorsey said...

“ Again, I find this to be more hand waving the argument rather than addressing the actual technical merits. You are arguing Schrödinger's logs are irrelevant. It's an open item, which should be easy to close. I cannot fathom a reason not to divulge logs as it only plays into the conspiracy theorists hand.”

The audit of physical ballots affirmed the official count. There is no reason to see more on the digital side unless you find proof that something might be amiss, like over 90% of the duplicate ballots were for one of the candidates. Giving them the logs only gives them more things to be confused over and more cover for their misguided theory.

JohnL said...

I believe you've missed another error in the credential management section, The first paragraph of this section paraphrases CISA, saying:

"CISA further recommends that usernames be assigned to a
specific person, not be shared, and be changed every 90 days."

Note that this entire paragraph is about usernames, not passwords. And the text makes it clear it's not just a typo, since that would imply that someone else is supposed to assign each user's passwords. The separate recommendation to also change passwords every 90 days doesn't come until a couple paragraphs later.

I don't recall any reputable expert ever recommending changing all user names every 90 days.

Until Cyber Ninjas, in their infinite wisdom, of course ...

Unknown said...

Wow what a bunch of nonsense. For a security guy you sound like a shill. Is this a joke? Surely this entire blog post needs to be rewritten, assuming you actually watched the 3-hour presentation everyone else did. How many different ways can you justify 'nothing to see here' lol I am impressed