Check: that Republican audit of Maricopa

Author: Robert Graham (@erratarob)

Later today (Friday, September 24, 2021), Republican auditors release their final report on what they found with elections in Maricopa county. Draft copies of the report have already circulated online. In this blogpost, I write up my comments on the cybersecurity portions of their draft.

https://arizonaagenda.substack.com/p/we-got-the-senate-audit-report

The three main problems are:

• They misapply cybersecurity principles that are meaningful for normal networks, but which don’t really apply to the "air gapped" networks we see here.
• They make some errors about technology, especially networking.
• They are overstretching themselves to find dirt, claiming the things they don't understand are evidence of something bad.

In the parts below, I pick apart individual pieces from that document to demonstrate these criticisms. I focus on section 7, the cybersecurity section, and ignore the other parts of the document, where others are more qualified than I to opine.

In short, when corrected, section 7 is nearly empty of any content.

7.5.2.1.1 Software and Patch Management, part 1

They claim Dominion is defective at one of the best-known cyber-security issues: applying patches.

It’s not true. The systems are “air gapped”, disconnected from the typical sort of threat that exploits unpatched systems. The primary security of the system is physical. Frequent patching isn't expected.

This is a standard in other industries with hard reliability constraints, like industrial or medical. Patches in those systems can destabilize computers and kill people, so these industries are risk averse and resist applying them. They prefer to mitigate the threat in other ways, such as with firewalls and air gaps.

Yes, this approach is controversial. There are some in the cybersecurity community who use lack of patches as a bludgeon with which to bully any who don’t apply every patch immediately. But this is because patching is more a political issue than a technical one. In the real, non-political world we live in, most things don’t get immediately patched all the time.

7.5.2.1.1 Software and Patch Management, part 2

The auditors claim new software executables were applied to the system, despite the rules against new software being applied. This isn’t necessarily true.

There are many reasons why Windows may create new software executables even when no new software is added. One reason is “Features on Demand” or FOD. You’ll see new executables appear in C:\Windows\WinSxS for these. Another reason is their .NET language, which causes binary x86 executables to be created from bytecode. You’ll see this in the C:\Windows\assembly directory.

The auditors simply counted the number of new executables, with no indication which category they fell into. Maybe they are right, maybe new software was installed or old software updated. It’s just that their mere counting of executable files doesn’t show understanding of these differences.

7.5.2.1.2 Log Management

The auditors claim that a central log management system should be used.

This obviously wouldn’t apply to “air gapped” systems, because it would need a connection to an external network.

Dominion already designates their EMSERVER as the central log repository for their little air gapped network. Important files from C: are copied to D:, a RAID10 drive. This is a perfectly adequate solution, adding yet another computer to their little network would be overkill, and add as many security problems as it solved.

One could argue more Windows logs need to be preserved, but that would simply mean archiving the from the C: drive onto the D: drive, not that you need to connect to the Internet to centrally log files.

7.5.2.1.3 Credential Management

Like the other sections, this claim is out of place given the airgapped nature of the network.

Dominion simply uses “role based security” instead of normal user accounts. It’s a well known technique, and considered very appropriate for this sort of environment.

The auditors claim account passwords must “be changed every 90 days”. This is a well-know fallacy in cybersecurity. It took years to get NIST to remove it from their recommendations. If CISA still has it in their recommendations for election systems, then CISA is wrong.

Ideally, accounts wouldn’t be created until they were needed. In practice, system administrators aren’t available (again, it’s an airgapped system, so no remote administration). Dominions alternative is to create the accounts ahead of time, suc has “adjuser09”, waiting for the 9th person you hire that might use that account.

They are all given the same default password to start, like “Arizona2019!!!”. Some customers choose to change the default password, but obviously Maricopa did not. This is weak – but not a big deal, since the primary security is from controlling physical access.

7.5.2.1.4 Lack of Baseline for Host and Network Activity

They claim sort of baselining should be done. This is absurd. Baselines are always problematic, but would be especially so in this case.

The theory of baselines is that a networks traffic is somewhat predictable on a day-to-day basis. This obviously doesn’t apply to elections systems, which are highly variable day-to-day, especially on election day.

Baselining is the sort of thing you do with a dedicated threat hunting team. It’s incredibly inappropriate for a small installation like this.

7.5.3.1.1 Network Related Data

The auditors asked for an unreasonable access to network data, in the worst way possible, triggering the refusal to hand it over. They didn’t ask for reasonable data. They blame Maricopa Count for the conflict, but it’s really themselves who are to blame.

A reasonable request would take the MAC addresses from the election machines and ask for any matching records the Maricopa might have in their Splunk, DHCP, or ARP logs. Matches shouldn’t be found, but if they were, the auditors should then ask for flow data for the associated IP addresses.

They are correct in identifying this as a very important issue. Dominion security depends upon an airgap. If auditors find a netowrk connection, it’s bad. It’s not catastrophic, and sometimes machines are disconnected from one network and attached to a network during other times than the election. But this would very much be a useful part of a report – if only they had made a reasonable request that didn’t demand Maricopa spend their entire yearly budget to comply.

7.5.3.1.? Other Devices Connected to the Election Network

The auditors complain they weren’t given access to the router identified by 192.168.100.1.

It probably doesn’t exist.

Routers aren’t needed by devices that are on the same local Ethernet. They wouldn’t exist on a single-segment air gapped network. But typical operating-system configuration demands one be configured anyway, so it’s common to put in a dummy router address even if it’s unused.

If you see messages like this one in the logs, it means the router wasn’t there:

The auditors are right in identifying this as an important issue. If there were such a router, then this would cast doubt whether the network was “airgapped”.

Note that if such a router did exist, it would almost certainly be a NAT. This would still offer some firewall protection, just not as strong as an air gap.

7.5.4 Anonymous Logins

They see something in the security logs they don’t understand, and blame Maricopa’s lack of network data ("the routers") for their inability to explain it.

This is an extraordinarily inappropriate claim, based not on expert understanding of what they see in the logs, but complete ignorance. There’s no reason to believe that getting access to Maricopa Count network logs would explain what’s going on here.

This demonstrates they are on a phishing expedition, and that everything they see that they can’t explain is used as evidence of a conspiracy, either of Maricopa to withhold data, or of election fraud.

The Dominion suite of applications and services is oddly constructed and will produce anomalies. Comparing against a general Windows server not running Dominion’s suite is meaningless.

7.5.5 Dual Boot System Discovered

The auditors claim something about “dual-homed hosts” or “jump-boxes”. That’s not how these terms are normally used. These terms normally refer to a box with access to two separate networks, not a box with two separate operating systems.

This requires no nefarious explanation. This is commonly seen in corporate networks, either because somebody simply added a new drive to re-install the operating-system, or repurposed an old drive from another system as a data drive, and simply forgot to wipe it. The BIOS points to one they intend to boot from and ignore the fact that the other can also boot.

There are endless non-nefarious explanations for what is seen here that doesn’t require a nefarious one. It’s not even clear its a failure of their build process, which focuses on what’s on the boot drive and not what’s on other drives in the system.

7.5.6 EMS Operating System Logs Not Preserved

It is true the EMS operating-system logs are not preserved (well, generally not preserved). By this I refer to the generic Windows logs, the same logs that your own Windows desktop keeps.

The auditors falsely claim that this violates the law. This is false. The “electron records” laws don’t cover the operating-system. The laws instead are intended to preserve the records of the election software running on top of the operating-system, not those of the operating-system itself.

This issue has long been known. You don’t need an auditor’s report to tell you that these logs aren’t generally preserved – everyone has known this for a long time, including those who certified Dominion.

The subtext of this claim is the continued argument by Republicans that the fact they can’t find evidence for 2020 election fraud is because key data is missing. That’s the argument of Tina Peters, the former clerk of a county in the neighboring state of Colorado, who claims their elections cannot be audited because they don’t have the Windows operating-system logs.

It’s not true. System logs are as likely to cause confusion, as they do above with the “anonymous logins” issue. They are unlikely to provide proof of votes being flipped in a hack. If there was massive fraud, as detected by recounts of paper ballots, I’d certainly want such system logs to search for how it happened. But I wouldn’t use such logs in order to audit the vote.

Note that the description of “deleting” log entries by overfilling the logs is wrong. If it were important to preserve such logs, then they would be copied right after the election. They wouldn’t be left to rot on the boot drive for months afterwards.

As a forensics guy, I would certainly support the idea that Dominion should both enable more logs and preserve them after each election. They don’t require excessive storage and can be saved automatically in the last phase of an election. But their lack really isn’t all that important, they are mostly just full of junk.

Conclusion

We live in a pluralistic democracy, meaning there are many centers of power, each competing with each other. It's inherently valid for one side to question and challenge the other side. But this can go too far, to the point where you are challenging the stability of our republic.

The Republican party is split. Some are upholding that principle of pluralism, wanting to make sure future elections are secure and fair. Others are attacking that principle, challenging the peaceful transfer of power in the last election with baseless accusations of fraud.

This split is seen in Arizona, where Republicans have demanded an audit by highly partisan auditors. An early draft of their report straddles that split, containing some reasonable attempt to create recommendations for future elections, while simultaneous providing fodder for the other side to believe the last election was stolen.

A common problem with auditors is that when they can’t find the clear evidence they were looking for, the fill their reports with things they don’t understand. I think I see that here. The auditors make technical errors in ways that question their competence, but that’s likely not true. Instead, they kept searching past where they were strong into areas where they were weak, looking for as much dirt as possible. Thus, in this report, we see where they are technically weak.

Trumpists, meaning those attacking the peaceful transfer of power with baseless accusations of fraud, will certainly use this report to champion their cause, despite the headline portion that confirms the vote count. But for the rest of us, we should welcome this report. Elections do need to be fixed, and while it’s unlikely we’ll fix them in the ways suggested in this report, it will add visibility into the process which we can use to debate improvements.

This blogpost is only a first draft. While the technical bits in section 7 look fairly straightforward to me, I'm guessing that people who don't understand them will come up with weird conspiracy-theories about them. Thus, I'm guessing I'll have to write another blogpost in a week debunking some of the crazier ideas.