Tuesday, November 01, 2016

Debunking Trump's "secret server"

According to this Slate article, Trump has a secret server for communicating with Russia. Even Hillary has piled onto this story.

This is nonsense. The evidence available on the Internet is that Trump neither (directly) controls the domain "trump-email.com", nor has access to the server. Instead, the domain was setup and controlled by Cendyn, a company that does marketing/promotions for hotels, including many of Trump's hotels. Cendyn outsources the email portions of its campaigns to a company called Listrak, which actually owns/operates the physical server in a data center in Philidelphia.


In other words,  Trump's response is (minus the political bits) likely true, supported by the evidence. It's the conclusion I came to even before seeing the response.

When you view this "secret" server in context, surrounded by the other email servers operated by Listrak on behalf of Cendyn, it becomes more obvious what's going on. In the same Internet address range of Trump's servers you see a bunch of similar servers, many named [client]-email.com. In other words, trump-email.com is not intended as a normal email server you and I are familiar with, but as a server used for marketing/promotional campaigns.



It's Cendyn that registered and who controls the trump-email.com domain, as seen in the WHOIS information. That the Trump Organization is the registrant, but not the admin, demonstrates that Trump doesn't have direct control over it.

When the domain information was changed last September 23, it was Cendyn who did the change, not the Trump Organization. This link lists a bunch of other hotel-related domains that Cendyn likewise controls, some Trump related, some related to Trump's hotel competitors, like Hyatt and Sheraton.

Cendyn's claim they are reusing the server for some other purpose is likely true. If you are an enterprising journalist with $399 in your budget, you can find this out. Use the website http://reversewhois.domaintools.com/ to get a complete list of the 641 other domains controlled by Cendyn, then do an MX query for each one to find out which of them is using mail1.trump-email.com as their email server.


This is why we can't have nice things on the Internet. Investigative journalism is dead. The Internet is full of clues like this if only somebody puts a few resources into figuring things out. For example, organizations that track spam will have information on exactly which promotions this server has been used for in the recent past. Those who operate public DNS resolvers, like Google's 8.8.8.8, OpenDNS, or Dyn, may have knowledge which domain was related to mail1.trump-email.com.

Indeed, one journalist did call one of the public resolvers, and found other people queried this domain than the two listed in the Slate story -- debunking it. I've heard from other DNS malware researchers (names remain anonymous) who confirm they've seen lookups for "mail1.trump-email.com" from all over the world, especially from tools like FireEye that process lots of spam email. One person claimed that lookups started failing for them back in late June -- and thus the claim of successful responses until September are false. In other words, the "change" after the NYTimes queried Alfa Bank may not be because Cendyn (or Trump) changed anything, but because that was the first they checked and noticed that lookup errors were happening.

Since I wrote this blog post at midnight, so I haven't confirmed this with anybody yet, but there's a good chance that the IP address 66.216.133.29 has continued to spew spam for Trump hotels during this entire time. This would, of course would generate lookups (both reverse and forward). It seems like everyone who works for IT for a large company should be able to check their incoming email logs and see if they've been getting emails from that address over the last few months. If you work in IT, please check your logs for the last few months and Tweet me at @erratarob with the results, either positive or negative.

And finally, somebody associated with Alfa Bank IT operations confirms that executives like to stay at Trump hotels all the time (like in Vegas and New York), and there was a company function one of Trump's golf courses. In other words, there's good reason for the company to get spam from, and need to communicate with, Trump hotels to coordinate events.

And so on and so forth -- there's a lot of information out there if we just start digging.

Conclusion

That this is just normal marketing business from Cendyn and Listrak is the overwhelming logical explanation for all this. People are tempted to pull nefarious explanations out of their imaginations for things they don't understand. But for those of us with experience in this sort of thing, what we see here is a normal messed up marketing (aka. spam) system that the Trump Organization doesn't have control over. Knowing who owns and controls these servers, it's unreasonable to believe that Trump is using them for secret emails. Far from "secret" or "private" servers as Hillary claims, these servers are wide open and obvious.

This post provides a logic explanation, but we can't count on this being provably debunked until those like Dyn come forward, on the record, and show us lookups that don't come from Alfa Bank. Or, those who work in big companies can pull records from their incoming email servers, to show that they've been receiving spam from that IP address over the last few months. Either of these would conclusively debunk the story.



But experts say...

But the article quotes several experts confirming the story, so how does that jibe with this blog post. The answer is that none of the experts confirmed the story.

Read more carefully. None of the identified experts confirmed the story. Instead, the experts looked at pieces, and confirmed part of the story. Vixie rightly confirmed that the pattern of DNS requests came from humans, and not automated systems. Chris Davis rightly confirmed the server doesn't look like a normal email server.

Neither of them, however, confirmed that Trump has a secret server for communicating with the Russians. Both of their statements are consistent with what I describe above -- that's it's a Cendyn operated server for marketing campaigns independent of the Trump Organization.



Those researchers violated their principles

The big story isn't the conspiracy theory about Trump, but that these malware researchers exploited their privileged access for some purpose other than malware research.

Malware research consists of a lot of informal relationships. Researchers get DNS information from ISPs, from root servers, from services like Google's 8.8.8.8 public DNS. It's a huge privacy violation -- justified on the principle that it's for the general good. Sometimes the fact that DNS information is shared is explicit, like with Google's service. Sometimes people don't realize how their ISP shares information, or how many of the root DNS servers are monitored.

People should be angrily calling their ISPs and ask them if they share DNS information with untrustworthy researchers. People should be angrily asking ICANN, which is no longer controlled by the US government (sic), whether it's their policy to share DNS lookup information with those who would attempt to change US elections.

There's not many sources for this specific DNS information. Alfa Bank's servers do their own resolution, direction from the root on down. It's unlikely they were monitoring Alfa Bank's servers directly, or monitoring Cendyn's authoritative servers. That means some sort of passive DNS on some link in between, which is unlikley. Conversely, they could be monitoring one of the root domain servers -- but this monitoring wouldn't tell them the difference between a successful or failed lookup, which they claim to have. In short, of all the sources of "DNS malware information" I've heard about, none of it would deliver the information these researchers claim to have (well, except the NSA with their transatlantic undersea taps, of course).

Update: this tweet points out original post mentions getting data from "ams-ix23" node, which hints at AMS-IX, Amsterdam InterXchange, where many root server nodes are located.

36 comments:

Your favorite uncle said...

Is the admin any relation to the GOP candidate running against Trump in Utah? Inquiring minds need to know, lol.

Your favorite uncle said...

Is the admin any relation to the GOP candidate running against Trump in Utah? Inquiring minds need to know, lol.

yowdyyay said...

One part of the story which seemed a bit hard to explain is that, according to the DNS logs presented, the Trump-Emails.com domain was only resolved by Spectrum Health and Alfa Bank.

Now, there's a report that Dyn has DNS logs showing queries for the domain from other hosts. I suspect Dyn isn't going to go into a lot of detail and start releasing their customers' DNS logs (except to the DNS cabal who shared around these logs in the first place, a conversation about logging and privacy vs security we all need to have pretty soon) to the public, and other DNS providers surely have even larger logs to search. Assuming the logs used for the Slate story were genuine, I have to ask how wide of a sample of DNS traffic they really represent and whether they happen to only include the queries between these entities by happenstance.

The other odd detail is that the story quotes Hope Hicks, of the Trump campaign, as saying that the server hasn't been used by the Trump Organization for six years. If this is the case, why the DNS queries and why did Spectrum say they received spam from Trump through the server? The simplest explanation, of course, which is quite possible without any sort of Russian plot, is that Hicks is simply wrong and they've used it for some hotel marketing emails.

maniacdev said...

They also spelled Organization wrong.

maniacdev said...
This comment has been removed by the author.
maniacdev said...

yowdyyay - that looks like the bank is simply scanning to see if the e-mail server is real.

George Erhard said...

The questions still remain, however.

Why just these systems interested in each other? A mass mailing system is useless if you don't use it to send mail out to your marketing contact lists. It's a big computer engaged in digital navel-gazing. It was UP, but as far as marketing was concerned, it was on a siding. As an IT break-fix tech, I get holy hell for a server sitting idle, especially something so directly involved with Making Money as a marketing email source. So there's THAT question.

Why more traffic at certain key points of the campaign? Again, marketing doesn't stop or start based on current events - there's always stuff to send to the rubes. Yet this server, configured and used as an email firehose, acted like a water fountain. Huh??

And why, when Alfa Bank was asked about this traffic pattern, was the Trump server summarily deleted from 2 of 3 NS systems, and only had the A record excised from NS #3 (and apparently still gives a partial DNS response)? That would indicate that someone at Alfa called either Trump's org or the marketing services company and said "shut it down". Again, that doesn't really make much sense from a mass-mailing perspective.

Perhaps this is a lot of brouhaha over nothing. However, there's enough discrepancies in the traffic usage to indicate it was NOT acting as an email mass mailer.

yowdyyay said...

George: one possible explanation is that the NYT reporter talked to Alfa Bank, someone from Alfa Bank then called up someone at the Trump Organization to say "I don't know what you guys are doing, but there's a reporter asking us weird questions about your email server," causing the Trump folks to call Cendyn, causing them to make the DNS change to immediately appease their angry and confused client. When the server, now with a new hostname, sent another bit of email to someone at Alfa Bank, that triggered a DNS query to the new hostname in response.

And if the server belongs to a third-party email marketing outfit, it may well not have been sitting idle at all: Cendyn claims it was shared with other clients, which is plausible, and Spectrum says they got Trump Hotel spam from it, so it may have still been in use despite Hicks' claim.

homer jones said...

Rubbish excuses and explanations.
Whoever owns the domain controls everything else.

Will Saturn said...
This comment has been removed by the author.
andy woods said...

Knowing making false public assumptions would put and end to your carrier. Means these guys must be pretty sure of their analysis. The servers reconnecting after the shut down and change says intentional traffic.
I do have issue with the assumption that is being implied as to the purpose of the server. There are legitimate reasons for secret transactions. Not wanting the competition to know what you are doing has merit. But adding this bit of information with the other things that are suspect. It is odd how there things exist.

Radical Truth US said...

This is really good work, man. I appreciate you doing this!

Unknown said...

This is an extreme amount of speculation that doesn't actually line up with the facts.

Fact: No one operates a marketing server that way. Gross ignorance, negligence, and cluelessness are not valid explanations for the traffic on that server, or the behavior after its discovery, including setting up a new server with the same very narrow communications parameters.

There is no reason for Cendyn to do this autonomously. And I question whether it is possible for them to establish a connection to this new "marketing server" without any conversations with, or participation or awareness from the server owners, despite the fact that they dismantled and installed a brand new system.

Look, if this was on the level, Hope Hicks and other members of the Trump campaign wouldn't be lying and saying the server wasn't even used for five years. They would say it's a marketing server and move on.

Unknown said...

This is an extreme amount of speculation that doesn't actually line up with the facts.

Fact: No one operates a marketing server that way. Gross ignorance, negligence, and cluelessness are not valid explanations for the traffic on that server, or the behavior after its discovery, including setting up a new server with the same very narrow communications parameters.

There is no reason for Cendyn to do this autonomously. And I question whether it is possible for them to establish a connection to this new "marketing server" without any conversations with, or participation or awareness from the server owners, despite the fact that they dismantled and installed a brand new system.

Look, if this was on the level, Hope Hicks and other members of the Trump campaign wouldn't be lying and saying the server wasn't even used for five years. They would say it's a marketing server and move on.

andy woods said...

What makes me question the real intent for the server. The responses that try and explain and debunk the analysis. THEY all state the narrative but none address the concerns. The two big ones. The limited access and the shut down and recognect. That it is not evident what it does. Who locks down a mass mailer

Ivo Blaauw said...

"I've heard from other DNS malware researchers (names remain anonymous)..." this is so contrary to things you've previously said about journalistic accountability (NYTimes vs. DNCleaks ).
"And finally, somebody associated with..." That's also rather vague.

j said...
This comment has been removed by the author.
Kalashnikov said...

If someone wanted to communicate with the Russians, it would be easier to just make a generic gmail account and encrypt the content with PGP. You know, the sort of thing team hillary was too dimwitted to do.

damian said...

While I agree with your broader point about the source of the data being curious, it's worth noting that Google explicitly does *not* share DNS data from 8.8.8.8, as you can see in the privacy policy: https://developers.google.com/speed/public-dns/privacy

Perhaps this was OpenDNS, whose privacy policy is a bit less clear? http://www.opendns.com/privacy-policy/

Adam said...

I nearly had to stop here: "That the Trump Organization is the registrant, but not the admin, demonstrates that Trump doesn't have direct control over it." This demonstrates profound ignorance of the way domains are registered. Then, I wonder whether the author of this blog thoroughly read the original article. He says, "one journalist did call one of the public resolvers, and found other people queried this domain than the two listed in the Slate story -- debunking it", while the original article notes that "Eighty-seven percent of the DNS lookups involved the two Alfa Bank servers." At no point did the author say that Alfa bank or any others were the ONLY entities looking for DNS resolution. Indeed, Paul Vixie, labeled a DNS expert by the Slate author noted, "The data has got the right kind of fuzz growing on it". I'm afraid the author of this blog post knows, as they say, enough to be dangerous, but not enough to be authoritative. Sadly, huge gaps of assumption are taking place in his knowledge.

j said...
This comment has been removed by the author.
j said...
This comment has been removed by the author.
judy said...

And the coincidences to suspect !!!

judy said...

Thanks for that checkpoint Adam !!!

Kellen Gracey said...

Paul Manafort resigned August 19th.

How is nobody treating this as politically relevant? Look at the spikes in activity leading up to the announcement of his resignation and a ton of investigations into his Russian ties becoming public.

Bunch of people squinting at the chart and laughing about it not lining up with politically relevant events. But they aren't considering politically relevant events that are not overlaid onto those DNS queries.

judy said...

Great deductions George!
I don't think this should be easily dismissed. Sounds a whole lot sketchier than Hillarys emails !

j said...
This comment has been removed by the author.
Unknown said...

Marketing vs. compromising national security? Only smart people know the correct answer there. If you think marketing = "collusion with the Russians", then you're not one of them.

Jkirk3279 said...

“In other words, trump-email.com is not intended as a normal email server you and I are familiar with, but as a server used for marketing/promotional campaigns.”

Except for one problem.

A server used for Marketing/Promotional campaigns would be running constantly, blasting out bits.

This one is sitting there quietly, sending nothing most of the time, and yes, it’s being “pinged” randomly by Alfa Bank in Russia.

When? During the periods coincident with office hours in Russia, and office hours in New York.

Dude, listen. You can’t put lipstick on this pig.

This is a Server configured so NO outside traffic will get a response EXCEPT for Alfa Bank and Spectrum Health.

Anybody else gets an error message.

I tried to explain to someone, “if Bill Clinton wanted to meet secretly with someone, he’d pick a less obvious rendezvous than the airport tarmac in BROAD daylight”.

In fact, IF someone wanted to set up a dead drop isolated from the rest of the Internet, this is how they’d do it.

Emails sent directly between two servers that can’t talk to any other IP address.

Encrypted packets broken up, sent over the Internet, re-assembled at the destination; with just a little elementary encryption, unreadable by anyone without the cipher.

The fact that someone noticed the DNS logs was luck on the scale of hitting two “hole in ones” in a single day.

j said...
This comment has been removed by the author.
j said...
This comment has been removed by the author.
Johnny Rhetoric said...

It is pretty apparent that Trump was communicating with somebody at the Russian bank. It was set up to do just that ... and not much else. When the secret setup came to light, it was reconfigured.

Trump has said he has no ties to Russia, no loans from Russian oligarchs, and so on. I don't believe him, as his son has mentioned doing business in Russia.

Does a Russian bank own Trump?

Richard Clayton said...

The people commenting that servers do not sit idle appear to believe that there is a one-to-one relationship between IP addresses and expensive boxes. This just isn't the case.

In particular, since blacklisting tends to be done (at first anyway) per /32 there's much to be said for senders giving individual IPs to each client. When they screw up then other customers are not affected by the listing. However, you can use the same box for all of them -- its all just a config file.

That said, there does not seem to be any evidence for mass mailing by the relevant IP address for many months. Yes it seems to have been used for more substantial mailshots in the past, but more recently not so much apart from to a handful of places.

Gee said...

The 'unknown investigator' was Paul Vixie. Among other credentials on Wikipedia, he is a trustee for American Registry for Internet Numbers Ltd.

j said...
This comment has been removed by the author.
j said...
This comment has been removed by the author.