Friday, October 03, 2014

Reading the Silk Road configuration

Many of us believe it wasn't the FBI who discovered the hidden Silk Road server, but the NSA (or other intelligence organization). We believe the FBI is using "parallel construction", meaning creating a plausible story of how they found the server to satisfy the courts, but a story that isn't true.

Today, Brian Krebs released data from the defense team that seems to confirm the "parallel construction" theory. I thought I'd write up a technical discussion of what was found.

The Tarbell declaration

A month ago, the FBI released a statement from the lead investigator, Christopher Tarbell, describing how he discovered the hidden server ("the Tarbell declaration"). This document had four noticeable defects.

The first is that the details are vague. It is impossible for anybody with technical skill (such as myself) to figure out what he did.

The second problem is that some of the details are impossible, such as seeing the IP address in the "packet headers".

Thirdly, his saved none of the forensics data. You'd have thought that had this been real, he would have at least captured packet logs or even screenshots of what he did. I'm a technical blogger. I document this sort of thing all the time. It's not hard for me, it shouldn't be hard for the FBI when it's the cornerstone of the entire case.

Lastly, Tarbell doesn't even deny it was parallel construction. A scenario of an NSA agent showing up at the FBI offices and opening a browser to the IP address fits within his description of events.

I am a foremost Internet expert on this sort of thing. I think Christopher Tarbell is lying.

The two servers involved

There were two serves involved.

The actual Tor "onion" server ran on a server in Germany at the IP address This was the front-end server.

The Silk Road data was held on a back-end server in Iceland at the IP address This is the server Tarbell claims to have found.

The data dumped today on Brian Krebs' site is configuration and log files from the second server.

The Icelandic configuration

The Icelandic backend had two "sites", one on HTTP (port 80) running the phpmyadmin pages, and a second on HTTPS (port 443) for communicating the Silk Road content to the German onion server.

The HTTP (port 80) configuration is shown below. Because this requires "basic authentication", Tarbell could not have accessed the server on this port.

However, the thing to note about this configuration is that "basic" authentication was used over port 80. If the NSA were monitoring links to/from Iceland, they could easily have discovered the password and used it to log onto the server. This is basic cybersecurity, what the "Wall of Sheep" at DefCon is all about.

The following picture shows the configuration of the HTTPS site.

Notice firstly that the "listen 443" specifies only a port number and not an IP address. Consequently, anybody on the Internet could connect to the server and obtain its SSL certificate, even if it cannot get anything but an error message from the web server. Brian Krebs quotes Nicholas Weaver as claiming "This suggests that the Web service specifically refuses all connections except from the local host and the front-end Web server". This is wrong, the web server accept all TCP connections, though it may give a "403 forbidden" as the result.

BTW: one plausible way of having discovered the server is to scan the entire Internet for SSL certificates, then correlate information in those certificates with the information found going across the Tor onion connection.

Next is the location information that allows only localhost, the German server, and then denies everything else ("deny all"). As mentioned above, this doesn't prevent the TCP connection, but does produce a "403 forbidden" error code.

However, there is a flaw: this configuration is overridden for PHP files in the next section down. I've tested this on my own server. While non-PHP files are not accessible on the server, anything with the .php file extension still runs for everyone.

Worse yet, the login screen uses "/index.php". The rules above convert an access of "/" automatically to "/index.php". If indeed the server has the file "/var/www/market/public/index.php", then Tarbell's explanation starts to make sense. He's still missing important details, and of course, there is no log of him having accessed the server this way,, but this demonstrates that something like his description isn't impossible. One way this could have been found is by scanning the entire Internet for SSL servers, then searching for the string "Silkroad" in the resulting webpage.

The log files

The FBI imaged the server, including all the log files. Typical log entries looked like the following: - - [14/Jul/2013:06:55:33 +0000] "GET /orders/cart HTTP/1.0" 200 49072 "http://silkroadvb5piz3r.onion/silkroad/item/0f81d52be7" "Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20100101 Firefox/17.0"

Since the defense could not find in the logfiles where Tarbell had access the system, the prosecutors helped them out by pointing to entries that looked like the following: - - [11/Jun/2013:16:58:36 +0000] "GET / HTTP/1.1" 200 2616 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36" - - [11/Jun/2013:16:58:36 +0000] "GET
/phpmyadmin.css.phpserver=1&lang=en&collation_connection=utf8_general_ci&token=451ca1a827cda1c8e80d0c0876e29ecc&js_frame=right&nocache=3988383895 HTTP/1.1" 200 41724 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36"

However, these entries are wrong. First, they are for the phpmyadmin pages and not the Silk Road login pages, so they are clearly not the pages described in the Tarbell declaration. Second, they return "200 ok" as the error code instead of a "401 unauthorized" login error as one would expect from the configuration. This means either the FBI knew the password, or the configuration has changed in the meantime, or something else is wrong with the evidence provided by the prosecutors.


As an expert in such topics as sniffing passwords and masscaning the Internet, I know that tracking down the Silk Road site is well within the NSA's capabilities. Looking at the configuration files, I can attest to the fact that the Dread Pirate Roberts sucked at op-sec.

As an expert, I know the Tarbell declaration is gibberish. As an expert reading the configuration and logs, I know that it doesn't match the Tarbell declaration. That's not to say that the Tarbell declaration has been disproven, it's just that "parallel construction" is a better explanation for what's going on than Tarbell actually having found the Silk Road server on his own.


dionyziz said...


Unknown said...

HTTP status code 200 is OK, which means success, not an error. The whole family of 2xx codes are successful statuses. Someone calling himself an expert should know this.

Robert Graham said...

It's a "200 ok" status code when a "401 unauthorized" is what the configuration file indicates should've been the status code. Thus, there is a mismatch between what the configuration says and what the logs say. This implies that something else is going on here, such as maybe the FBI having a valid password, or maybe the password file is misconfigured, or maybe the configuration has changed, or something. Thus, "200 ok" indicates something is wrong with the data that we see.

Jakub Vrána said...

Did you notice the IP address on the phpMyAdmin screenshot?

It's 192.168.*. Do you have some explanation for that?

Unknown said...

Uhm. Well, why has no one considered the possibility that the backend server leaked its ip as part of an error message?

Apache may in some cases put a footer in http 500 errors complete with its nic ip (which, for servers, is often also their public ip).
Or, it could fall back to the ip reverse dns, or the server hostname if permitted, which could well be set to its ip (common with 'administration panels').

Sure, in this case the server was nginx, not apache, so we don't know the technique used by the fbi for sure, and can't exclude parallel construction.

but we also don't know the full configuration of silk road.

why wouldn't it have been possible that something else generated an error page with the backends' local ip included?

eg, a faulty php script.

do we know which external CAPTCHA service was used by silkroad? it could have leaked the server ip to the client, or the entity behind the CAPTCHA service may have been forced to turn over records associating individual captchas entered with the server that made the request.

do we know what was running on the frontend?

squid for example may put http "Via:" headers in the response by default, thus revealing to the client the IP of every proxy in the chain...

Wiener said...

A couple of things.

1) Wouldn't you get a 401 even if you knew the password? In my (admittedly distant and fuzzy) experience, even if you KNOW the password and have successfully authenticated, the server always sends back a 401 and your browser always sends back the authentication.

2) They provided logs for the port 80 server and not the 443 one, it seems.

cschramm said...

"Second, they return "200 ok" as the error code instead of a "401 unauthorized" login error as one would expect from the configuration. This means either the FBI knew the password"

No! The second dash (third column) in the log would be the username then.

Unknown said...

I swear I read that they had found the Silk Road due to a backdoor vulnerability that was found in the version of Firefox that was being distributed for TOR access? Those who failed to update it were then targeted by the FBI and had their systems infected to monitor everything going on.