Wednesday, July 27, 2016

NYTimes vs. DNCleaks

People keep citing this New York Times article by David Sanger that attributes the DNCleaks to Russia. As I've written before, this is propaganda, not journalism. It's against basic journalistic ethics to quote anonymous "federal officials" in a story like this. The Society of Professional Journalists repudiates this [1] [2]. The NYTime's own ombudsman has itself criticized David Sanger for this practice, and written guidelines to specifically ban it.

Quoting anonymous federal officials is great, when they disagree with government, when revealing government malfeasance, when it's something that people will get fired over.

But the opposite is happening here. It's either Obama himself or some faction within the administration that wants us to believe Russia is involved. They want us to believe the propaganda, then hide behind anonymity so we can't question them. This evades obvious questions, like whether all their information comes from the same public sources that already point to Russia, or whether they have their own information from the CIA or NSA that points to Russia.

Everyone knows the Washington press works this way, and that David Sanger in particular is a journalistic whore. The NetFlix series House of Cards portrays this accurately in its first season, only "Zoe Barnes" is "David Sanger". In exchange for exclusive access to stories, the politician gets to plant propaganda when it suits his purpose.

All this NYTimes article by Sanger tells us is that some faction within the administration wants us to believe this, not whether it's true. That's not surprising. There are lots of war hawks that would want us to believe this. There are also lots who support Hillary over Trump -- who want us to believe that electing Trump plays into Putin's hands. Of course David Sanger would write such a story quoting anonymous sources, like he does after every such incident. You can pretty much write the story yourself.

Thus, we should fully discount Sanger's story. If government officials are willing to come forward an be named, and be held accountable for the information, then we should place more faith in them. As long as a faithless journalists protects them with anonymity, we shouldn't believe anything they say.

Friday, July 22, 2016

My Raspeberry Pi cluster

So I accidentally ordered too many Raspberry Pi's. Therefore, I built a small cluster out of them. I thought I'd write up a parts list for others wanting to build a cluster.

To start with is some pics of the cluster What you see is a stack of 7 RPis. At the bottom of the stack is a USB multiport charger and also an Ethernet hub. You see USB cables coming out of the charger to power the RPis, and out the other side you see Ethernet cables connecting the RPis to a network. I've including the mouse and keyboard in the picture to give you a sense of perspective.

Here is the same stack turn around, seeing it from the other side. Out the bottom left you see three external cables, one Ethernet to my main network and power cables for the USB charger and Ethernet hub. You can see that the USB hub is nicely tied down to the frame, but that the Ethernet hub is just sort jammed in there somehow.

The concept is to get things as cheap as possible, on per unit basis. Otherwise, one might as well just buy more expensive computers. My parts list for a 7x Pi cluster are:

$35.00/unit Raspberry Pi
 $6.50/unit stacking case from Amazon
 $5.99/unit micro SD flash from Newegg
 $4.30/unit power supply from Amazon
 $1.41/unit Ethernet hub from Newegg
 $0.89/unit 6 inch and 1-foot micro USB cable from Monoprice
 $0.57/unit 1 foot Ethernet cable from Monoprice

...or $54.65 per unit (or $383 for entire cluster), or around 50% more than the base Raspberry Pis alone. This is getting a bit expensive, as Newegg. always has cheap Android tablets on closeout for $30 to $50.

So here's a discussion of the parts.

Raspberry Pi 2

These are old boards I'd ordered a while back. They are up to RPi3 now with slightly faster processors and WiFi/Bluetooth on board, neither of which are useful for a cluster. It has four CPUs each running at 900 MHz as opposed to the RPi3 which has four 1.2 GHz processors. If you order a Raspberry Pi now, it'll be the newer, better one.

The case

You'll notice that the RPi's are mounted on acrylic sheets, which are in turn held together with standoffs/spaces. This is a relatively expensive option.

A cheaper solution would be just to buy the spaces/standoffs yourself. They are a little hard to find, because the screws need to fit the 2.9mm holes, where are unusually tiny. Such spaces/standoffs are usually made of brass, but you can also find nylon ones. For the ends, you need some washers and screws. This will bring the price down to about $2/unit -- or a lot cheaper if you are buying in bulk for a lot of units.

The micro-SD

The absolute cheapest micro SD's I could find were $2.95/unit for 4gb, or half the price than the ones I bought. But the ones I chose are 4x the size and 2x the speed. RPi distros are getting large enough that they no longer fit well on 4gig cards, and are even approaching 8gigs. Thus, 16gigs are the best choice, especially when I could get hen for $6/unit. By the time you read this, the price of flash will have changed up or down. I search on Newegg, because that's the easiest way to focus on the cheapest. Most cards should work, but check to avoid any known bad chips.

Note that different cards have different speeds, which can have a major impact on performance. You probably don't care for a cluster, but if you are buying a card for a development system, get the faster ones. The Samsung EVO cards are a good choice for something fast.

USB Charging Hub

What we want here is a charger not a hub. Both can work, but the charger works better.

A normal hub is about connecting all your USB devices to your desktop/laptop. That doesn't work for this RPi -- the connector is just for power. It's just leveraging the fact that there's already lots of USB power cables/chargers out there, so that it doesn't have to invite a custom one.

USB hubs an supply some power to the RPi, enough to boot it. However, under load, or when you connect further USB devices to the RPi, there may not be enough power available. You might be able to run a couple RPis from a normal hub, but when you've got all seven running (as in this stack), there might not be enough power. Power problems can outright crash the devices, but worse, it can lead to things like corrupt writes to the flash drives, slowly corrupting the system until it fails.

Luckily, in the last couple years we've seen suppliers of multiport chargers. These are designed for families (and workplaces) that have a lot of phones and tablets to charge. They can charge high-capacity batteries on all ports -- supplying much more power than your RPi will ever need.

If want to go ultra cheaper, then cheap hubs at $1/port may be adequate. Chargers cost around $4/port.

The charger I chose in particular is the Bolse 60W 7-port charger. I only need exactly 7 ports. More ports would be nicer, in case I needed to power something else along with the stack, but this Bolse unit has the nice property that it fits snugly within the stack. The frame came with extra spacers which I could screw together to provide room. I then used zip ties to hold it firmly in place.

Ethernet hub

The RPis only have 100mbps Ethernet. Therefore, you don't need a gigabit hub, which you'd normally get, but can choose a 100mbps hub instead: it's cheaper, smaller, and lower power. The downside is that while each RPi only does 100-mbps, combined they will do 700-mbps, which the hub can't handle.

I got a $10 hub from Newegg. As you can see, it fits within the frame, though not well. Every gigabit hub I've seen is bigger and could not fit this way.

Note that I have a couple extra RPis, but I only built a 7-high stack, because of the Ethernet hub. Hubs have only 8 ports, one of which is needed for the uplink. That leaves 7 devices. I'd have to upgrade to an unwieldy 16-port hub if I wanted more ports, which wouldn't fit the nice clean case I've got.

For a gigabit option, Ethernet switches will cost between $23 and $35 dollars. That $35 option is a "smart" switch that supports not only gigabit, but also a web-based configuration tool, VLANs, and some other high-end features. If I paid more for a switch, I'd probably go with the smart/managed one.

Cables (Ethernet, USB)

Buying cables is expensive, as everyone knows whose bought an Apple cable for $30. But buying in bulk from specialty sellers can reduce the price to under $1/cable.

The chief buy factor is length. We want short cables that will just barely be long enough. in the pictures above, the Ethernet cables are 1-foot, as are two of the USB cables. The colored USB cables are 6-inches. I got these off Amazon because they looked cool, but now I'm regretting it.

The easiest, cheapest, and highest quality place to buy cables is It allows you to easily select the length and color.

To reach everything in this stack, you'll need 1-foot cables. Though, 6-inch cables will work for some (but not all) of the USB devices. Although, instead of putting the hubs on the bottom, I could've put them in the middle of the stack, then 6-inch cables would've worked better -- but I didn't think that'd look as pretty. (I chose these colored cables because somebody suggested them, but they won't work for the full seven-high tower).

Power consumption

The power consumption of the entire stack is 13.3 watts while it's idle. The Ethernet hub by itself was 1.3 watts (so low because it's 100-mbps instead of gigabit).

So, round it up, that's 2-watts per RPi while idle.

In previous power tests, it's an extra 2 to 3 watts while doing heavy computations, so for the entire stack, that can start consuming a significant amount of power. I mention this because people think terms of a low-power alternative to Intel's big CPUs, but in truth, once you've gotten enough RPis in a cluster to equal the computational power of an Intel processor, you'll probably be consuming more electricity.

The operating system

I grabbed the lasted Raspbian image and installed it on one of the RPis. I then removed it, copied the files off (cp -a), reformatted it to use the f2fs flash file system, then copied the files back on. I then made an image of the card (using dd), then wrote that image to 6 other cards. I then I logged into each one ad renamed them rpi-a1, ..., rpi-a7. (Security note: this means they all have the same SSH private key, but I don't care).

About flash file systems

The micro SD flash has a bit of wear leveling, but not enough. A lot of RPi servers I've installed in the past have failed after a few months with corrupt drives. I don't know why, I suspect it's because the flash is getting corrupted.

Thus, I installed f2fs, a wear leveling file system designed especially for this sort of situation. We'll see if that helps at all.

One big thing is to make sure atime is disabled, a massively brain dead feature inherited from 1980s Unix that writes to the disk every time you read from a file.

I notice that the green LED on the RPi, indicating disk activity, flashes very briefly once per second, (so quick you'll miss it unless you look closely at the light). I used iotop -a to find out what it is. I think it's just a hardware feature and not related to disk activity. On the other hand, it's worth tracking down what writes might be happening in the background that will affect flash lifetime.

What I found was that there is some kernel thread that writes rarely to the disk, and a "f2fs garbage collector" that's cleaning up the disk for wear leveling. I saw nothing that looked like it was writing regularly to the disk.

What to use it for?

So here's the thing about an RPi cluster -- it's technically useless. If you run the numbers, it's got less compute power and higher power consumption than a normal desktop/laptop computer. Thus, an entire cluster of them will still perform slower than laptops/desktops.

Thus, the point of a cluster is to have something to play with, to experiment with, not that it's the best form of computation. The point of individual RPis is not that they have better performance/watt -- but that you don't need as much performance but want a package with very low watts.

With that said, I should do some password cracking benchmarks with them, compared across CPUs and GPUs, measuring power consumption. That'll be a topic for a later post.

With that said, I will be using these, though as individual computers rather than as a "cluster". There's lots of services I want to run, but I don't want to run a full desktop running VMware. I'd rather control individual devices.


I'm not sure what I'm going to do with my little RPi stack/cluster, but I wanted to document everything about it so that others can replicate it if they want to.

Thursday, June 23, 2016

Use the freakin' debugger

This post is by a guy who does "not use a debugger". That's stupid. Using a friendly source-level debugger (Visual Studio, XCode, Eclipse) to step line-by-line through working code is what separates the 10x programmers from the wannabes. Yes, it's a bit of a learning hurdle, and creating "project" files for small projects is a bit of a burden, but do it. It'll vastly improve your coding skill.

That post quotes people like Rob Pike saying that stepping line-by-line is a crutch, that instead you should be able to reason about code. And that's true, if you understand what you are doing completely.

But in the real world, you never do. Programmers are constantly forced to stretch and use unfamiliar languages. Worse yet, they are forced to use unfamiliar libraries. Documentation sucks, there's no possible way to understand APIs than to step through code -- either watching the returned values, or compiling their source and stepping into it.

As an experienced programmer, it's true I often don't step through every line. The lines I understand completely, the ones I can fully reason about, I don't bother. But the programmer spends only a small percentage of their time on things they understand -- most of the time spent coding is noodling on the things they don't understand, and that's where the debugger comes in.

And this doesn't even take into account that in the real world, where programmers spend a lot of time working on other people's code. Sometimes the only way to figure it out is to set a breakpoint and run the unit test until it reaches that point.

Programmers fetishize editors. Real programmers, those who produce a lot of code, fetishize debuggers, both the one built into the IDE for debugging working code, and also the specialized tools for diagnosing problems in buggy code.

Seriously, if you are learning to program, learn to use the debugger in the integrated IDE. Step line-by-line through every line of code, until you grok it.. Microsoft's Visual Code is a good system for debugging JavaScript (which is a good starting language to learn). You'll thank me later when you are pulling down seven figures as a 10x programmer.

Wednesday, June 22, 2016

Reverse Turing testing tech support

So I have to get a new Windows license for a new PC. Should I get Windows 10 Home or Windows 10 Professional? What's the difference?

So I google the question, which gives me this website:

Ooh, a button that says "Download Table". That's exactly what I want -- a technical list without all the fluff. I scroll down to the parts that concern me, like encryption.

Wait, what? What's the difference between "Device Encryption" and "BitLocker"? I though BitLocker was Device Encryption?? Well, the purchase screen for Windows 10 has this friendly little pop out offering to help. Of course, as a techy, I know that such things are worse than useless, but I haven't tried one in a while, so I thought if I'd see if anything changed.

So up pops a chat window and we start chatting:

So at first he says they are the same. When I press him on the difference, he then admits they are different. He can't read the document I'm reading, because it's on a non-Microsoft "third party" site. While it's true it's on "", that's still a Microsoft site, but apparently he's not allowed to access it. I appears Microsoft firewalls their access to the Internet so jerks like me can't social engineer them.

So he goes on to find other differences:

At this point, he's acting as a Markov bot, searching Microsoft's internal site with the terms I give him, then selecting random phrases to spew back at me, with no understanding. Support for TPM has nothing to do with the difference.

Finally, he admits he can't answer the question, and offers to send me to more technical people:

I know this isn't going to work, but I'm in this far, and already planning to write a blog post, I continue the game.

At this point, I've learned to be more precise in my questioning:

It takes him awhile to research the answer. During this time, with more careful sleuthing with Google, I find the real answer (below). But eventually he comes back with this:

Like the previous person, it's all gibberish. He looked up my phrases, then spewed back random sampling of tech terms.

So I did figure eventually what "Device Encryption" was. It's described in this Ars Technica post. It's designed for when Windows is installed on tablet-style devices -- things that look more like an iPad and less like a notebook computer. It has strict hardware requirements, so it's probably not going to work on a desktop or even laptop computer. It requires a TPM, SSD, no fans (at least while sleeping), and so on.

The tl;dr is for people in my situation with desktop computer, Win10-Home's "Device Encryption" won't work -- it only works for tablets. If I want full disk encryption on my desktop, I'll need "Win10-Pro".

The idea is that in the future, tech support will be replaced AI bots that use natural language processing to answer questions like. But that's what we already have: tech support search text, finds plausible answers they don't understand, and regurgitates them back at us.

In other words, when the Turing Test is finally won, it's going to be in tech support, where a well-designed bot will outperform humans on answering such questions.

Saturday, June 18, 2016

Tesla review: What you need to know about charging

Before you buy an electric car, you need to understand charging. It’s a huge deal. You think it works almost like filling the gas tank. It doesn’t. Before going on long trips, you first need to do math and a bit of planning.

The Math

Like BMW model numbers indicate engine size, Tesla model numbers indicate the size of the battery, so my "Tesla S P90D" has a 90kwh (killowatt-hour) battery, with a 286mile range. Their lowest end model is the “Tesla S 60”, which has a 60kwh hour battery, or a 208mile advertised range.

In the United States, a typical plug is a 120volt circuit with a maximum of 15amps. Doing the math, this is how long it’ll take for me to recharge the battery:

That’s right, 1.4 days (or 2.1 days for a 90kwh car). This is the absolute worse case scenario, mind you, but it demonstrates that you have to pay attention to charging. You can't simply drive up to a station, fill up the tank in a couple minutes, and drive away.

Let’s say you live in Austin, Texas, and you have a meeting in Dallas. You think that you can drive up to Dallas in your new Tesla S 60, let the car charge while you are in the meeting, and then drive home. Or, maybe you have dinner there, letting the car charge longer. Or maybe you even stay overnight.

Nope, even 24 hours later, you still might not have enough charge left to get home. At 195 miles, it's at the range of the 60kwh battery, which would take more than a day to recharge using a normal electric circuit.

Faster Charging

That was a worst case scenario. Luckily, you probably won’t be charging using a normal 120volt/15amp circuit. That’s just the emergency backup if all else fails.

In your home, for high-watt devices like ovens, air conditioners, and clothes dryers, you have higher wattage circuits. The typical max in your home will be a 240volt/50amp circuit. It has a different power connector than a normal circuit, thicker wires, and so forth. Doing the math on this sucker, you get:

For our 190 mile drive, then, you can except to drive to Dallas, charge during the meeting and dinner for 5 hours, then you’ll have enough juice to get back home.

When you buy a Tesla, the first thing you’ll do is hire and electrician, and for $1000 to $5000, pay them to install this high-end circuit in your garage or car port. Since you garage is usually where the circuit breaker is located anyway, it’s usually the low-end of this range. You have to choose either the NEMA 14-50 plug, which can be used to power any electric car, or the Tesla HPWC (“High Power Wall Charger”) that just bundles the cord and everything together, making it easier to charge. Just back into your garage, get out of the car, pull off the cord, and plug it in. Real simple.

Standard NEMA 14-50 plug.
Different layout so you don't accidentally plug the wrong thing into it and blow a circuit.
Tesla proprietary wall charger.
Now for our trip to Dallas, though, we have a problem. While we can get the right charging circuit at home, we might not be able to find one on the road. How common can they possibly be? They sound like they'll be hard to find.

Well, no. Electric cars have become a thing. People are into them, unnaturally so. Even if you haven't noticed they EV (electric vehicle) plugs around you, they are everywhere. People are way ahead of you one this.

In our story of driving to Dallas, the first thing you'll do is visit, and lookup where to find charging stations. You'll find a ton of them, even in oil-rich cities like Dallas:

On the left coast, like California, it's insane. Chances are if you go to a business meeting, you'll find one in the parking lot, if not that building, then one next door. Drive in, go to the meeting, have some drinks or dinner afterwards, and you'll be able to drive home on a full charge.

Note that these charging stations primarily use the J1772 plug, a standard that all electric cars support. Your car comes with the standard electrical plug, the NEMA 14-050, and a J1772, so you can use any.

Also note that these charging stations are either run for profit, or part of a network. Even if the charging station is free, you still have to become a member. The most popular network nationwide is ChargePoint, but which one is most popular in a city varies. You may have to join a couple networks (I've just joined ChargePoint -- they have a free one down the street in a park, so I drive there and go for a walk and suck free juice).

These are sort of a franchise network. Somebody owns the parking space. They sign up with a network like ChargePoint, buy their unit and pay for installation, then get payments back from ChargePoint when they use the parking space. Since some businesses want to encourage you to visit, they don't charge you.

Ideally, all these charging stations should deliver max power. In practice, they are usually a bit weaker. Luckily, you can read people's reviews online and figure that out before you go.

One thing I want to stress is there is that charging is parking. The cost of electricity is negligible, that's not what they are charging your for. Instead, they charge your for time. Almost always, you'll be charge for how much time you leave your car parked there, not how much power you use.

As a Tesla owner, you can use these plugs, but also special Tesla plugs. Tesla makes up to two HPWC chargers available for free to business owners, especially hotels. They call this "destination charging", because you charge once you reach your destination. This is rather awesome as a Tesla owner, that you get vastly more options to charge than normal electric cars.

Level 1 and Level 2 charging

When you look at a charging map, like this one from ChargePoint, you'll see it mentions different "levels" of charging. What does this mean?

Level 1 means the standard 120volt/15amp, 1.8kw plug that you have for all electrical devices. Business (or even homes) that have an external plug will often put themselves on the map, but you don't care, since the charging at this level is so slow.

Level 2 means anything faster than Level 1, using the J1772 connector usually. There are a wide range of charging speeds. Original Nissan Leaf could only charge at 3.3kw, so a lot are in that range. More cars can deal with 6.6kw, so some are in that range. Only the Tesla and a Mercedes model go to the full 10kw, so many chargers don't support that much juice.

"Level 2 Tesla", in the map above means the HPWC mentioned above. They have appeared in the last 6 months as Tesla has aggressively pushed them to businesses, though it's usually just hotels. These may be 10kw (40amp), but may go up to 20kw (80amp). Note your car can only handle the 10kw/40amp speeds unless you upgrade to 20kw/80amp dual-charger.

"Level 2 NEMA" didn't use to be on the charging maps 6 months ago, but have appeared now. From what I can tell, a big reason is that when businesses put in a Tesla HPWC, they also put in the NEMA plug because it's really cheap, and allows them to attract more cars than just Teslas (though many don't come standard with that plug). Another reason this exists is because camping parks usually have the plug. You drive in with your campter/trailer, then hook up to the local electricity with this plug. You can also drive in with your Tesla. Indeed, the back of your car is big enough to sleep in.

The next options I'm going to describe below.

DC Fast Charging

Electricity from our walls is AC, or alternating current. Batteries, however, deal only with DC, or direct current. When you plug in your car, a device inside called the charger must convert the current. That's why cars have limitations on how much juice they consume, they are limited by the charger's capacity.

An alternative is to place the charger outside the car, where it can be much bigger, and feed DC direct current to the car. There is a lot of complexity here, because the car's computers need to talk to the external charger many times a second in order to adjust the flow of current. It's as much as a computer networking protocol as it is a copper connection for power.

If you look at your car, you'll see that the regenerative braking often charges the battery at 50kw, which is already many times faster than the AC Level 2 chargers mentioned above. We know the battery pack can handle it. Indeed, according to Tesla, SuperChargers can charge at a rate of 120kw, or 170 miles driving range in 30 minutes of charging.

Tesla has a network of SuperChargers placed along freeways in the United States so that you can drive across country. The idea is that every couple hours, you stop for 30 minutes, relax, drink some coffee, take a pee, and charge your car for the next stage.

In our driving scenario above, there's a SuperCharger in Waco, halfway between Austin and Dallas with 8 stalls available:

From Waco, it’s 76 miles to Dallas, meaning if you fully charge there, you can make it to Dallas and back to Waco without recharging – though it’s cutting it a bit close with the 60kwh model. Though, if your destination is the east side of Dallas, then maybe going through Corsicana and using their SuperCharger would be easier.

The SuperCharger is Tesla's solution to the problem but there are two other standards. There is the Asian standard known as CHAdeMO, and the European standard often called the COMBO adapter, as the DC component is combined with the J1772 AC standard. The CHAdeMO was the early standard coming with Japanese cars, but the COMBO adapter appears to be the winning standard in the long run. Tesla doesn't support either of these standards. These other standards are also far behind the Tesla in charging speed, maxing out around 60kw (and usually less), whereas the Tesla does 120kw.

As I mentioned, the direct DC charging dumps power into the battery as fast as the battery can take it -- which means it's the battery that now becomes the bottleneck. As you know from experience charging phones and laptops, charging to 50% full, but it seems to take forever to get from 90% to 100%. The same is true of your Tesla.

So here's a trick. The 60kwh car actually ships with the 75kwh battery. Tesla is allows you to "upgrade" your car later to the higher range for $9k, which will consist of just Tesla turning on a switch enabling the entire battery. But, for SuperCharging, it's still a 75kwh battery. 60 is 80% of 75. That means, you can charge to 100% full in 40minutes rather than 75minutes.

Charging to 100% lowers the lifetime of the battery, to Tesla recommends you only charge to 80% anyway. Again, with the 60kwh battery, charging to 100% means only 80% anyway. Thus, you get most of the benefits of a larger battery without paying for it. If you truly want extra range, you might consider the 90kwh upgrade instead of going from 60kwh to 75kwh.

The new Model 3s that come out in a few years won't have free SuperCharger access. That may also be true of the cheapest Model S (they keep changing this). You should check on this.

Off-peak Charging

Among the many revolutions recently has been smart electrical meters. They can now bill you by time-of-day when you consume power, charging you more during peak hours, and less during off-peak. If you haven't been paying attention to this, you are still probably on the old billing plan. You might want to look into changing.

Where I live, they charge $0.01 (one cent) per kilowatt-hour during off-off-peak, between 11pm and 7am. That's insane, compared to $0.20 elsewhere in the country.

Your Tesla can be configured to when it charges, so you get home, plug it in, and it won't start charging right away, but wait until off-peak or off-off-peak to start charging.


Electric charging math has given me a new appreciation for the power of gasoline. Filling up your gas tank is the equivalent of charging at multiple megawatt speeds – something electric cars will never get close to. Driving at the range limits of the car requires planning – you just can’t do it without doing some math.

Ethereum/TheDAO hack simplified

The news in the Bitcoin world is the Ethereum/DAO hack. I thought I'd write up a simplified explanation.

What is Bitcoin?

I'm sure you know, or have an idea what Bitcoin is, but I'll explain it again in terms that are important here.

Bitcoin is just a public ledger (the "blockchain"), of all transaction there ever was. This ledger is huge (80-gigabytes) and growing, but Moore's Law says computers grow even faster, so that shouldn't be a problem.

Each entry in the ledger says to move the coins received in these previous entries, and give them to this recipient. In other words:
move these coins I received there, to this guy here
In the future when that guy spends the coin in a new transaction, he'll refer back to this transaction here. That's why it's called a "chain" -- every transaction refers to a previous one, back to the original creation of the coins.

Actually, these transactions aren't fixed as simply moving money around. Instead, a script is used, written in  a bitcoin-specific programming language that can do things like add, multiply, and compare numbers. However, it's not a full programming language. It can't call functions or execute loops. It's complex enough to do allow some creativity, such as allowing transactions to be escrowed by a third party, but it can't do much more than that.

What are alt-coins?

After Bitcoin took off, other people started to create their own variations.

Most of these variations have offered no meaningful improvement over Bitcoin. They instead are just popular among speculators who pump and dump, hoping to buy in at a low price and sell to some fool at a higher price.

There are only two alt-coins that have actually made an improvement: ZeroCoin and Ethereum. ZeroCoin works by making transactions anonymous. Ethereum is the alt-coin that is being discussed here.

What is Ethereum?

Ethereum is a variation of Bitcoin that uses a full programming language instead of a simple scripting language. The reason for this is that Ethereum supports smart contracts.

Let's say that we want to organize a small conference. We need 100 people to sign up and pay/deposit money, so we can rent a hotel and such. But if not enough people sign up by a certain date, then the deposits need to be refunded. With Ethereum, we can write in a JavaScript-like language to code up this contract. It'll guarantee that everyone will get a ticket to the conference, or everyone will get their money refunded, depending on how many sign up.

That's a simple example. The possibilities are endless, which has got a lot of people really excited. Which, of course, makes the believers insufferable among non-believers in the system.

What is The DAO?

DAO stands for decentralized autonomous organization. It's notionally like the example above for getting funds for a small conference, except that it includes much more. Members buy shares in the DAO, and according to the number of shares they have, can vote on things. The dreamers have the idea they'll replace Democracy and run entire countries this way.

There are many types of DAOs on the Ethereum blockchain, but one in particular is TheDAO, which is some sort of DAO-based mutual-fund/investment-fund. You buy in, vote on which investments the fund should make, then reap the returns. It looks like a big organization, but it actually just runs as bits of code and data within the Etherium blockchain.

This investment scheme has gobbled up 10% of Ethereum coins, or $100 million worth out of Ethereum's $1 billion ecosystem.

When you want to leave the fund, you split from it. This includes your share of unspent Ethereum coins, but also tokens representing returns on investments you were part of, but which haven't yet paid out. That's why you "split" rather than "cash out", you need your own DAO shard to track those investments until they pay out.

How did The DAO get hacked?

When a member exits the investment scheme, they call the function name splitDAO(). There are two issues.

The first is that the member will supply some of their own code with the transaction. Among the things that code will do is tell the DAO code how to transfer Ethereum coin. It's a necessary feature, part of Bitcoin as well.

The second issue is that Ethereum code is recursive. That means when a function is running, it may call itself a second time.

The bug is that when splitDAO() is called, it will then call the recipients code to transfer Ethereum coin, after which the recipients code will call splitDAO() again before finishing. This causes the process to repeat itself, transferring more Ethereum coin, then calling splitDAO() again, which calls the hacker's code, which calls splitDAO(), which calls the hacker's code, and so on. The process will continue endlessly, until it drains all of TheDAO's coin.

When you split like this, they still force you to wait 27 days before you get your Etherium coin. Thus, the hacker doesn't actually have use of it for another month.

How can they recover the stolen money?

They can't -- at least not without destroying the entire principle of cryptocurrencies. It's like trying to cure cancer with a Howitzer.

One solution is to roll-back the blockchain before the theft. Of course, that means screwing over everybody who made a transaction since then. You'd be screwing people out of $1 million in order to compensate the theft of $100 million. This is, of course, the type of corrupt thinking that gets us into banking failures in the real world, as we screw over everyone else in order to protect those banks who are too big to fail.

Another solution is to update the Ethereum code to blacklist this address, or better yet, insert a magic key that will give control over those funds back to TheDAO.

The problem with changing the code is that it forks the blockchain. Transactions are added to the chain by miners -- a decentralized group of people. It can include you, if you want to run mining software on your computer. Those miners who update their code to the change will be working off a different blockchain than those who don't. The blockchain will repair itself if 51% of the miners update to the latest code (a soft-fork), but it'll be hopeless broken if they don't (a hard-fork of two irreconcilable chains).

What does this all mean?

I'm a crypto-anarchist. The entire point of cryptocurrencies to get around corrupt humans. And that's what trying to repair this problem is -- corruption. It's a violation of TheDAO's own contract, which says the code is the contract, not to be superseded by human re-interpretation. It's tacit acknowledgement that TheDAO deserves special treatment that would not be given to smaller makers of contracts that have similar recursion bugs. That the miners vote on it (by choosing which software to run) is besides the point, nobody else with Ethereum's favoritism would be able to get miner consensus. It's a profound betrayal of trust that those maintaining the code would even consider helping TheDAO.

Obviously, the complexity of Ethereum is a huge security issue that will likely be repeatedly exploited in the future. This "recursion" issue demands that everybody writing contracts needs to write code extraordinarily carefully. In hindsight, it's obvious that recursion shouldn't have been an allowed feature, but it likely can't be fixed now without a hard-fork. Also, in the future, we'll likely see additional problems as dangerous as the recursion issue.

The fascinating thing now is that in the past, people hired lawyers to review complicated contracts. In the future, they'll need to hire hackers. After a contract is signed, I'm now motivated to hire a very good hacker that will keep reading the code until they can find some hack to my advantage.

In any case, the original concept of TheDAO is useless utopian nonsense. The original Bitcoin was created by people who actually understood a lot about currency. TheDAO was created by people who are hopelessly naive about investing, who then put the system in the hands of trained monkeys. This isn't "wisdom of the crowds", as they proposed, but "ignorance of the mob". If the hacker hadn't put a stop to this nonsense, it would have slid into some sort of Ponzi scheme, getting its creators arrested a couple years from now.

Update: The hacker is bribing the miners not to fork [*], by promising to give the miners a third of Etherium. This is possible with the protocol, to create transactions that takes money from the blacklisted address and donate it directly to whoever mines a block, as a sky-high transaction fee. Choosing to fork would mean forgoing these payments. Thus, the hacker has a power to corruptly subvert the system to counterbalance Etherium's power.


Official statement by Ethereum code maintainers declaring their intention to blacklist/soft-fork

More detailed walk through of splitDAO()

Bloomberg "Smart contracts were dumb", a cool look at the issue

Monday, June 13, 2016

Scanning for ClamAV 0day

Last week an 0day was released for ClamAV. Well, not really an 0day so much as somebody noticed idiotic features in ClamAV. So I scanned the Internet for the problem.

The feature is that the daemon listens for commands that tell it to do things like scan files. Normally, it listens only locally for such commands, but can be reconfigured to listen remotely on TCP port 3310. Some packages that include ClamAV sometimes default to this.

It's a simple protocol that consists of sending a command in clear text, like "PING", "VERSION", "SHUTDOWN", or "SCAN
So I ran masscan with the following command:

masscan -p3310 --banners --hello-string[3310] VkVSU0lPTg==
Normally when you scan an address range (/0) and port (3310), you'd just see which ports are open/closed. That's not useful in this case, because it finds 2.7 million machines. Instead, you want to establish a full TCP connection. That's what the --banners option does, giving us only 38 thousand machines that successfully establish a connection. The remaining machines are large ranges on the Internet where firewalls are configured to respond with SYN-ACK, with the express purpose of frustrating port scanners.

But of those 38k machines, most are actually things like web servers running on odd ports. 51 machines running VNC, 641 machines running SSH, and so on.

To find specifically ClamAV, I send a command using the --hello-string feature. I send the text "VERSION", which must be encoded with base64 on the command-line for masscan (in case you need to also send binary).

This finds 5950 machines (i.e. 6k) that respond back with a ClamAV signature. typical examples of this response are:

At first I thought the date was when they last updated the software, maybe as a page. Roughly half had dates of either this morning or the day before. But no, it's actually the dates when they last updated their signatures.

From this we can conclude that roughly half of ClamAV installations are configured to auto-update their signatures.

Roughly 2400 machines (nearly half) had the version 0.97.5. This was released in June 2012 (four years old). I'm thinking some appliance maker like Barracuda bundled the software -- appliances are notorious for not getting updated software. That hints at why this non-default configuration is so common -- it's not users who made this decision, but the software that bundles ClamAV with other things. Scanning other ports gives me no clues -- they appear all over the map, with different versions of SSH, different services running, different SSL versions, and so on. I thought maybe "mail server" (since that'd be a common task for ClamAV), but there were only a few servers, and they ran different mail server software. So it's a mystery why this specific version is so popular.

I manually tested various machines with "SCAN foo". They all replied "file not found", which hints that all the units I found are vulnerable to this 0day.

As for other things, I came across a bunch of systems claiming to be ChinaDDoS systems:


This sort of stuff shouldn't exist. The number of ClamAV systems available on the public Internet should be zero.

Even inside a corporate network, the number should be 0. If that stuff is turned on, then it should be firewalled (such as with iptables) so that only specific machines can access it.

Two important results are that half the systems are really old (EOLed, no longer supported), and only half the systems have the latest updates. There's some overlap -- systems with latest signature but out-of-date software.

Tuesday, June 07, 2016

No, Musky, Feudalism is best for Mars

Recently, the press fawned all over Elon Musk's comments at a conference. Among them was Musk's claim that "direct democracy" would be the best system, where citizen's vote directly for laws, rather than voting for (corrupt) representatives/congressmen. This is nonsense. The best political system would be feudalism.

There is no such thing as "direct democracy". Our representatives in congress are only the first layer on top of a bureaucracy. Most rules that restrict us are not "laws" voted by congress but "regulations" decided by some bureaucrat.

Consider the BP Gulf Oil spill, as an example. It happened because oil companies got cozy with their regulators, the minerals Management Service (MMS), part of the Department of the Interior. The bureaucrats had a dual mandate: to protect the environment, and to promote economic activity. Oil companies lobbied them to risk the environment in favor of profits.

Consider  Obamcare's controversial mandate that health insurers must pay for abortions. This was not part of the law pass by congress, but a decision by the bureaucrats in charge of all the little details in carrying out the law.

Consider the Federal Communication Commission (FCC) regulation of the Internet. It bases its power to regulate the Internet on laws that essentially predate the Internet as we know it.

No matter how ideal this "direct Democracy" of Musk's, you are still going to leave most decision making in the hands of a bureaucracy. This is especially true on space flight to Mars. If something's wrong with the air system, you want a technician making quick decisions to fix it. Otherwise, people would suffocate long before they had a chance to vote on the issue. Technicians must be trusted with important decisions, like jettisoning that one pod killing 10 people in order to save the remaining 100.

No matter the political system, you are going to have the bureaucracy making tactical, day-to-day decisions. You are also going to have an upper tier, making long term strategic decisions. It's how all political systems work, from monarchies to "direct democracy". They largely just change the names of the bureaucrats, rather than being substantively different.

The corruption in Democracies doesn't necessarily come from those in power, but from the voters themselves. Voters are idiots and vote like idiots. That's why you have candidates like those of the U.S's current election season -- populist demagogues preying on people's ignorance proposing solutions that educated people believe to be unworkable. The majority of voters have never taken an economics class, do not understand foreign policy, or have any other qualification to make the decisions they make.

Instead of education, voters overwhelming decide what's best for themselves, not dispassionately what's best for society as a whole. College students vote for free college. Old people vote for social security and health care. Mothers vote for child leave and child care. Racists vote to keep unwanted types out of their community. And so on. That's corruption at it's core.

As de Tocqueville is famous for noting, democracy only lasts up to the point that 51% of the population realizes they can vote to just take everything away from the other 49%. You call it corruption, but our current system allows a member of the 49% to lobby congress so that they don't get screwed by the 51%.  Indeed, that's what most lobbyists do -- they aren't asking for special favors from the government so much as trying to alleviate special punishments. It's a sort of corruption defending themselves from the voter's corruption.

As the famous quote goes, "Democracy is the worst form of government -- except for all the others". It's a horrible system, it's just we haven't found any better.

But in space exploration, the old rules no longer apply. We can imagine better political systems.

Overwhelmingly, the best system is "vote with your feet". In the future, billionaires will creation space stations around Earth and out in the asteroid belt. Yes, they will be absolutely dictators in their own artificial worlds, but they can't be too evil. They'll be competing for people to come work and live on their space stations.

Such a system doesn't work well on Earth because the barrier in changing countries is just too high. Consider the European Union as an example, where citizens of one country can move to any country they wish. They don't, because they are tied to the language and culture of their own community. In space exploration, such barriers to movement don't exist. Space will look more like the United States, where people do move around a lot, and who do move to the state that they like best. Sure, the culture of the South is different from New York, which is different from the Midwest, which is different from Texas, which is different from the West Coast, but these are tiny cultural/language barriers compared to those that have stopped movement in the past.

Thus, people will vote -- vote by deciding which space station is best. Those who want free health care will go to those space stations. Those who want more money in their pockets now will go to those stations without free health care.

The benefit of the "vote with your feet" system is that there's no coercion. Democracies are always backed up with a police state that coerces you, at the point of a gun. to conform to what the majority has decided. On Earth, you have to submit, because in most countries leaving just isn't a viable option.

But imagine the petit dictators like Musky in his space station. If he tries to coerce people, they'll simply leave. Following laws will therefore always be voluntary -- take it or leave it.

And you will be able to leave. You might imagine that Musky might just surprise the inhabitants and seal all the airlocks, enslaving them all the sudden. That might happen, but only once. Then all the other space station dictators will get together, agree on some sort of "big charter" guaranteeing people rights, such as to leave any space station, and agree that if any member violates the charter, the rest of will just break in from the outside, freeing the people.

Another plausible scenario is that billionaires try to trick people into slavery. A good example is Uber, which provides new workers with cheap loans for new cars. The worry among activists is that it then "enslaves" the worker, because they have to keep working for long hours at low pay in order to satisfy the agreement with Uber -- and agreement they signed without realizing the consequences. That can happen in the future where workers can never leave the space station until they've paid off their debts -- which they can never do in a system rigged against them.

This might happen, but as you'll note above, it can also happen under the current system. Word gets around. Uber has to deal more fairly with its workers who hear such stories, and billionaire dictators of space stations will likewise have to deal a bit more fairly.

Such a system won't be just a billionaire (or corporation) as the dictator with everyone serfs below them. Monolithic corporations are a disaster. Instead, space stations will outsource. They'll have a life support company managing life support. They'll have a propulsion company managing the rockets. They'll contract with a food service company. They me dictators of their own little worlds, but they'll still have to deal with banking corporations outside their worlds -- just like how monarchies in midevil times had to borrow money from banks to conduct their little wars.

What I'm getting at here is that the best political system for space exploration looks a lot like feudalism, though one full of yeomen (who were free to move about) rather than serfs (tied to the land). It's the only morally defensible system of government -- nobody is coerced to follow laws they dislike, but is able to vote with their feet, and choose the laws that best suite them. Those with bad rules will suffer, those who make good rules will prosper.