When we'll get a 128-bit CPU

On Hacker News, this article claiming "You won't live to see a 128-bit CPU" is trending". Sadly, it was non-technical, so didn't really contain anything useful. I thought I'd write up some technical notes.

The issue isn't the CPU, but memory. It's not about the size of computations, but when CPUs will need more than 64-bits to address all the memory future computers will have. It's a simple question of math and Moore's Law.

Today, Intel's server CPUs support 48-bit addresses, which is enough to address 256-terabytes of memory -- in theory. In practice, Amazon's AWS cloud servers are offered up to 24-terabytes, or 45-bit addresses, in the year 2020.

Doing the math, it means we have 19-bits or 38-years left before we exceed the 64-bit registers in modern processors. This means that by the year 2058, we'll exceed the current address size and need to move 128-bits. Most people reading this blogpost will be alive to see that, though probably retired.

There are lots of reasons to suspect that this event will come both sooner and later.

It could come sooner if storage merges with memory. We are moving away from rotating platters of rust toward solid-state storage like flash. There are post-flash technologies like Intel's Optane that promise storage that can be accessed at speeds close to that of memory. We already have machines needing petabytes (at least 50-bits worth) of storage.

Addresses often contain more just the memory address, but also some sort of description about the memory. For many applications, 56-bits is the maximum, as they use the remaining 8-bits for tags.

Combining those two points, we may be only 12 years away from people starting to argue for 128-bit registers in the CPU.

Or, it could come later because few applications need more than 64-bits, other than databases and file-systems.

Previous transitions were delayed for this reason, as the x86 history shows. The first Intel CPUs were 16-bits addressing 20-bits of memory, and the Pentium Pro was 32-bits addressing 36-bits worth of memory.

The few applications that needed the extra memory could deal with the pain of needing to use multiple numbers for addressing. Databases used Intel's address extensions, almost nobody else did. It took 20 years, from the initial release of MIPS R4000 in 1990 to Intel's average desktop processor shipped in 2010 for mainstream apps needing larger addresses.

For the transition beyond 64-bits, it'll likely take even longer, and might never happen. Working with large datasets needing more than 64-bit addresses will be such a specialized discipline that it'll happen behind libraries or operating-systems anyway.

So let's look at the internal cost of larger registers, if we expand registers to hold larger addresses.

We already have 512-bit CPUs -- with registers that large. My laptop uses one. It supports AVX-512, a form of "SIMD" that packs multiple small numbers in one big register, so that he can perform identical computations on many numbers at once, in parallel, rather than sequentially. Indeed, even very low-end processors have been 128-bit for a long time -- for "SIMD".

In other words, we can have a large register file with wide registers, and handle the bandwidth of shipping those registers around the CPU performing computations on them. Today's processors already handle this for certain types of computations.

But just because we can do many 64-bit computations at once ("SIMD") still doesn't mean we can do a 128-bit computation ("scalar"). Simple problems like "carry" get difficult as numbers get larger. Just because SIMD can do multiple small computations doesn't tell us what one large computation will cost. This was why it took an extra decade for Intel to make the transition -- they added 64-bit MMX registers for SIMD a decade before they added 64-bit for normal computations.

The above discussion is about speed, but it's also a concern for power consumption. Mobile devices were a decade later (than desktops) adopting 64-bits, exceeding the 32-bit barrier just now. It's likely they be decades late getting to 128-bits. Even if you live to see supercomputers transition to 128-bits, you probably won't live to see your mobile device transition.

Now let's look at the market. What the last 40 years has taught us is that old technology doesn't really day, it's that it stops growing -- with all the growth happening in some new direction. 40 years ago, IBM dominated computing with their mainframes. Their mainframe business is as large as ever, it's just that all the growth in the industry has been in other directions than the mainframe. The same thing happened to Microsoft's business, Windows still dominates the desktop, but all the growth in the last 15 years has bypassed the desktop, moving to mobile devices and the cloud.

40 years from now, it won't be an issue of mainstream processors jumping from 64-bits to 128-bits, like the previous transitions. I'm pretty sure we'll have ossified into some 64-bit standard like ARM. Instead, I think 128-bit systems will come with a bunch of other radical changes. It'll happen on the side of computers, much like how GPUs evolved separately from mainstream CPUs can became increasingly integrated into them.