Quantumrun

IMAGE CREDIT: Quantumrun

The digital storage revolution: Future of Computers P3

David Tal, Publisher, Futurist
@DavidTalWrites
LinkedIn

Tue, 09/15/2020 - 15:43

Most of you reading this probably remember the humble floppy disk and it's solid 1.44 MB of disk space. Some of you were probably jealous of that one friend when he whipped out the first USB thumb drive, with its monstrous 8MB of space, during a school project. Nowadays, the magic is gone, and we've become jaded. One terabyte of memory comes standard in most 2018 desktops—and Kingston even sells one terabyte USB drives now.

Our obsession with storage grows year over year as we consume and create ever more digital content, whether it's a school report, travel photo, your band's mixtape, or a GoPro video of you skiing down Whistler. Other trends like the emerging Internet of Things will only accelerate the mountain of data the world produces, adding further rocket fuel to the demand for digital storage

This is why to discuss data storage properly, we recently decided to edit this chapter by splitting it in two. This half will cover the tech innovations in data storage and its impact on average digital consumers. Meanwhile, the next chapter will cover the coming revolution in the cloud.

Data storage innovations in the pipeline

(TL;DR - The following section outlines the new tech that will enable ever larger quantities of data to be stored onto ever smaller and more efficient storage drives. If you don't care about the tech, but instead want to read about the wider trends and impacts around data storage, then we recommend skipping to the next subheading.)

Many of you have already heard of Moore’s Law (the observation that the number of transistors in a dense integrated circuit doubles roughly every two years), but on the storage side of the computer business, we have Kryder’s Law—basically, our ability to squeeze ever more bits into shrinking hard drives is also doubling roughly every 18 months. That means the person who spent $1,500 for 5MB 35 years ago can now spend $600 for a 6TB drive.

This is jaw-dropping progress, and it's not stopping anytime soon.

The following list is a brief glimpse into the near- and long-term innovations digital storage manufacturers will use to satisfy our storage-hungry society.

Better hard disk drives. Until the early 2020s, manufacturers will continue building traditional hard disk drives (HDD), packing in more memory capacity until we can no longer build hard disks any denser. The techniques invented to lead this final decade of HDD tech include Shingled Magnetic Recording (SMR), followed by Two-Dimensional Magnetic Recording (TDMR), and potentially Heat-Assisted Magnetic Recording (HAMR).

Solid state hard drives. Replacing the traditional hard disk drive noted above is the solid state hard drive (SATA SSD). Unlike HDDs, SSDs don’t have any spinning disks—in fact, they don’t have any moving parts at all. This allows SSDs to operate far faster, at smaller sizes, and with more durability than their predecessor. SSDs are already a standard on today’s laptops and are gradually becoming standard hardware on most new desktop models. And while originally far more expensive than HDDs, their price is falling faster than HDDs, meaning their sales could overtake HDDs outright by the mid-2020s.

Next generation SSDs are gradually being introduced as well, with manufacturers transitioning from SATA SSDs to PCIe SSDs that have at least six times the bandwidth of SATA drives and growing.

Flash memory goes 3D. But if speed is the goal, nothing beats storing everything in memory.

HDDs and SSDs can be compared to your long-term memory, whereas flash is more akin to your short-term memory. And just like your brain, a computer traditionally needs both types of storage to function. Commonly referred to as random access memory (RAM), traditional personal computers tend to come with two sticks of RAM at 4 to 8GB each. Meanwhile, the heaviest hitters like Samsung are now selling 2.5D memory cards that hold 128GB each—amazing for hardcore gamers, but more practical for next-generation supercomputers.

The challenge with these memory cards is that they’re running into the same physical constraints hard disks are facing. Worse, the tinier transistors become inside RAM, the worse they perform over time—the transistors get harder to erase and write accurately, eventually hitting a performance wall that forces their replacement with fresh RAM sticks. In light of this, companies are beginning to build the next generation of memory cards:

3D NAND. Companies like Intel, Samsung, Micron, Hynix, and Taiwan Semiconductor are pushing for the wide-scale adoption of 3D NAND, which stacks transistors into three dimensions inside a chip.
Resistive Random Access Memory (RRAM). This tech uses resistance instead of an electric charge to store bits (0s and 1s) of memory.
3D chips. This will be discussed in more detail in the next series chapter, but in brief, 3D chips aim to combine computing and data storage in vertically stacked layers, thereby improving processing speeds and reducing energy consumption.
Phase Change Memory (PCM). The tech behind PCMs basically heats and cools chalcogenide glass, shifting it between crystallized to non-crystallized states, each with their unique electrical resistances representing the binary 0 and 1. Once perfected, this tech will last far longer than current RAM variants and is non-volatile, meaning it can hold data even when the power’s off (unlike traditional RAM).
Spin-Transfer Torque Random-Access Memory (STT-RAM). A powerful Frankenstein that combines the capacity of DRAM with the speed of SRAM, along with improved non-volatility and near unlimited endurance.
3D XPoint. With this tech, instead of relying on transistors to store information, 3D Xpoint uses a microscopic mesh of wires, coordinated by a "selector" that are stacked on top of one another. Once perfected, this could revolutionize the industry since 3D Xpoint is non-volatile, will operate thousands of times faster than NAND flash, and 10 times denser than DRAM.

In other words, remember when we said “HDDs and SSDs can be compared to your long-term memory, whereas flash is more akin to your short-term memory”? Well, 3D Xpoint will handle both and do so better than either than either separately.

Regardless of which option wins out, all of these new forms of flash memory will offer more memory capacity, speed, endurance and power efficiency.

Long-term storage innovations. Meanwhile, for those use cases where speed matters less than the preservation of large amounts of data, new and theoretical technologies are currently in the works:

Tape drives. Invented over 60 years ago, we originally used tape drives to archive tax and healthcare documents. Today, this tech is being perfected near its theoretical peak with IBM setting a record by archiving 330 terabytes of uncompressed data (~330 million books) into a tape cartridge around the size of your hand.
DNA storage. Researchers from the University of Washington and Microsoft Research developed a system to encode, store and retrieve digital data using DNA molecules. Once perfected, this system may one day archive information millions of times more compactly than current data storage technologies.
Kilobyte rewritable atomic memory. By manipulating individual chlorine atoms on a flat sheet of copper, scientists wrote a 1-kilobyte message at 500 terabits per square inch—roughly 100 times more info per square inch than the most efficient hard drive on the market.
5D data storage. This specialty storage system, spearheaded by the University of Southampton, features 360 TB/disc data capacity, thermal stability up to 1,000°C and a near unlimited lifetime at room temperature (13.8 billion years at 190°C ). In other words, 5D data storage would be ideal for archival uses at museums and libraries.

Software-Defined Storage Infrastructure (SDS). It’s not just storage hardware that’s seeing innovation, but the software that runs it is also undergoing exciting development. SDS is used mostly in large company computer networks or cloud storage services where data is stored centrally and accessed through individual, connected devices. It basically takes the total amount of data storage capacity in a network and separates it among the various services and devices that run on the network. Better SDS systems are being coded all the time to more efficiently use existing (instead of new) storage hardware.

Will we even need storage in the future?

Okay, so storage tech is going to improve a whole lot over the next few decades. But the thing we have to consider is, what difference does that make anyway?

The average person will never use up the terabyte of storage space now available in the latest desktop computer models. And in another two to four years, your next smartphone will have enough storage space to horde a year's worth of pictures and videos without having to spring clean your device. Sure, there's a minority of people out there who like to horde massive amounts of data on their computers, but for the rest of us, there are a number of trends reducing our need for excessive, privately-owned disk storage space.

Streaming services. Once upon a time, our music collections involved collecting records, then cassettes, then CDs. In the 90s, songs became digitized into MP3s to be hoarded by the thousands (first through torrents, then more and more through digital stores like iTunes). Now, instead of having to store and organize a music collection on your home computer or phone, we can stream an infinite number of songs and listen to them anywhere through services like Spotify and Apple Music.

This progression first reduced the physical space music took up at home, then the digital space on your computer. Now it can all be replaced by an external service that provides you with cheap and convenient, anywhere/anytime access to all the music you could want. Of course, most of you reading this probably still have a few CDs lying around, most will still have a solid collection of MP3s on their computer, but the next generation of computer users won't waste their time filling their computers with music they can access freely online.

Obviously, copy everything I just said about music and apply it to film and television (hello, Netflix!) and the personal storage savings keep growing.

Social media. With music, film, and TV shows clogging up less and less of our personal computers, the next largest form of digital content is personal pictures and videos. Again, we used to produce pictures and videos physically, ultimately to collect dust in our attics. Then our pictures and videos went digital, only to again collect dust in the nether reaches of our computers. And that’s the issue: We rarely look at most of the pictures and videos we take.

But after social media happened, sites like Flickr and Facebook gave us the ability to share an infinite number of pictures with a network of people we care about, while also storing those pictures (for free) in a self-organizing folder system or timeline. While this social element, coupled with miniature, high-end phone cameras, greatly increased the number of pictures and video produced by the average person, it also reduced our habit of storing photos on our private computers, encouraging us to store them online, privately or publicly.

Cloud and collaboration services. Given the last two points, only the humble text document (and a few other niche data types) remains. These docs, compared to the multimedia we just discussed, are usually so small that storing them on your computer will never be a problem.

However, in our increasingly mobile world, there’s a growing demand to access docs on the go. And here again, the same progression we discussed with music is happening here—where first we transported docs using floppy disks, CDs, and USBs, now we use more convenient and consumer-oriented cloud storage services, like Google Drive and Dropbox, which store our docs at an external data center for us to access securely online. Services like these allow us to access and share our docs anywhere, anytime, on any device or operating system.

To be fair, using streaming services, social media, and cloud services doesn’t necessarily mean we will move everything to the cloud—some things we prefer to keep overly private and secure—but these services have cut, and will continue to cut, the total amount of physical data storage space we need to own year over year.

Why exponentially more storage matters

While the average individual may see less need for more digital storage, there are big forces at play that are driving Kryder’s Law forward.

First off, due to the near-annual list of security breaches across a range of tech and financial services companies—each endangering the digital information of millions of individuals—concerns over data privacy are rightfully growing among the public. Depending on individual needs, this may drive public demand for larger and cheaper data storage options for personal use to avoid depending on the cloud. Future individuals may even set up private data storage servers inside their homes to connect to externally instead of depending on servers owned by the large tech companies.

Another consideration is that data storage limitations are currently blocking progress in a number of sectors from biotech to artificial intelligence. Sectors that depend on the accumulation and processing of big data need to store ever larger amounts of data to innovate new products and services.

Next, by the late 2020s, the Internet of Things (IoT), autonomous vehicles, robots, augmented reality, and other such next-gen ‘edge technologies' will spur investment into storage tech. This is because for these technologies to work, they will need to have the computing power and storage capacity to understand their surroundings and react in real time without a constant dependence on the cloud. We explore this concept further in chapter five of this series.

Finally, the Internet of Things (explained fully in our Future of the Internet series) will result in billions-to-trillions of sensors tracking the movement or status of billions-to-trillions of things. The immense amounts of data these countless sensors will produce will demand effective storage capacity before it can be effectively processed by the supercomputers we'll cover near the end of this series.

All-in-all, while the average person will increasingly reduce their need for personally owned, digital storage hardware, everyone on the planet will still benefit indirectly from the infinite storage capacity future digital storage technologies will offer. Of course, as hinted at earlier, the future of storage lies in the cloud, but before we can dive nose deep into that topic, we first need to understand the complimentary revolutions happening on the processing (microchip) side of the computer business—the topic of the next chapter.