sense in the world in which it is finished, not the world in which it is started."
A lot can happen in ten years. Poke around in the archives to see what was top-of-mind a decade ago, and you will quickly be reminded how fast things change. Just recently, I ran across a link to a seminal paper published by Silicon Systems on managing Flash SSD reliability. As it happens, it was published almost exactly ten years ago, a nagging hint that flash has been on the market in various forms for quite some time. This paper predated the founding of Fusion IO, Violin Memory, and most of the other flash storage systems players that are recognizable today. In those days, flash was regarded as a game changer for data storage. While the topic of the paper is rather unremarkable today, it is instructive to consider everything that has occurred since then.
To get a sense of where things ended up, ask anyone who owned Fusion IO stock last June. Almost exactly four years ago, that PCIe-attached flash vendor filed to go public. By November of that year, the company had a market capitalization of around $4 billion. In June of 2014, amongst howls of protest from analysts and securities lawyers everywhere, they were bought for a surprisingly low $1.1 billion. Sadly, they weren’t alone in underperforming overly enthusiastic expectations. All-flash Violin Memory went public in late 2013 and has been in a world of hurt, most recently being valued at a paltry $350 million. One can argue that both of these companies had their own self-inflicted issues, but to do so overlooks two very important facts.
First, flash-based storage is no longer a unique value proposition. The commoditization of this technology has occurred rather quickly. One does not have to look hard to find that every single system vendor now boasts some form of flash-based storage solution. It should be abundantly clear that the widely anticipated disruption of the storage ecosystem never really happened. The same players that existed five years ago are still alive, and stumbling along in pretty much in the same rank order as before. What has happened instead, is that the component suppliers – notably the disk manufacturers – have had to rationalize their higher end product lines. The result we can plainly see is that flash is now just a component in the same ecosystem that has always existed.
Second, and more important, the value proposition of high performance storage is overshadowed by the value of all the storage management technology that already exists in the market -- i.e., the software that runs storage arrays everywhere. While going faster is great, enterprise IT gets hammered if they lose data. Hence, they spend disproportionately on technologies that deliver the greatest degree of reliability, durability, resiliency, security, and disaster tolerance. For this community, flash-level speed does not obviate the need for any of the enterprise accoutrements.
Moreover, the speed comes at a substantial cost. Not only will flash continue to be substantially more expensive than spinning disks for the foreseeable future, but any associated system and application software will only dilute the overall performance. It’s hard to overstate this last point: de-duplication, encryption, replication, snapshotting, monitoring, and all the things we love about enterprise storage arrays are actually computationally intense. To do them even with the most powerful processors takes time. When you are measuring your storage response time in microseconds, the time budget starts to add up quickly. In the end, your performance boost is smaller than expected but still very expensive.
What we’ve been seeing all along over the last few years is that the big, hairy problems in storage are, at best, only marginally related to performance. The real issues that enterprises are concerned about are problems of scale: How does one effectively scale three or four different types of storage without wasting too much money? How does one figure out where to place each workload in that continuum? How on earth can one upgrade the firmware on a wall of storage arrays without any downtime? These and a myriad of other intractable problems are going to be the ones that the next truly disruptive technology will address.