Blog Viewer

Why you need to care about NVMe over Fabrics – now

By phansen posted 11-01-2016 05:28 AM


Part 1 of a 2-part Series - You can find the second part of the series, here.


Anyone paying even the slightest bit of attention at the recent Flash Memory Summit would come away with three compelling observations regarding the future of storage.  Importantly, these observations have a significant and profound impact on critical infrastructure decisions that IT leaders are making today.  And if made wrong, the implications could be devastating for the organization.


#1 - Flash is the Technology of Choice for Storage

Flash is very fast already, and getting much faster with the release of 3D NAND flash technology.  The growth of AFA (all flash arrays) in to the data center is still embryonic, yet the industry’s expectation is in the long-term flash will ultimately eliminate spinning hard disk drives for most applications except for warm data repositories (near line storage).  This means only 7200 RPM HDDs survive, whereas the performance-oriented market for 10K and 15K RPM HDD products will succumb to flash storage. This essentially removes HDDs from the performance tier in the storage device hierarchy.


Along with well-publicized performance and power savings, Flash vendors are also increasing density very rapidly.  Toshiba bc_hand.jpgannounced 3D NAND technologies that in 2017 will allow a single 1TB chip the size of a US penny (see image), while Seagate released a 60TB SSD in a 3.5-inch form factor and boosts of 1PB capacity arrays using only 17 SSDs. 


Flash is more than raw performance; this technology is being deployed in high capacity storage arrays supporting low-latency applications in a shared data environment.  As these devices become more mainstream, they will become an integral component in traditional storage area networks.  


#2 – NVMe is the Protocol for Flash

One of the industry luminaries at the Flash Memory Summit made an interesting observation.  He commented that the “SSD guys got it right from the beginning”.  When SSD vendors first released flash-based products, they cleverly supported the existing HDD environment.  This meant a 2.5-inch form factor SSD with 100% plug-compatible SAS or SATA interfaces.  No changes required, a perfect replacement for 2.5-inch HDDs where essentially the only “user change” was faster access to your data and improved application response times.  Recognizing that most data centers are one of the most risk-adverse places in the planet, this evolutionary approach was the ideal way to launch SSD products.


But today, the fears and risks of emerging flash technologies have all but disappeared and SSDs have established a solid market foundation on their own.  Unfortunately, the legacy SAS and SATA interfaces that relied on the aging SCSI protocol stack … good for hard disk drives with many heads and spinning platters … was orders of magnitude too slow for today’s ultra-fast solid-state memories (see graph). A new protocol - optimized for flash - would bring forward the full capabilities of this technology.  This protocol is called NVMe (Non-Volatile Memory express). 




NVMe replaces the SCSI protocol and can reduce latency by a few orders of magnitude.  Today, NVMe is fully adopted by the flash industry and will definitely dominate flash over the coming years.  As SSDs overtake the storage market, so will NVMe as the protocol of choice for flash.  This progression may take years, but will happen.  The benefits of NVMe are simply too compelling.







#3 – The Storage Network is the new Bottleneck

At this year’s Summit, almost every presentation that amplified the latest performance benefits, also sent a message of “insufficient storage network bandwidth” to keep up with these developments.  With wide scale technology advancements in server-side compute and solid-state storage, the infrastructure bottleneck has now shifted to legacy networks.  No longer can slower Fibre Channel or Ethernet transports keep up with the new performance within the data center.  Presentations at the Summit explained how a seemingly small number of NVMe-based SSDs can saturate many legacy networks.  This graphic shows how just four NVMe drives saturate a 100GbE link.



To further this point, in a recent Gartner report on The Future of Storage Protocols (G00307902), the research firm said that flash_memory_summit.jpg“Storage performance bottlenecks are moving out of arrays and into the storage networks” and that “Future protocols (such as 40GbE used for iSCSI), file-based protocols (such as NFS and SMB) and current block protocols (such as 16Gbs Fibre Channel) will be too slow for the next generation of solid-state arrays and hybrid arrays.”  Their summary recommendation is that “storage networking investments are becoming a critical top priority” and that “IT leaders, therefore, must revisit and review their budget plans for storage networking infrastructure and ensure their ability to meet increasingly performance-sensitive service levels.”   Read the full report .


Help is on the Way

Luckily, there is relief to this problem and better yet - it is just around the corner.   It’s called NVMe over Fabrics, or NVMe-oF.  Now the benefits of the ultra-low latency NVMe protocol can be applied to shared storage environments, leveraging the proven benefits of fabric network technologies like FC or converged Ethernet.  But there are choices here as well … NVMe over Fibre Channel, RoCE (RDMA over Converged Ethernet), IB (InfiniBand) and others.  Each with specific benefits and limitations. 


Next Steps

IT professionals must become familiar with these new technologies, especially if they are planning an infrastructure upgrade with flash storage anytime soon.  Failure to anticipate the future of storage and storage networks could be disastrous.  Find out more at, or read the next blog in this series.    



Read part two of this series, here.