Category: QoM 2016

QoM1610: Will NVMe over Fabric GA in enterprise AFA by Oct’2017

Posted on by Ray in Block Storage, CIFS/SMB, Ethernet, FC, FCoE, File Storage, Forecasting, Infiniband, iSCSI, Networking, NFS, NVMe, NVMe storage, Omni-Path, QoM 2016, RDMA, RoCE, SSD storage, Storage performance, Strategic Inflection Points

NVMeNVMe over fabric (NVMeoF) was a hot topic at Flash Memory Summit last August. Facebook and others were showing off their JBOF (see my Facebook moving to JBOF post) but there were plenty of other NVMeoF offerings at the show.

NVMeoF hardware availability

When Brocade announced their Gen6 Switches they made a point of saying that both their Gen5 and Gen6 switches currently support NVMeoF protocols. In addition to Brocade’s support, in Dec 2015 Qlogic announced support for NVMeoF for select HBAs. Also, as of  July 2016, Emulex announced support for NVMeoF in their HBAs.

From an Ethernet perspective, Qlogic has a NVMe Direct NIC which supports NVMe protocol offload for iSCSI. But even without NVMe Direct, Ethernet 40GbE & 100GbE with  iWARP, RoCEv1-v2, iSCSI SER, or iSCSI RDMA all could readily support NVMeoF on Ethernet. The nice thing about NVMeoF for Ethernet is not only do you get support for iSCSI & FCoE, but CIFS/SMB and NFS as well.

InfiniBand and Omni-Path Architecture already support native RDMA, so they should already support NVMeoF.

So hardware/firmware is already available for any enterprise AFA customer to want NVMeoF for their data center storage.

NVMeoF Software

Intel claims that ~90% of the software driver functionality of NVMe is the same for NVMeoF. The primary differences between the two seem to be the NVMeoY discovery and queueing mechanisms.

There are two fabric methods that can be used to implement NVMeoF data and command transfers: capsule mode where NVMe commands and data are encapsulated in normal fabric packets or fabric dependent mode where drivers make use of native fabric memory transfer mechanisms (RDMA, …) to transfer commands and data.

12679485_245179519150700_14553389_nA (Linux) host driver for NVMeoF is currently available from Seagate. And as a result, support for NVMeoF for Linux is currently under development, and  not far from release in the next Kernel (I think). (Mellanox has a tutorial on how to compile a Linux kernel with NVMeoF driver support).

With Linux coming out, Microsoft Windows and VMware can’t be far behind. However, I could find nothing online, aside from base NVMe support, for either platform.

NVMeoF target support is another matter but with NICs/HBAs & switch hardware/firmware and drivers presently available, proprietary storage system target drivers are just a matter of time.

Boot support is a major concern. I could find no information on BIOS support for booting off of a NVMeoF AFA. Arguably, one may not need boot support for NVMeoF AFAs as they are probably not a viable target for storing App code or OS software.

From what I could tell, normal fabric multi-pathing support should work fine with NVMeoF. This should allow for HA NVMeoF storage, a critical requirement for enterprise AFA storage systems these days.

NVMeoF advantages/disadvantages

Chelsio and others have shown that NVMeoF adds ~8μsec of additional overhead beyond native NVMe SSDs, which if true would warrant implementation on all NVMe AFAs. This may or may not impact max IOPS depending on scale-ability of NVMeoF.

For instance, servers (PCIe bus hardware) typically limit the number of private NVMe SSDs to 255 or less. With an NVMeoF, one could potentially have 1000s of shared NVMe SSDs accessible to a single server. With this scale, one could have a single server attached to a scale-out NVMeoF AFA (cluster) that could supply ~4X the IOPS that a single server could perform using private NVMe storage.

Base level NVMe SSD support and protocol stacks are starting to be available for most flash vendors and operating systems such as, Linux, FreeBSD, VMware, Windows, and Solaris. If Intel’s claim of 90% common software between NVMe and NVMeoF drivers is true, then it should be a relatively easy development project to provide host NVMeoF drivers.

The need for special Ethernet hardware that supports RDMA may delay some storage vendors from implementing NVMeoF AFAs quickly. The lack of BIOS boot support may be a minor irritant in comparison.

NVMeoF forecast

AFA storage systems, as far as I can tell, are all about selling high IOPS and very-low latency IOs. It would seem that NVMeoF would offer early adopter AFA storage vendors a significant performance advantage over slower paced competition.

In previous QoM/QoW posts we have established that there are about 13 new enterprise storage systems that come out each year. Probably 80% of these will be AFA, given the current market environment.

Of the 10.4 AFA systems coming out over the next year, ~20% of these systems pride themselves on being the lowest latency solutions in the market, and thus command high margins. One would think these systems would be the first to adopt NVMeoF. But, most of these systems have their own, proprietary flash modules and do not use standard (NVMe) SSDs and can use their own proprietary interface to their proprietary flash storage. This will delay any implementation for them until they can convert their flash storage to NVMe which may take some time.

On the other hand, most (70%) of the other AFA systems, that currently use SAS/SATA SSDs, could boost their IOP counts and drastically reduce their IO  response times, by implementing NVMe SSDs and NVMeoF. But converting SAS/SATA backends to NVMe will take time and effort.

But, there are a select few (~10%) of AFA systems, that already use NVMe SSDs in their AFAs, and for these few, they would seem to have a fast track towards implementing NVMeoF. The fact that NVMeoF is supported over all fabrics and all storage interface protocols make it even easier.

Moreover, NVMeoF has been under discussion since the summer of 2015, which tells me that astute AFA vendors have already had 18+ months to develop it. With NVMeoF host drivers & hardware available since Dec. 2015, means hardware and software exist to test and validate against.

I believe that NVMeoF will be GA’d within the next 12 months by at least one enterprise AFA system. So my QoM1610 forecast for NVMeoF is YES, with a 0.83 probability.

Comments?

 

 

 

QoM1608: The coming IOT tsunami or not

Posted on by Ray in Forecasting, QoM 2016

Techpinions ChartSaw an interesting chart the other day in a post in TechPinions (Searching for What’s Next) showing the sales in millions over time of PCs, Tablets and Smart Phones.  From the chart, PC sales peaked 2010-2012 and that Tablet sales have at flat lined (2016). Not sure what’s projections vs. actuals but the story on SmartPhones have yet to run out and they had rapid sales growth between 2008 and 2014.

The other thing to take from this chart is that device adoption is speeding up. It took 20 years to reach peak PC sales but it only took ~10 years to reach peak Smartphones sales.
Continue reading “QoM1608: The coming IOT tsunami or not” →

(QoM16-002): Will Intel Omni-Path GA in scale out enterprise storage by February 2016 – NO 0.91 probability

Posted on by Ray in Clustered storage, Ethernet, Forecasting, Infiniband, Omni-Path, QoM 2016, QoW 2015, SSD storage

opa-cardQuestion of the month (QoM for February is: Will Intel Omni-Path (Architecture, OPA) GA in scale out enterprise storage by February 2016?

In this forecast enterprise storage are the major and startup vendors supplying storage to data center customers.

What is OPA?

OPA is Intel’s replacement for InfiniBand and starts out at 100Gbps. It’s intended more for high performance computing (HPC), to be used as an inter-cluster server interconnect or next generation fabric. Intel says it “will maintain consistency and compatibility with existing Intel True Scale Fabric and InfiniBand APIs by working through the open source OpenFabrics Alliance (OFA) software stack on leading Linux* distribution releases”. Seems like Intel is making it as easy as possible for vendors to adopt the technology.
Continue reading “(QoM16-002): Will Intel Omni-Path GA in scale out enterprise storage by February 2016 – NO 0.91 probability” →

QoM 16-001: Will NVMe GA in enterprise storage over the next 12 months? Yes 0.68 probability

Posted on by Ray in Forecasting, PCIe, QoM 2016, SSD storage, Storage performance

NVMeThe latest analyst forecast contest Question of the Month (QoM 16-001) is on whether NVMe PCIe-SSDs will GA in enterprise storage over the next 12 months? For more information on our analyst forecast contest, please check out the post.

There are a couple of considerations that would impact NVMe adoption.

Availability of NVMe SSDs?

Intel, Samsung, Seagate and WD-HGST are currently shipping 2.5″ & HH-HL NVMe PCIe SSDs for servers. Hynix, Toshiba, and others had samples at last year’s Flash Memory Summit and promised production early this year. So yes, they are available, from at least 3 sources now, including enterprise class storage vendors, with more coming online over the year.

Some advantages of NVMe SSDs?

Advantages of NVMe (compiled from NVMe organization and other NVMe sources):

  • Lower SSD write and read IO access latencies
  • Higher mixed IOPS performance
  • Widespread OS support (not necessarily used in storage systems
  • Lower power consumption
  • X4 PCIe support
  • NVMe over FC Fabric (new RDMA) support

Disadvantages of NVMe SSDs?

Disadvantages of NVMe (compiled from NVMe drive reviewers and other sources):

  • Smaller form factors limit (MLC) capacity SSDs
  • New cabling (U.2) for 2.5″ SSDs
  • BIOS changes to support boot from NVMe (not much of a problem in storage systems)

Not many enterprise storage vendors use PCIe Flash

Current storage vendors that use PCIe flash (sourced from web searches on PCIe flash for major storage vendors):

  • Using PCIe SSDs as part or only storage tier
    • Kamanario K2 all flash array
    • NexGen storage hybrid storage
  • NetApp (PCIe) FlashCache
  • Others (?2) with Volatile cache backed by PCIe SSDs
  • Others (?2) using PCIe SSD as Non-volatile cache

Only a few of these will have new storage hardware out over the next 12 months. I estimated (earlier) about 1/3 of current storage vendors will release new hardware over the next 12 months.

The advantages of NVMe don’t matter as much unless you have a lot of PCIe flash in your system, so the 2 vendors above that use PCIe SSDs as storage are probably most likely to move to NVMe, but the limited size of NVMe drives, the meagre performance speed up to storage available from NVMe, may make NVMe adoption less likely.  So maybe there’s a 0.3 probability * 1/3 (of vendors with hardware refresh) * 2 (vendors using PCIe flash as storage) or ~0.2.

For the other 5 candidates listed above, the advantages for NVMe aren’t that significant, so if they are refreshing their hardware, there’s maybe a low chance that they will take on NVMe, mainly because it’s going to become the prominent PCIe flash protocol, So maybe that adds another 0.15 of probability * 1/3 * 5 or ~0.25. (When I originally formulated the NVMe QoM I had not anticipated NVMe SSDs backing up volatile cache but they certainly exist, today.)

Other potential candidate for NVMe are all start ups. EMC DSSD uses PCIe fabric for it’s NAND support, and could already be making use of NVMe. (Although, I would  not classify DSSD as an enterprise storage vendor.)

But there may be other start ups out there using PCIe flash that would consider moving to NVMe. A while back, I estimated there’s ~3 startups likely to emerge over the next year. It’s almost a certainty that they would all have some sort of flash storage., but maybe only one of them would make use of PCIe SSDs. And it’s unclear whether they would use NVMe drives as main storage or for caching. So, splitting the difference in probabilities, we will use 0.23 probability * 1 or ~0.23.

So total up my forecast we forecast for NVMe adoption in GA enterprise storage hardware over the next 12 months to be Yes with 0.68 probability. 

The other likely candidates that will support NVMe are software defined storage or hyper converged storage. I don’t list these as enterprise storage vendors but I could be convinced that this was a mistake. If I add in SW defined storage the probability goes up, to high 0.80s to low 0.90s.

Comments?

 

(Storage QoM 16-001): Will we see NVM Express (NVMe) drives GA’d in enterprise storage over the next year

Posted on by Ray in data access, Forecasting, NVMe storage, PCIe, QoM 2016, QoW 2015, SSD storage, Storage performance

NVMeFirst, let me state that QoM stands for Question of the Month. Doing these forecast can be a lot of work, and rather than focusing my whole blog on weekly forecast questions and answers, I would like to do something else as well. So, from now on we are doing only one new forecast a month.

So for the first question of 2016, we will forecast whether NVMe SSDs will be GA’d in enterprise storage over the next year.

NVM Express (NVMe) means the new PCIe interface for SSD storage. Wikipedia has a nice description of NVMe. As discussed there, NVMe was designed for higher performance and enhanced parallelism which comes with the PCI Express (PCIe) bus. The current version of the NVMe spec is 1.2a (available here).

GA means generally available for purchase by any customer.

Enterprise storage systems refers to mid-range and enterprise class storage systems from major AND non-major storage vendors, which includes startups.

Over the next year means by 19 January 2017.

Special thanks to Kacey Lai (@mrdedupe), Primary Data for suggesting this months question.

Current and updates to previous forecasts

 

Update on QoW 15-001 (3DX) forecast:

News out today indicates that 3DX (3D XPoint non-volatile memory) samples may be available soon but it could take another 12 to 18 months to get it into production. 3DX manufacturing is more challenging than current planar NAND technology and uses about 100 new materials, many of which are currently single sourced. We already built into our 3DX forecast potential delays in reaching production in 6 months. The news above says this could be worse than  expected. As such, I feel even stronger that there is less of a possibility of 3DX shipping in storage systems by next December. So I would update my forecast for QoW 15-001 to NO with an 0.75 probability at this time.

So current forecasts for QoW 15-001 are:

A) YES with 0.85 probability; and

B) NO with 0.75 probability

Current QoW 15-002 (3D TLC) forecast

We have 3 active participants, current forecasts are:

A) Yes with 0.95 probability;

B) No with 0.53 probability; and

C) Yes with 1.0 probability

Current QoW 15-003 (SMR disk) forecast

We have 1 active participant, current forecast is:

A) Yes with 0.85 probability