Intel Cloud Day 2016 news and views

 A couple of weeks back I was at Intel Cloud Day 2016 with the rest of the TFD team. We listened to a number of presentations from Intel Management team mostly about how the IT world was changing and how they planned to help lead the transition to the new cloud world.

The view from Intel is that any organization with 1200 to 1500 servers has enough scale to do a private cloud deployment that would be more economical than using public cloud services. Intel’s new goal is to facilitate (private) 10,000 clouds, being deployed across the world.

In order to facilitate the next 10,000, Intel is working hard to introduce a number of new technologies and programs that they feel can make it happen. One that was discussed at the show was the new OpenStack scheduler based on Google’s open sourced, Kubernetes technologies which provides container management for Google’s own infrastructure but now supports the OpenStack framework.

Another way Intel is helping is by building a new 1000 (500 now) server cloud test lab in San Antonio, TX. Of course the servers will be use the latest Xeon chips from Intel (see below for more info on the latest chips). The other enabling technology discussed a lot at the show was software defined infrastructure (SDI) which applies across the data center, networking and storage.

According to Intel, security isn’t the number 1 concern holding back cloud deployments anymore. Nowadays it’s more the lack of skills that’s governing how quickly the enterprise moves to the cloud.

At the event, Intel talked about a couple of verticals that seemed to be ahead of the pack in adopting cloud services, namely, education and healthcare.  They also spent a lot of time talking about the new technologies they were introducing today.
Continue reading “Intel Cloud Day 2016 news and views”

A tale of two AFAs: EMC DSSD D5 & Pure Storage FlashBlade

There’s been an ongoing debate in the analyst community about the advantages of software only innovation vs. hardware-software innovation (see Commodity hardware loses again and Commodity hardware always loses posts). Here is another example where two separate companies have turned to hardware innovation to take storage innovation to the next level.

DSSD D5 and FlashBlade

DSSD-d5Within the last couple of weeks, two radically different AFAs were introduced. One by perennial heavyweight EMC with their new DSSD D5 rack scale flash system and the other by relatively new comer Pure Storage with their new FlashBlade storage system.FB

These two arrays seem to be going after opposite ends of the storage market: the 5U DSSD D5 is going after both structured and unstructured data that needs ultra high speed IO access (<100µsec) times and the 4U FlashBlade going after more general purpose unstructured data. And yet the two have have many similarities at least superficially.
Continue reading “A tale of two AFAs: EMC DSSD D5 & Pure Storage FlashBlade”

(QoM16-002): Will Intel Omni-Path GA in scale out enterprise storage by February 2016 – NO 0.91 probability

opa-cardQuestion of the month (QoM for February is: Will Intel Omni-Path (Architecture, OPA) GA in scale out enterprise storage by February 2016?

In this forecast enterprise storage are the major and startup vendors supplying storage to data center customers.

What is OPA?

OPA is Intel’s replacement for InfiniBand and starts out at 100Gbps. It’s intended more for high performance computing (HPC), to be used as an inter-cluster server interconnect or next generation fabric. Intel says it “will maintain consistency and compatibility with existing Intel True Scale Fabric and InfiniBand APIs by working through the open source OpenFabrics Alliance (OFA) software stack on leading Linux* distribution releases”. Seems like Intel is making it as easy as possible for vendors to adopt the technology.
Continue reading “(QoM16-002): Will Intel Omni-Path GA in scale out enterprise storage by February 2016 – NO 0.91 probability”

QoM 16-001: Will NVMe GA in enterprise storage over the next 12 months? Yes 0.68 probability

NVMeThe latest analyst forecast contest Question of the Month (QoM 16-001) is on whether NVMe PCIe-SSDs will GA in enterprise storage over the next 12 months? For more information on our analyst forecast contest, please check out the post.

There are a couple of considerations that would impact NVMe adoption.

Availability of NVMe SSDs?

Intel, Samsung, Seagate and WD-HGST are currently shipping 2.5″ & HH-HL NVMe PCIe SSDs for servers. Hynix, Toshiba, and others had samples at last year’s Flash Memory Summit and promised production early this year. So yes, they are available, from at least 3 sources now, including enterprise class storage vendors, with more coming online over the year.

Some advantages of NVMe SSDs?

Advantages of NVMe (compiled from NVMe organization and other NVMe sources):

  • Lower SSD write and read IO access latencies
  • Higher mixed IOPS performance
  • Widespread OS support (not necessarily used in storage systems
  • Lower power consumption
  • X4 PCIe support
  • NVMe over FC Fabric (new RDMA) support

Disadvantages of NVMe SSDs?

Disadvantages of NVMe (compiled from NVMe drive reviewers and other sources):

  • Smaller form factors limit (MLC) capacity SSDs
  • New cabling (U.2) for 2.5″ SSDs
  • BIOS changes to support boot from NVMe (not much of a problem in storage systems)

Not many enterprise storage vendors use PCIe Flash

Current storage vendors that use PCIe flash (sourced from web searches on PCIe flash for major storage vendors):

  • Using PCIe SSDs as part or only storage tier
    • Kamanario K2 all flash array
    • NexGen storage hybrid storage
  • NetApp (PCIe) FlashCache
  • Others (?2) with Volatile cache backed by PCIe SSDs
  • Others (?2) using PCIe SSD as Non-volatile cache

Only a few of these will have new storage hardware out over the next 12 months. I estimated (earlier) about 1/3 of current storage vendors will release new hardware over the next 12 months.

The advantages of NVMe don’t matter as much unless you have a lot of PCIe flash in your system, so the 2 vendors above that use PCIe SSDs as storage are probably most likely to move to NVMe, but the limited size of NVMe drives, the meagre performance speed up to storage available from NVMe, may make NVMe adoption less likely.  So maybe there’s a 0.3 probability * 1/3 (of vendors with hardware refresh) * 2 (vendors using PCIe flash as storage) or ~0.2.

For the other 5 candidates listed above, the advantages for NVMe aren’t that significant, so if they are refreshing their hardware, there’s maybe a low chance that they will take on NVMe, mainly because it’s going to become the prominent PCIe flash protocol, So maybe that adds another 0.15 of probability * 1/3 * 5 or ~0.25. (When I originally formulated the NVMe QoM I had not anticipated NVMe SSDs backing up volatile cache but they certainly exist, today.)

Other potential candidate for NVMe are all start ups. EMC DSSD uses PCIe fabric for it’s NAND support, and could already be making use of NVMe. (Although, I would  not classify DSSD as an enterprise storage vendor.)

But there may be other start ups out there using PCIe flash that would consider moving to NVMe. A while back, I estimated there’s ~3 startups likely to emerge over the next year. It’s almost a certainty that they would all have some sort of flash storage., but maybe only one of them would make use of PCIe SSDs. And it’s unclear whether they would use NVMe drives as main storage or for caching. So, splitting the difference in probabilities, we will use 0.23 probability * 1 or ~0.23.

So total up my forecast we forecast for NVMe adoption in GA enterprise storage hardware over the next 12 months to be Yes with 0.68 probability. 

The other likely candidates that will support NVMe are software defined storage or hyper converged storage. I don’t list these as enterprise storage vendors but I could be convinced that this was a mistake. If I add in SW defined storage the probability goes up, to high 0.80s to low 0.90s.

Comments?

 

(Storage QoM 16-001): Will we see NVM Express (NVMe) drives GA’d in enterprise storage over the next year

NVMeFirst, let me state that QoM stands for Question of the Month. Doing these forecast can be a lot of work, and rather than focusing my whole blog on weekly forecast questions and answers, I would like to do something else as well. So, from now on we are doing only one new forecast a month.

So for the first question of 2016, we will forecast whether NVMe SSDs will be GA’d in enterprise storage over the next year.

NVM Express (NVMe) means the new PCIe interface for SSD storage. Wikipedia has a nice description of NVMe. As discussed there, NVMe was designed for higher performance and enhanced parallelism which comes with the PCI Express (PCIe) bus. The current version of the NVMe spec is 1.2a (available here).

GA means generally available for purchase by any customer.

Enterprise storage systems refers to mid-range and enterprise class storage systems from major AND non-major storage vendors, which includes startups.

Over the next year means by 19 January 2017.

Special thanks to Kacey Lai (@mrdedupe), Primary Data for suggesting this months question.

Current and updates to previous forecasts

 

Update on QoW 15-001 (3DX) forecast:

News out today indicates that 3DX (3D XPoint non-volatile memory) samples may be available soon but it could take another 12 to 18 months to get it into production. 3DX manufacturing is more challenging than current planar NAND technology and uses about 100 new materials, many of which are currently single sourced. We already built into our 3DX forecast potential delays in reaching production in 6 months. The news above says this could be worse than  expected. As such, I feel even stronger that there is less of a possibility of 3DX shipping in storage systems by next December. So I would update my forecast for QoW 15-001 to NO with an 0.75 probability at this time.

So current forecasts for QoW 15-001 are:

A) YES with 0.85 probability; and

B) NO with 0.75 probability

Current QoW 15-002 (3D TLC) forecast

We have 3 active participants, current forecasts are:

A) Yes with 0.95 probability;

B) No with 0.53 probability; and

C) Yes with 1.0 probability

Current QoW 15-003 (SMR disk) forecast

We have 1 active participant, current forecast is:

A) Yes with 0.85 probability

 

(Storage QoW 15-003): Will we see SMR disks in GA enterprise storage systems over the next 12 months?

SMR refers to very high density (10TB), shingled magnetic recording (SMR) hard disk devices.

GA means generally available for purchase by any customer.

Enterprise storage systems refers to mid-range and enterprise class storage systems from major AND non-major storage vendors, which includes startups.

Over the next 12 months means by 22 December 2016.

We discussed our Analyst Forecasting and previous Questions of the week (QoW) on 3D XPoint technology ( forecast) and 3D TLC NAND technology (forecast), with present status below.

Present QoW forecasts:

(#Storage-QoW 15-001) – Will 3D XPoint be GA’d in  enterprise storage systems within 12 months? 2 active forecasters, current forecasts are:

A) YES with 0.85 probability; and

B) NO with 0.62 probability.

(Storage-QoW 15-002) 3D TLC NAND GA’d in major vendor storage next year? 3 active participants, current forecasts are:

A) Yes with 0.95 probability;

B) No with 0.53 probability; and

C) Yes with 1.0 probability

 

 

 

(Storage-QoW 15-002) 3D TLC NAND GA’d in major vendor storage next year – NO 0.53

Latest forecast question is: Will 3D TLC NAND be GA’d in major storage products in 12 months?

Splitting up the QoW into more answerable questions:

A) Will any vendor be shipping 3D TLC NAND SSDs/PCIe cards over the next 9 months?

Samsung will is reportedly already shipping 3D TLC NAND SSDs and PCIe cards as of August 13, 2015 and will be producing 48 layer 256Gb 3D TLC NAND memory soon.  Unclear what 3D TLC NAND technology will be shipping in the next generation drives due out soon but they are all spoken of as read-intensive/write-light storage.

One consideration is that major storage vendors typically will not introduce new storage technologies unless it’s available from multiple suppliers. This is not always the case and certainly not for internally developed storage but has been a critical criteria for most major vendors. But in the above reference, it was reported that SK Hynix and Toshiba are gearing up for 2016 shipments of 48 layer 3D TLC NAND as well, how long it takes to get these into SSD/PCIe cards is another question.

A number of startups are rumored to be using 3D TLC and Kamanario has publicly announced that their systems already use 3D TLC.

My probability of a second source for 3D TLC storage coming out within the first 9 months of next year is 0.75 

B) What changes will be required for storage vendors to utilize 3D TLC NAND storage?

The important changes will be SSD endurance and IO performance.

NAND endurance is rated at DWPD (drive writes per day). Current Samsung 3D TLC SSDs are reportedly rated anywhere from 1.3 to 3.5 DWPD for a 5 year warranty period and newer 3D TLC SSDs are rated at 5 DWPD (unknown warranty period). Current enterprise (800GB) MLC drives are reportedly rated at 10-25 DWPD (for 5 years). So if we use 3.5 DWPD for 3D TLC and 17.5 DWPD for MLC, 3D TLC NAND has a ~5X reduction in endurance.

As for performance, if we use the Samsung reported performance of 160K random reads and 18K random writes vs. an HGST 800GB MLC SSD that has 145K random read and 100K random write performance. There is a reduction of ~5.6X in write performance.  Read performance is actually better with 3D TLC NAND.

In order for major vendors to handle, a reduction in 3D TLC endurance, they will need to limit the amount of data written to these devices. Conveniently, in order for major vendors to deal with the reduction in 3D TLC write performance, they will also have to limit the amount of data written to these devices.

Hence, one potential solution is a multi-tiering, all flash array which uses standard MLC SSD/PCIe cards to absorb the heavy write activity and data from this tier, that is relatively unused, could be archived (?) over time to a 2nd tier of storage consisting of 3D TLC SSD/PCIe cards.

This is not that unusual and it’s being done today for hybrid (disk-SSD) systems with automated storage tiering. Only in this case, data is moved to SSD only if it’s accessed frequently. For 3D TLC the tiering policy should be changed from access frequency to time since last access. Doing so in a hybrid array with disk, MLC SSD and TLC SSD, would require the creation of an additional pool of storage and could be accomplished with software changes alone. There are current major vendor storage systems that already support 3 tiers of storage. And some which already support archiving to cloud storage, so these sorts of changes are present in current shipping product.

So yes there’s a reduction in endurance and yes it has worse write performance but it’s still much faster than disk and most major vendors already have software to be able to handle diverse performance storage. So accomodating the new 3D TLC storage shouldn’t be much of a problem.

New storage technology like this usually doesn’t require a hardware change to use. So the only thing that needs to be changed to accomodate the new 3D TLC is software functionality

So if the 3D TLC 2nd source was available there’s a 0.9 probability that some major storage vendor would adopt the technology over the next year.

3) What are the advantages of 3D TLC storage?

Price should be cheaper than MLC storage and the density (GB/volume) should be better. So in this case, it’s a reduction in cost/GB and increase GB/volume. So for these reasons alone it should probably be adopted.

The advantages are good and would certainly give a major vendor an edge in capacity density and in $/GB or at least get them to parity (barring any  functionality differential) with startups adopting the technology.

So given the advantages present in the technology, I would say there should be a 0.7 probability of adoption within the next 12 months.  

Forecast for QoW 15-002 is:

0.75*0.90*0.70 = 0.47 probability of YES adoption or .53 probability of NO adoption of 3D TLC NAND in major storage vendor products over the next 12 months

Update on QoW 15-001 forecast:

I have an update to my post that forecast for QoW 15-001 as a No with 0.62 probability. This question was on the adoption of 3D XPoint (3DX) technology in any enterprise storage vendor product within a year.

It has been brought to my attention that Intel mentioned the cost of producing 3DX was somewhere between 1/2 and 1/4 the cost of DRAM. Also, recent information has come to light that Intel-Micron will price 3DX between 3D NAND and DRAM. So my analysis as to the cost differential for caching technologies is way off (20X). So there would be a significant cost advantage in using the technology for volatile and non-volatile cache. Even if the chips cost nothing, it might be on the order of $3-5K cheaper with 3DX than battery/superCap backed up DRAM and volatile DRAM caching. So it exists but less than a significant cost saver.  So this being the case, I would have to adjust my 0.35 probability of adoption in this use up to 0.65.  I failed to incorporate this parameter in my final forecast, so all that analysis was for nothing. 

Another potential use is as a non-volatile write buffer for SSDs and even more important for 3D TLC NAND (see above). As this is in an SSD, software and hardware integration is commonplace so there’s a higher probability of adoption there as well. And as there are more SSDs than DRAM caching the cost differential could be more significant. Then again, it would depend on two technologies being adopted (TLC and 3DX) so it’s less likely than any one alone.

The other news (to me) was that Intel announced they would incorporate proprietary changes in DIMM bus to support 3DX as one approach. This does not lend credence to widespread adoption.  But probably only applies to server support for the technology, so I would reduce my probability there to 0.55

Updated forecast for QoW 15-001 is now:

  1. chip in production stays at .85, so there’s still 2.6 potential systems that could adopt the technology directly
  2. 0.85 probability that chips in production * 0.55 probability of servers with the technology  * 0.65 probability that a storage vendor would adopt the technology to replace caching, so (=) ~0.30 probability of server adoption in storage, and with 18 potential vendors thats another 5.5 systems potentially adopting the technology.
  3. Add in the two-three startups that likely will emerge, with similar probability of adoption, or 0.30, which is another 0.9 systems

For a total of 2.6+5.5+0.9=9 systems out of ~24 or 0.38 probability of adoption.

So my updated forecast still stands at No with a .62 probability.

3D NAND, how high can it go?

450_x_492_3d_nand_32_layer_stackI was at the Flash Memory Summit a couple of weeks ago and a presenter (from Hynix, I think) got up and talked about how 3D NAND was going to be the way forward for all NAND technology. I always thought we were talking about a handful of layers. But on the slide he had what looked to be a skyscraper block with 20-40 layers of NAND.

Currently shipping 3D NAND

It seems all the major NAND fabs are shipping 30+ layer 3D NAND. Samsung last year said they were shipping 32-layer 3D (V-)NANDToshiba announced earlier this year that they had 48-layer 3D NANDHynix is shipping 36-layer 3D NAND.  Micron-Intel is also shipping 32-layer 3D NAND. Am I missing anyone?

Samsung also said that they will be shipping a 32GB, 48-layer V-NAND chip later this year. Apparently, Samsung is also working on 64-layer V-NAND in their labs and are getting good results.  In an article on Samsung’s website they mentioned the possibility of 100 layers of NAND in a 3D stack.

The other NAND fabs are also probably looking at adding layers to their 3D NAND but aren’t talking as much about it. i5QVjaOmlEZHmjM34GrH3NFORjU9A-xAk_JUvkzS8Os

Earlier this year on a GreyBeards on Storage Podcast we talked with Jim Handy, Director at Objective Analysis on what was going on in NAND fabrication. Talking with Jim was fascinating but one thing he said was that with 3D NAND, building a hole with the right depth, width and straight enough was a key challenge. At the time I was thinking a couple of layers deep. Boy was I wrong.

How high/deep can 3D NAND go?

On the podcast, Jim said he thought that 3D NAND would run out of gas around 2023. Given current press releases, it seems NAND fabs are adding ~16 layers a year to their 3D-NAND.

So if 32 to 48 layers is todays 3D-NAND and we can keep adding 16 layers/year through 2023 that’s 8 years *16 layers or an additional 128 layers  to the 32  to 48 layers currently shipping. With that rate we should get to 160 to 176 layer 3D NAND chips. And if 48 layers is 32GB then we maybe we could see  ~+100GB  3D NAND chips.

This of course means that there is no loss in capacity as we increase layers. Also that the industry can continue to add 16 layers/year to 3D-NAND chips.

I suppose there’s one other proviso, that nothing else comes along that is less expensive to fabricate while still providing ever increasing capacity of lightening fast, non-volatile storage (see a recent post on 3D XPoint NVM technology).

Photo Credit(s):

  1. Micron’s press release on 3D NAND, (c) 2015 Micron
  2. Toshiba’s press release as reported by AnandTech, (c) 2015 Toshiba