096: GreyBeards YE2019 IT Industry Trends podcast

In this, our yearend industry wrap up episode, the GreyBeards discuss trends and technologdies impacting the IT industry in 2019 and what’s ahead for 2020. This year we have Matt and Keith on the podcast along with Ray. Just like last year, we start off with NVMeoF.

NVMeoF unleashed

This year just about every major storage vendor announced new systems that either have support for NVMeoF or currently offer NVMeoF on their storage systems. Most offer FC based NVMeoF but a few offer NVMeoF/Ethernet, fewer still offer both.

All of the NVMeoF/Ethernet seem to be using RoCE or iWARP. Unclear if one is more often used that the other, so for now both continue to be used in the market. Some storage vendors are offering NVMeoF as an internal fabric to access storage while still using iSCSI or FC/SCSI to access the data. This works better than SAS but won’t provide all the performance you can get from end-to-end NVMeoF.

NVMeoF is all about increasing IOPS and reducing response times. That and getting ready for SCM SSDs. In the mean time the SSD industry has introduced some very attractive NVMe (NAND) SSDs that in NVMeoF storage system can increase IOPS and reduce latencies.

We talked last year about NVMeoF standards finally stabilizing and this year the rollout across enterprise storage systems is testament to that.

SCM hits the enterprise

Most of us attended an Intel Data Center Event earlier this past yea,r where Optane DC PM was introduced. Optane DC PM is the memory version of Optane SCM (3DX Crosspoint) technology. Intel offers two distinct modes of accessing Optane DC PM as memory: 1) App Direct mode, where data in Optane DC PM persists across power cycles but requires one to use a special AP; and 2) Memory mode where Optane DC PM is cleared during a power cycle, (see our RayOnStorage post Need memory, Intel’s Optane DC PM…).

Vendors seem to be using Optane both memory and SCM technology differently. Pure is using Optane SSDs plugged into their FlashArray as sort of a read cache for customer IO. They suggest for well behaved applications this can reduce IO response times considerably.

Dell EMC introduced SCM as a storage tier and are using their automated storage tiering to move the hottest data to SCM. Oracle’s latest Exadata appliance uses Optane DC PM as both a read and write caching layer.

It won’t be long before every enterprise vendor offers SCM drives in their storage systems with a few offering Optane DC PM as in memory caching technology.

Of course, the big news for Optane DC PM is its use in memory databases, specifically SAP HANA. HANA can take advantage of the (6) TB of memory to to handle larger databases. Keith mentioned that even Microsoft SQL server can take advantage of the additional memory to provide faster responses to queries.

Keith also mentioned that there are some systems out there that can be configured to share Optane memory (or storage). When SAP or other databases use this solution they are able to amortize the cost of the technology over more use cases.

Of course, Optane DC PM are only available on the lastest generation Intel processors. None of us have heard anything from AMD (or Micron) on providing a second source for support of Optane DC PM (or the memory technology itself). Presumably most customers would want a second source for Optane DC PM processor support (as well as the technology)

Cloud enterprise storage hits mainstream

The other thing we saw more of this year is enterprise vendors offering versions of storage in public cloud environments. NetApp was an early proponent of doing this.

We saw at Pure that they have a new Cloud Block Store witch is a re-architected version of FlashArray//X storage using AWS hardware and networking services. We were very impressed with what they have accomplished and it was the subject of more than one late night discussion. Listen to the Keith & Ray show at Pure//Accelerate2019 podcast to learn more.

Matt mentioned Nimble’s cloud volume storage which is cloud adjacent. Most enterprise vendors offer something similar today. They differentiate on how easy it is to configure, use and where (which regions) it’s available in.

NetApp has arguably been at this the longest and has the deepest offerings available from cloud adjacent file and block storage, to offering native enterprise file services for all public cloud environments, to supplying a suite of dedicated data services to surround all of their storage technology operating in public clouds and on premises.

While Dell EMC may have missed the turn to the cloud, they are quickly trying to catch up. Keith mentioned Faction, a Dell partner that offers cloud storage services using VMware with VMC. With Faction and vSAN customers have access to software defined storage that uses cloud hardware to support data services.

What’s driving data growth

There seems to be no end for the need for storage to store data. The GreyBeards point to three trends driving data growth today.

  1. IoT seems to have no bounds. A recent RayOnStorage post Internet of Tires discussed how tire companies were tying their tires to the internet. And that’s just the start, pretty soon every artifact, every device, every manufactured item will have a number of sensors attached all of which will be creating massive amounts of data.
  2. AI ML DL has an insatiable appetite for data. IoT is being used largely to optimize products and services. But it’s DL, with a large dollop of data, that is behind much of that optmization.
  3. SaaS applications is a relatively new application approach that’s being rolled out to more arenas and as it’s online and user oriented, seems to generate lots of data.

Containers storage debate

We closed the podcast with a heavy debate on whether container applications have need for storage. Keith was adamant that containers by their very nature are stateless and that Kubernetes ability to stop and start container applications at will almost requires stateless operations.

Ray was a bit more theoretical on the topic and believed that most container applications today take advantage of some sort of database or other services to store state and that state is just another word for storage.

Keith mentioned encoding as a typical container app. Encoding containers can be fired up and taken down at will without hurting anything but throughput. Yes, but those encoder container apps must access some database or other state information to find out what work is left to do and as they complete their work they update this data as well as store their newly encoded segments. This all involves the use of state information.

In the end, I think we were talking about the same thing but using different terminology. Keith believes that persistent state information is needed and Ray says that this is just another word for (containers) storage. Matt said we probably need Nigel (@NigelPoulton) on the podcast to straighten us both out.

The podcast ran a bit long and could have run longer. Keith and Matt bring systems level perspective to what’s happening in the storage market. But they come at it from different sides. Ray seems to frame everything from a storage perspective. Diverse perspectives lead to a more fuller and interesting discussion. Listen to the podcast to learn more.


This image has an empty alt attribute; its file name is Spotify_Logo_CMYK_Black-1024x307.png
This image has an empty alt attribute; its file name is Subscribe_on_iTunes_Badge_US-UK_110x40_0824.png
This image has an empty alt attribute; its file name is play_prism_hlock_2x-300x64.png

Ray Lucchesi ( @RayLucchesi) is the host of GreyBeardsOnStorage and is President/Founder of Silverton Consulting, and a prominent blogger at RayOnStorage.com.

Keith Townsend (@CTOAdvisor) is a IT thought leader who has written articles for many industry publications, interviewed many industry heavyweights, worked with Silicon Valley startups, and engineered cloud infrastructure for large government organizations. Keith is the co-founder of The CTO Advisor, blogs at Virtualized Geek

Matt Leib (@MBLeib), one of our co-hosts, has been blogging in the storage space for over 10 years, with work experience both on the engineering and presales/product marketing. His blog is at Virtually Tied to My Desktop.


89: Keith & Ray show at Pure//Accelerate 2019

There were plenty of announcements at Pure//Accelerate in Austin this past week and we were given a preview of them at a StorageFieldDay Exclusive (SFDx), the day before the announcement.

First up is Pure’s DirectMemory. They have added Optane SSDs to FlashArray//X to be used as a read cache for customer data. As you may know, Pure already has an NVRAM write cache. With DirectMemory, customers can have 3TB or 6TB of Optane storage in a FlashArray//X70 or //X90 storage. It almost looks plug and play, you take out one or two flash modules and plug in Optane SSD(s) and off it goes. DirectMemory went GA at the show.

Pure also announced FlashArray//C at Accelerate. This is a new capacity optimized storage solution. They have re-designed their flash module to support higher capacity flash, and supply higher capacity storage (targeted for QLC flash but will originally ship with TLC). FlashArray//C supplies ~5PB of effective (~1.4PB raw) capacity in 9U. Although, FlashArray//C offers cheaper storage on $/GB basis it is also much slower (RT latency on order of 2-4msec) than FlashArray//X storage.. Pure like other vendors we have talked with are trying to drive disk technology out of the enterprise. We had some interesting discussions with Pure (and others) on this topic at the reception. Just remember, tape is still alive and well in the enterprise AND cloud, 52 years after being pronounced dead.

Pure had announced CloudBlockStore (CBS) previously but it is now GA through partners or on AWS marketplace. Give them kudos for their approach as they have taken a different approach to Pure storage in the cloud. With CBS, they have effectively re-archetected and re-implemented Pure FlashArray using AWS EC2, IO1, EBS and S3 storage and ended up with a highly available (iSCSI) block software defined storage. It will be interesting to see how well it’s adopted. Picture is from me explaining CBS architecture to @DVellante.

For Pure’s FlashBlade storage, they have doubled the number of blades in a cluster (or name space), from 75 to 150 FlashBlades. Each FlashBlade contains storage and compute (almost computational storage), so one should see an increase in bandwidth with the added blades. None at Pure would go on record with specific numbers on any performance improvement because it’s still undergoing testing.

Finally, FlashArray//X will offer full NFS and SMB file support. This is coming from a recent acquisition (Compuverde). They plan to differentiate between file on FlashArraiy//X file storage and FlashBlade by saying that FlashArray//X file is for those customers with mostly block storage requirements but also need small amount of file storage and FlashBlade for everyone else that needs file.

The podcast is ~23 minutes. Keith is a long time friend and co-host of our GreyBeards On Storage podcast. He’s always got an interesting perspective on how new technology can benefit the data center today. Listen to the podcast to learn more.

This image has an empty alt attribute; its file name is Subscribe_on_iTunes_Badge_US-UK_110x40_0824.png
This image has an empty alt attribute; its file name is play_prism_hlock_2x-300x64.png

Keith Townsend, The CTO Advisor

Keith Townsend (@CTOAdvisor) is a IT thought leader who has written articles for many industry publications, interviewed many industry heavyweights, worked with Silicon Valley startups, and engineered cloud infrastructure for large government organizations. Keith is the co-founder of The CTO Advisor, blogs at Virtualized Geek, and can be found on LinkedIN.

86: Greybeards talk FMS19 wrap up and flash trends with Jim Handy, General Director, Objective Analysis

This is our annual Flash Memory Summit podcast with Jim Handy, General Director, Objective Analysis. It’s the 5th time we have had Jim on our show. Jim is also an avid blogger writing about memory and SSD at TheMemoryGuy and TheSSDGuy, respectively.

NAND market trends

Jim started off our discussion on the significant price drop in the NAND market over the last two years. He said that prices ($/GB) have dropped 60% last year and are projected to drop about 30% this year.

The problem is over production and as vendors are prohibited from dropping prices below cost, they tend to flatten out at production cost. NAND pricing will remain there until supplies start tightening again. Jim doesn’t see that happening until 2021.

He says although this NAND price drops don’t end up reducing SSD prices, it does allow us to buy more SSD storage for the same price. So maybe back earlier this century NAND cost $10K/GB, now it’s around $0.05/GB.

Jim also mentioned that Chinese NAND fabs should start coming online in 2021 too. They have been spending lots of money trying to get their own NAND manufacturing running. Jim said the reason they want to do this is because the Chinese are spending more $s on chips , than they do for oil.

Computational storage, a bright spot

At the show, computational storage (for more hear our GBoS podcast with Scott Shadley, NGD Systems) was hot again this year. Jim took a shot at defining computational storage and talked about the proliferation of ARM cores in SSDs. Keith mentioned that Moore’s law is making the incremental cost of adding more cores close to zero.

Jim said SAMSUNG already have 6 ARM cores in their SSDs, but most other vendors use 3 cores. I met with NetInt at the show who are focused on computational storage for video transcoding. Keith doesn’t think this would be a good fit, because it takes a lot of computation. But maybe as it’s easily distributable (out to a gaggle of SSDs) and it’s data intensive it might work ok. Jim also mentioned while adding cores may be cheap, increasing memory (DRAM) is not.

According to Jim, hyper-scalars are starting to buy computational storage technology. He’s not sure if they are just trying it out or have some real work running on the technology.

SCM news

We talked about Toshiba’s new XC flash and SSDs. Jim said this is just SLC NAND (expensive $/GB and high endurance) with increased parallelism and reduced latency data paths. Samsung’s Z-NAND is similar. Toshiba claims XL Flash SSDs are another storage class memory (SCM, see our 3DX blog post). Toshiba are pricing XL Flash SSDs at about 10X the $/GB price of 3D TLC NAND, or roughly the same as Optane SSDs.

We next turned to Optane DC PM, which Intel is selling at a loss but as it works only with Cascade Lake CPUs, can help increase CPU adoption. So Intel can absorb Optane DC PM losses by selling more (highly profitable) Cascade Lake systems.

Keith mentioned that SAP HANA now works with Cascade Lake-Optane DC PM. This is driving up demand for the new DC PM and new CPUs. Keith said with the new larger size in memory databases from DC PM, HANA able to do more work, increasing Cascade Lake-Optane DC PM-SAP HANA adoption.

Micron also manufacturers 3DX. Jim said they are in an enviable position as they can . supply the chips (at costs) to Intel, so they know chip volumes and can see what Intel is charging for the technology. So, if at some point, it has runway to become profitable, they can easily enter as a sole secondary source for the technology.

Other NAND news

How high can 3D TLC NAND go? Jim said most 3D NAND sold on the market is 64 layers high but suppliers are already shipping more layers than that. All NAND suppliers, bar one, have said their next generation 3D TLC NAND will be over 100 layers. Some years back one vendor said the technology could go up to 500 layers. This year Samsung, said they see the technology going to 800 layers.

We’ve heard of SLC, MLC, TLC and QLC but at the show there was talk of PLC or five level cell NAND technology. If they can make the technology successful, PLC should reduce manufacturing costs, another 10% ($/GB).

We discussed a lot more that was highlighted at the show, including PCIe fabric/composable infrastructure, zoned (NVMe) name spaces (redux SMR disks) and the ongoing success of the show. We had a brief discussion on when if ever NAND costs will be less than disk ($/GB).

The podcast is a little under ~40 minutes. Jim is an old friend, who is extremely knowledgeable about NAND & DRAM technology as well as semiconductor markets in general. Jim’s always been a kick to talk with. Listen to the podcast to learn more.

Jim Handy, General Director, Objective Analysis

Jim Handy of Objective Analysis has over 35 years in the electronics industry including 20 years as a leading semiconductor and SSD industry analyst. Early in his career he held marketing and design positions at leading semiconductor suppliers including Intel, National Semiconductor, and Infineon.

A frequent presenter at trade shows, Mr. Handy is known for his technical depth, accurate forecasts, widespread industry presence and volume of publication.

He has written hundreds of market reports, articles for trade journals, and white papers, and is frequently interviewed and quoted in the electronics trade press and other media. 

He posts blogs at www.TheMemoryGuy.com, and www.TheSSDguy.com

77: GreyBeards talk high performance databases with Brian Bulkowski, Founder & CTO, Aerospike

In this episode we discuss high performance databases and the storage needed to get there, with Brian Bulkowski, Founder and CTO of Aerospike. Howard met Brian at an Intel Optane event last summer and thought he’d be a good person to talk with. I couldn’t agree more.

Howard and I both thought Aerospike was an in memory database but we were wrong. Aerospike supports in memory, DAS resident and SAN resident distributed databases.

Database performance is all about the storage (or memory)

When Brian first started Aerospike, they discovered that other enterprise database vendors were using fast path SAS SSDs for backend storage and so that’s where Aerospike started with on storage.

As NVMe SSDs came out, Brian expected higher performance but wasn’t too impressed with what he found out with NVMe SSD’s real performance as compared to SAS SSDs. However lately, the SSD industry has bifurcated into fast, low-capacity (NVMe) SSDs and slow, large capacity (SAS) SSDs. And over time the Linux Kernel (4.4 and above) has sped up NVMe IO stack. So now he has become more of a proponent of NVMe SSDs for high performing database storage.

In addition to SAS and NVMe SSDs, Aerospike supports SAN storage. One recent large customer uses SAN shared storage and loves the performance. Moreover, Aerospike also offers an in memory database option for the ultimate in high performance (low capacity) databases.

Write IO performance

One thing that Aerospike is known for is their high performance under mixed R:W workloads. Brian says just about any database can perform well with an 80:20 R:W IO mix, but at 50:50 R:W, most databases fall over.

Aerospike did detailed studies of SSD performance with high write IO and used SSD native APIs to understand what exactly was going on with SAS SSDs. Today, they understand when SSDs go into garbage collection and and can quiesce IO activity to them during these slowdowns. Similar APIs are available for NVMe SSDs.

Optane memory

The talk eventually turned to Optane DIMMs (3D Crosspoint Memory). With Optane DIMMs, server memory address space will increase from 1TB to 6TB. From Brian’s perspective this is still not enough to host a copy of a typical database but it would suffice to hold cache a  database index. Which is exactly how they are going to use Optane DIMMs.

Optane DIMMs are accessed via PMEM (an Intel open source memory access API) and can specify  caching (L1-L2-L3) characteristics, so that the processor(s) data and instruction caching tiers don’t get flooded with database information. Aerospike has done for in-memory databases in the past, it’s just requires a different API.

As a distributed database, they support data protection for DAS and in memory databases through mirroring, dual redundancy.  But Aerospike was developed as a  distributed database, so data can be sharded, across multiple servers to support higher, parallelized performance.

With Optane DIMMs being 1000X faster than NVMe SSD, the performance bottleneck has now moved back to the network. Given the dual redundancy data protection scheme, any data written on one server would need to be also written (across the network) to another server.

Data consistency in databases

This brought us around to the subject of database consistency.  Brian said Aerospike database consistency for reads was completely parameterized, e.g. one can specify linear (database wide) consistency to session level consistency, with some steps in between. Aerospike is always 100% write consistent but read consistency can be relaxed for better performance.

Howard and I took a deep breath and said data has to be a 100% consistent. Brian disagreed, and in fact, historically relational databases were not fully read consistent. Somehow this felt like a religious discussion and in the end, we determined that database consistency is just another knob to turn if you want high performance.

Brian mentioned that  Aerospike is available in an open source edition which anyone can access and download. He suggested we tell our DBA friends about it, maybe, if we have any…

The podcast runs ~44 minutes. Brian’s been around databases for a long time and seemingly, most of that time has been figuring out the best ways to use storage to gain better performance. He has a great perspective on  NVMe vs. SAS SSD performance as well as (real) memory vs SCM performance, which we all need to understand better as SCM rolls out. Possibly, barring the consistency discussion, Brian was also easy to talk with.  Listen to our podcast to learn more.

Brian Bulkowski, Founder and CTO, Aerospike

Brian is a Founder and the CTO of Aerospike. With almost 30 years in Silicon Valley, his motivation for starting Aerospike was the confluence of what he saw as the rapidly advancing flash storage technology with lower costs that weren’t being fully leveraged by database systems as well as the scaling limitations of sharded MySQL systems and the need for a new distributed database.

He was able to see these needs as both a Lead Engineer at Novell and Chief Architect at Cable Solutions at Liberate – where he built a high-performance, embedded networking stack and high scale broadcast server infrastructure.