SCI’s latest SPEC sfs2014 performance report as of Dec.’19

In Build MinRT-ORT, Builds, DB workload, ORT, SPECsfs, SPECsfs2014, Streams, SW Build workload, VDA workloadby AdministratorLeave a Comment

This Storage Intelligence (StorInt™) dispatch covers SPEC sfs2014 benchmark[1]. These were many new submissions this past quarter, DDN SFA14KXE with EXAScaler (vda) and SFA15KX with GridScaler (swbuild, vdi, eda, & database), Oracle ZFS ZS7-2 (vda, swbuild, database & eda), and NetApp AFF A800 8- and 4-node (swbuild). As we have now reached at least 10 submissions for vda (10) and swbuild (16), we will show only top 10 rankings for those workloads. 

SPEC SFS2014_vda (video data acquisition) results

SPECsfs VDA workload simulates video data streams. One stream equates to a single camera being recorded. Figure 1 shows the VDA concurrent stream counts. Recall that VDA represents video camera data acquisition. The first metric of interest is number of concurrent streams supported.

Figure 1 SPEC SFS2014_vda top 10 number of concurrent streams supported

In Figure 1, the new DDN SFA14KXE with EXAScaler file system was able to support over 3400 concurrent streams and came in as the new #1. Oracle’s ZFS ZS7-2 also showed well supporting 2520 concurrent streams of video data and is our new #2. 

The DDN SFA14KXE with EXAScaler is a LUSTRE parallel file system using disk only storage with 420 10TB 7200 RPM disks plus another 4 300GB 15KRPM drives. In total the DDN EXAScaler had over 2.8PB (RAID6? DCR protected) storage in its file system.  At its max stream MB/sec, this disk only submission was pushing over 15.7GB/sec of video streams. Unclear how the parallel LUSTRE file system might have helped. Both the SFA14KXE and EXAScaler appliance were connected over EDR (100Gb) InfiniBAND. The backend storage was FC connected to the SFA14KXE and EXAScaler solutions.

The Oracle ZFS ZS7-2 used SAS connected SSD only and had 264 3TB SSDs and 24 200GB SSDs for its backend storage across 2 ZS7-2 controllers attached via 10GbE to client systems. The Oracle ZFS ZS7-2 had ~830TB of SSD stable storage but only supported 366TB of file system (RAID-10) storage. 

Next, we show the Minimum (Min RT) and Overall Response Time (ORT) for VDA submissions in Figure 2. We sort this ranking based on ORT, least to largest.

Figure 2 SPEC sfs2014_swbuild top 10 minimum (Min RT) and overall response times (ORT)

In Figure 2, one can see the downside of using all disk (and possible a parallel file system) as the DDN SFA14KXE EXAScaler system came in last with an over 50 msec ORT. Surprisingly the Oracle ZS7-2 submission came in sixth with a 30 msec ORT. It’s hard to see here but the SSD SPECsfs reference submission had an ORT of 2.9 msec and a Min RT of 2.4 msec and the two IBM Spectrum Scale submissions (#2-3) had an ORT of 2.9 and 4.1, and a Min RT of 0.8 and 1.6 msec, respectively. 

As ORT is an average of response times across a submissions workloads, it depends a lot on how hard the storage system is being driven as well as storage media used. Disk only systems can’t match the RT of SSD media and systems that are being driven hard tend to have worse ORTs.

SPEC SFS2014_swbuild (software build) results

In Figure 3 we show the top 10 SWBUILD concurrent builds metric

Figure 3 SPEC sfs2014_swbuild top 10 concurrent builds

In Figure 3, the two new NetApp AFF A800 submissions (8- and 4-node) come in as the new #1 & 2, supporting 4200 and 2200 concurrent software builds, respectively.  Coming in 3rdplace is the new DDN SFA14KX with GridScaler file systems, supporting 1500 concurrent software build streams. The new Oracle ZFS ZS7-2 came in at #6 supporting 1120 concurrent software builds. 

The two NetApp solutions show close to linear performance scaling on a per node basis with ~525-550 concurrent software builds per node. The 8- and 4-node AFF A800 used either 192 and 96 3.8TB NVMe SSDs in a RAID-DP configuration for file system storage and included 8.7 and 4.4 TB of DRAM, respectively. NetApp’s FlexGroup is a scale out, distributed performance volume that holds file systems spanning a multi-node cluster. The NetApp AFF A800s were connected to clients via 100GbE and 25GbE links with 100GbE intercluster links.

The new DDN SFA14KX with GridScalers submission had 6 GridScaler NSDs connected to a SFA14KX storage appliance holding 73 400GB SSDs (RAID6? DCR protected) storage media. The GridScaler NSDs were connected to the client systems using 100Gb EDR InfiniBAND links. 

Next, in Figure 4 we show the Top 10 ORT (ranked and sorted) and Min (RT) for the SWBUILD workload.

Figure 4 Top 10 SPEC sfs2014_swbuild ORT

In Figure 4, the new DDN SFA14KX with GridScaler came in as the new #1 with an ORT of 0.19 msec and Min RT of 0.1 msec, very impressive for SAS SSDS. The NetApp AFF A800 4- and 8- node clusters came in at #5 & #6 with an ORT of 0.73 and 0.78 msec respectively. The new Oracle submission did not show up in our top 10 ORT, as it only had an ORT of 1.07 msec. 

SPEC SFS2014_database results

In Figure 5, we present our first chart for the SPEC sfs2014_database workload.

Figure 5 SPEC sfs2014_database concurrent databases supported

 In Figure 5, the two new submissions, Oracle ZFS ZS7-2 and DDN SFA14KX with GridScaler supported 2240 and 750 concurrent databases respectively. Oracle did well on the database workload. 

The DATABASE workload represents a typical transactional workload for a SQL database. The workload consists of two types of IO: Table and LOG IO. For Table IO, it generates 79% random reads and 20% random writes and for Log IO it generates 80% (sequential) writes and 20% random writes. Our best guess is that every random write to the Table, generates a sequential writes to the Log, which means that ~83% is Table IO and ~17% is Log IO, this doesn’t account for the random writes to the Log but it’s close enough

Significance

We are always pleased to see new SPEC sfs2014 submissions, especially when they are in workloads where we have never plotted before. Over the next few quarters, we will be reviewing the other (EDA and VDI) workloads. At the moment EDA has 2 vendor submissions and the VDI workload only has one. 

It’s nice to see NetApp AFF A800 clusters taking the software build performance up to a whole new level. If anything, NetApp cut its teeth on software development teams and one can see here why. The same applies to Oracle and the database workload. 

Still trying to determine the best way to report SPEC sfs2014 results. We may experiment with a few variants of the above charts. Any ideas on other metrics of interest to report, please do let us know

Furthermore, suggestions on how to improve any of our performance analyses are always welcomed.

[Also we offer more file storage performance information plus our NFS and CIFS/SMB  ChampionsCharts™ charts in our recently updated (April 2019) NAS Storage Buying Guide available for purchase on our website.]

[This performance dispatch was originally sent out to our newsletter subscribers in December of 2018.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Dispatches are posted to our website at least a quarter or more after they are sent to our subscribers, so if you are interested in current results please consider signing up for our newsletter.

Silverton Consulting, Inc., is a U.S.-based Storage, Strategy & Systems consulting firm offering products and services to the data storage community


[1]All SPEC SFS2014 information is available athttps://www.spec.org/sfs2014/ as of 26Dec2018

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.