SPEC SFS®2014 Performance Report as of March 2020

In DB workload, EDA workload, ORT, SPECsfs, SPECsfs2014, VDI workload by AdministratorLeave a Comment

This Storage Intelligence (StorInt™) dispatch covers SPEC sfs2014 benchmarks . There was one new vendor submission for each of the 5 workloads (VDA, SWBUILD, VDI, DATABASE & EDA) which was an Oracle ZFS ZS7-2 Midrange Hybrid (disk-SSD) 4 tray storage solution. If you have been following our SPEC sfs2014 reports you may recall that some workloads only have a few vendor submissions, namely VDI, EDA & DATABASE. So, Oracle’s submissions across all workloads are good news.

SPEC sfs2014_vdi (virtual desktop infrastructure) results

The SPEC sfs2014_vdi workload simulates a high intensity knowledge worker environment similar to IO behavior of ESXi, Hyper-V, KVM and Xen environments. We have described this workload more fully in prior performance reports. Figure 1 presents the maximum desktop results for VDI submissions.

Figure 1 SPECsfs2014_vdi top number of desktops supported

In Figure 1, the new Oracle ZFS ZS7-2 midrange hybrid storage came in at #3 with a maximum desktop support of 320. The other non-SPEC submissions here were AFA.

The WekaIO Matrix 3.1.9 had over 138 NVMe SSDs across over 46 servers, while the Oracle midrange hybrid had only 8 (4-200GB SAS & 4 7.7TB SAS) SSDs and 88 14TB (nearline) disk drives, over 2 controllers. As such, on a per controller basis, the Oracle hybrid did ~160 desktops/controller while the #1 WekaIO only did about 35 desktops/server. Similarly, the Oracle hybrid solution did 40 desktops/SSD while the #1 WekaIO did about 12 desktops/SSD. The DDN used 72 400GB SAS SSDs for about 11 desktops/SSD. Given all the above, the Oracle midrange hybrid storage did very well for its configuration.

Recall that the other two solutions on the above chart (#4 & #5) are SPEC reference storage systems, essentially Linux servers, running storage software with DAS, `all SSD or all disk, respectively.

Next, we show the Overall Response Time (ORT) for VDI submissions in Figure 2.

Figure 2 SPEC sfs2014_vdi top overall response times (ORT)

In Figure 2, the Oracle midrange hybrid storage came in at #4 (behind all the other AFA systems). Recall that ORT is the average response time across all workload levels (from 10% all the way up to max).

We have stopped showing Min- and Max-RT for VDI. However, if we did, the Oracle midrange hybrid storage system would have the second lowest Min RT of any of the systems, at 0.27msec. The lowest Min RT would have been the DDN system with 0.2msec.

It is very interesting that a hybrid system would outperform (in min RT) some all SSD storage systems. Yes at 10% of achievable workloads (where Min-RT is measured), all IO data is probably coming off of SSD, if not memory. Nonetheless, with hybrid systems, one can often customized the ratio of disk to SSDs to achieve better or worse performance. Unclear whether in reality Oracle’s ZFS ZS2-7 storage performance would benefit from more SSDs as they are used for ZIL (write accelerator) drives and read caches. But my guess is yes.

SPEC sfs2014_eda (electronic design automation) performance results

SPEC sfs2014_eda simulates the type of IO workload that goes into the IC and printed circuit board design, such as placement-routing, design verification and logic simulation tools. In Figure 3, we show the top EDA concurrent jobsets results.

Figure 3 SPEC sfs2014_eda top concurrent jobsets

In Figure 3, the new Oracle midrange hybrid storage came in at #4. Please note the other Oracle ZFS system was an all (288) SSD across 4 controller nodes system configuration. That the all SSD SPEC reference submission came in 4th behind the Oracle hybrid storage, says that Oracle’s ZFS system architecture is well optimized for EDA workloads.

Next, in Figure 4 we show ORT results for the EDA workload.

Figure 4 Top 10 SPEC sfs2014_eda ORT

It seems odd that the #3 Oracle midrange hybrid storage solution would beat the ORT performance of the same Oracle ZFS ZS-7-2 storage solution using all SSDs (#4 above).

Possibly, this could mean that the Oracle midrange submission wasn’t being driven hard enough or at its maximum performance. Also the small ORT difference (0.01msec) for the WekaIO AFA and the Oracle midrange hybrid is yet another indicator of this.

SPEC sfs2014_database performance results

The DATABASE workload represents a typical transactional workload for a SQL database. In Figure 5 below we present the maximum databases supported metric.

Figure 5 SPEC sfs2014_database Max databases supported

In Figure 5, the Oracle midrange hybrid storage solution came in at #4, behind all the other vendor AFA submissions but ahead of the SPEC (all SSD) reference submission. As discussed earlier, the Oracle’s hybrid storage did better in maximum databases supported per SSD, than any of all AFA DATABASE submissions.

In Figure 6 we present Min- and Max-RT for the DATABASE workload.

Figure 6 SPEC sfs2014_database top Min- & Max-RT

In Figure 6, we show both the Min- & Max-RT (minimum and maximum response times) for the DATABASE submissions which is sorted by Min-RT. The #2 Oracle midrange hybrid solution actually supplied a better Min-RT than the Oracle ZFS ZS7-2 all SSD configuration as well as the WekaIO AFA system.

The WekaIO solution supplied a faster Max-RT (0.29 msec.) than their Min-RT (0.501 msec.) We’ve mentioned this in prior performance reports but apparently, at low workload demand, the WekaIO can take more time to perform IO (possibly because it’s highly optimized to perform well at high workload demand) than it does at high workload demand.

Significance

It’s great to see new submissions for SPEC sfs2014, especially ones that perform every workload. At the moment, EDA, DATABASE and VDI have too few submissions to present an adequate picture of how well the industry storage can perform these benchmark workloads. But more is always better.

The Oracle midrange hybrid storage proved to be a great performer given it’s storage handicap (disk). In the two workloads we didn’t present, VDA and SWBUILD, it did ok given that it was a hybrid solution and most other vendor submissions were AFA.

We are still trying to determine the best way to report SPEC sfs2014 results and as such, we may experiment with variants of the above charts. Any ideas on other metrics of interest to report, please do let us know. Furthermore, suggestions on how to improve any of our performance analyses are always welcomed.

[This storage performance was originally sent out to our newsletter subscribers in March of 2020.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Dispatches are posted to our website at least a quarter or more after they are sent to our subscribers. ]  

Silverton Consulting, Inc., is a U.S.-based Storage, Strategy & Systems consulting firm offering products and services to the data storage community

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.