SPECstorage_2020 Performance report as of 27Jan2022

In Genomics, NewVDA, SPECstorage_2020 by AdministratorLeave a Comment

This Storage Intelligence (StorInt™) dispatch covers SPECstorage_2020 benchmarks . These are a new series of benchmarks which replace SPECsfs2014. They have introduced two new workloads, AI and Genomics and have dropped VDI and Database from the old benchmark. They have also made some tweaks to the EDA workload and now call it EDA Blended.

There are only a few (non-SPEC) vendor submissions so this is just a start of our analysis of this benchmark. The non-SPEC vendor submissions include Oracle ZFS ZS9-2, Samsung PM9A3 NVMe & WekaFS, and Super Micro SYS-220U-TNR 22 NVMe (& xfs). For SPEC’s reference submissions, they included one running in AWS with WekaFS and the other running on prem under virtualization with FreeNAS. Not all of these were submitted for every workload, but all had at least 4 submissions.

SPECstorage_2020 Genomics results

The SPECstorage_2020 Genomics workload simulates “the entire pipeline of the Genomics workflow .” It is mostly a read sequential intensive activity (70% sequential read, 12% stat, 8% sequential write). Average file size is 1.6MB with data transfer at 128KB (96%). The metrics of interest for the genomic workload are peak concurrent jobs, ORT, jobs Ops/sec and job MB/sec.

Figure 1 shows the top (4) systems for Genomics peak concurrent jobs supported.

Figure 1 SPECstorage_2020 top maximum number of concurrent genomic jobs supported

In Figure 1, we can see  the Samsung NVMe SSD solution with WekaFS running, 6 storage nodes with 15 PM9A3 3.84TB NVMe SSDs and 512GB DRAM each (90 SSDs and 3TB DRAM total, was #1 ranked system with over 1100 peak concurrent (genomic) jobs supported. 

Furthermore, the #2 ranked solution was a SPEC reference submission running in AWS cloud with WekaFS and achieved 200 peak concurrent (genomic) jobs. This submission was running 8 (i3en.12xlarge) EC2 instance storage servers, with 4 7.5TB NVMe SSDs and 384GB of DRAM each (32 SSDs and 3TB of DRAM total).

As my readers will no doubt recall, WekaFS pretty much dominated the old SPECsfs2014 benchmark results and looks to continue that tradition here. For our last report on that benchmark, WekaFS systems placed #1 in 3 of the old workloads and #2 in the remainderp

It’s somewhat surprising that the AWS instances running WekaFS was ~5.6X slower than the on prem version. However, their networking configuration was the main difference between the two submissions. The AWS solution used (16 ports of) 50GbE networking connections while the Samsung version used (24 ports of) 200GbE. And of course, the performance of the CPU cores was probably not similar, even though the total number (384) of cores was the same.

Next, we show the top (4) Overall Response Time (ORT) for Genomic workload submissions in Figure 2. Recall that ORT is the average response time for a submission, from the smallest number of concurrent jobs to the peak.

Figure 2 SPECstorage_20202 Genomics workload top overall response times (ORT)

In Figure 2, the Super Micro SYS-220U-TNR 22 NVMe system came in as our #1 with a 0.2 msec ORT. The Super Micro s was running a single server with the xfs file system and 22 7.68TB NVMe SSDs, 1TB of DRAM and 8 100GbE networking connections.

As ORT is an average response time over a whole submission, the maximum number of jobs attained by a solution can matters a lot. For the Super Micro submission, it only attained 33 concurrent genomic jobs at its peak, so its ORT is from a workload of around 18 jobs.

In contrast, the #2 Samsung and WekaFS solution with an ORT of 0.4msec, attained over 1100 jobs at its peak. And the minimum number of jobs for the Samsung and WekaFS submission was 80 with a response time of ~0.2msec. So, we would say that the WekaFS at a similar workload as the Super Micro would meet or beat, its response time.

Unclear how to fix ORT. By its definition, it’s the average response time over a submissions total number of workload increments. It would be much more meaningful if it was at, say the same level of activity. But for these benchmarks the average activity can span 2 orders of magnitude, and at such a wide difference ORT loses its relevance.

SPECstorage_2020 VDA results

Like the SPECsfs2014 VDA workload which precedes it, the SPECstorage_2020 workload simulates video capture (100% sequential write) with a companion application (84% random read) which examines video. The metrics of significance for VDA is concurrent video streams, ORT, stream Ops/sec and stream MB/sec. Each stream operates at roughly the rate (36Mb/s) of hi-def video.

In Figure 3, we show the top (5) systems in peak number of concurrent video streams for the VDA workload

Figure 3 SPECstorage_2020 VDA top concurrent streams

In Figure 3, the #1 ranked system with 8000 peak concurrent streams, is once again, the Samsung and WekaFS solution. The hardware and software configuration used for the VDA benchmark submission matches that used for the Genomic benchmark above.

The #2 ranked submission with 1650 peak concurrent streams, is the Oracle ZFS ZS9-2. The Oracle system used 2 storage servers each supporting a single ZFS file system with 2TB of DRAM and 80 14TB disks plus 4 0.2TB log and 4 7.68TB read cache SAS SSDs (total of 2 ZFS file systems, 4TB DRAM, 160 disk drives and 16 0.2TB and 16 7.68TB SAS SSDs). The Oracle system used 100GbE networking over 8 connections per server (16 total).

Networking speed appears to the key factor in the maximum amount of concurrent video streams that can be supported.

In Figure 4, we show the SPECstorage_2020 VDA top (5) ranked ORT systems.

Figure 4 SPECstorage_2020 VDA top ORT results

We call it a tie for 1st place, with both the SPEC ref solution using AWS-WekaFS and Samsung-WekaFS at ~2.0 msec ORT. The SPEC Ref AWS-WekaFS hardware configuration was the same as discussed above in the Genomics workload section.

Again, ORT is an average and for the AWS-WekaFS, it represents the response time at ~100 concurrent video streams while for the Samsung-WekaFS system, it represents the response time at ~4000 concurrent streams.

Significance

It’s about time we see some emphasis on AI and Genomics workloads in a serious storage benchmark. These activities process a lot of file (and object) data. Kudos to SPEC for taking this needed IO benchmarking activity on.

Follow on SCI SPECstorage performance reports will cover the AI, EDA Blended and SWbuild workloads. And over time, we hope to generate our own metrics to assess the performance of various submissions for these workloads.

This is our initial analysis of SPECstorage workloads. So, if you have any ideas on other metrics of interest to report, please do let us know. Furthermore, suggestions on how to improve any of our performance analyses are always welcomed.

[This system/storage performance reopen was originally sent out to our newsletter subscribers in January of 2022.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Dispatches are posted to our website generally a month or more after they are sent to our subscribers. ]

Silverton Consulting, Inc., is a U.S.-based Storage, Strategy & Systems consulting firm offering products and services to the data storage community

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.