This Storage Intelligence (StorInt™) dispatch covers the new SPECsfs2014 and SPECsfs2008 benchmarks. SPECsfs2008 is in “retirement status” and as a result, SPEC is no longer accepting any new submissions for SPECsfs2008.
As for SPECsfs2014, there are still no new submissions other than the original 4 (one for each workload) submitted by the SPEC SFS(R) Subcommittee as reference solutions. Rather than re-hashing some of the SPECsfs2008 metrics we have accumulated over the years, we thought examining the SPECsfs2014 reports to see what how the reports look and if they can be improved.
The following discussion is based on the SPEC SFS(R) 2014 ITM-33 Reference Solution database workload report. There are a number of sections to the report. SPEC has once again provided a results summary page and an HTML and Text based detailed report (linked to from the results summary page). We critique the SPECsfs2014 by describing the Good, Bad and Ugly of the report. Specifically,
- On the results summary page, SPEC now reports system configuration information such as Memory (GB) and Total Capacity (GB) as well as result metrics Databases, ORT and MB/s.
- In the header of the detailed report they list two metrics number of databases and Overall Response Time. The number of databases matches the maximum Business Metric (Databases) in the details of the report. The Overall Response Time is an average response time across the whole workload activity.
- In the Performance section of the detailed report, the database workload report now shows average latency statistics at various points during the benchmark execution. The details include Database Ops/Sec., Database MB/Sec. and the aforementioned Average Latency (msec.). In the rightmost column of this table is a Business Metrics (Databases) count that in the case of the reference submission, ranges from 2 (databases) up to 26. Presumably, the number of databases is used to scale the IO activity against the system from almost idle to maximum activity.
- In the Processing Elements – Physical section of the detailed report SPECsfs2014 is back to reporting on CPUs in the system under test and the load generators. It’s apparent when you consider the 4-storage node configuration that the 8 CPUs is the number of chips not cores. I would think the cores would also be a useful number to record in the table although it’s easily computable from the description and Qty information.
- In the Transport Configuration – Physical section of the detailed report also provides more details about the cluster interconnect and port activity. In the case of the reference architecture the cluster interconnect was QDR Inifiniband.
- The “stable storage” configuration numbers in the Storage and Filesystems table are hard to understand in the report and the “configuration diagram“ doesn’t shed much light on the subject. According to the report there were (Qty) 96 600GB disk drives used for stable storage. In the Data Protection column of this table it was reported as “16+2/2 Parity protected (default)”. Unclear to me whether this means that the system had a RAID group of 16 data + 2 parity drives with 2 spares, which would say 18 drives in the RAID group and 2 global spares for 20 disk drives per storage node, across the 4-nodes would be 80 disk drives (not 96 as reported). For the stable storage to be made up of 96 disk drives, there would need to be 24 disk drives per node. How that maps to “16+2/2 Parity protected (default)” is pretty hard to figure out. It would be useful, if somewhere the report included a detailed BoM or configuration diagram would show one of the storage nodes with a description of the number of disks per storage node.
- The same Storage and Filesystems table reports Usable GiB as 26.1TiB. It’s somewhat unclear how the 16-600GB data drives per node created a 26.1TiB usable GiB out of 38.4TiB but at ~68% of the overall capacity it’s probably ok. Probably ought to stick with one abbreviation for GiB and if the title of the table column is “Usable GiB” listing the item capacity in “TiB” is probably not good.
- In the header section of the detailed report SPEC chose to list the maximum number of Databases achieved by the system under test and ORT. It’s unclear to me what a customer would want to see most but just the number of databases seems to shortchange what is reported in the details. On the results summary page at least they add the MB/s achieved by the tested system. The only thing missing was the Database Ops/Sec. It would seem to me that this would be a useful metric to prominently display on the results page and in the header section of the report.
- The Memory configuration item provided on the results summary page and in the body of the detailed report in the Memory – Physical section is the sum of the memory used by the load generator and the storage system under test. I don’t see the value in reporting the sum of the total of memory used by the storage system and its load generator. Just providing the system under test memory would be better. I think the load generator memory configuration would be better placed in the Hardware Configuration and Tuning – Physical section, which seems to specify load generator hardware information or even a separate section altogether that just described the load generator configuration.
- Ditto for the load generator’s disk storage configuration, which is listed in the Storage and Filesystems
- Ditto for the load generator’s transport network configuration information listed in the Transport Configuration – Physical
- You almost have to look into the text of the detailed report (its also identified under the Storage and Filesystems section of the report) to understand what file protocol was actually in use for the reference submission (NFSv3). I believe this should be available on the results summary page and prominently displayed in the detailed report header.
I like the additional information provided in the database workload detailed report. The table provided in the performance section is especially interesting and will make a good addition to our analysis of NAS system performance.
Briefly looking at the SW Build, VDA, and VDI versions of the SPEC SFS(R) 2014 ITM-33 Reference Solution reports, I find that much the same critiques can be applied to these reports as well. The only exception being that each workload has a metric, specific to that workload which is not being reported on the summary page. Specifically Builds Ops/Sec, Streams Ops/Sec and Desktops Ops/Sec for the SW Build, VDA and VDI reports respectively should be reported on the results summary page and in each detailed report’s header section.
It’s now been over 2 quarters since SPECsfs2014 has been released. By this time in the SPECsfs2008 introduction there were at least 5 non-reference submissions. The lack of any non-reference submissions at this point is perplexing.
On the other hand, the changeover from SPEC SFS97_R1 to SPECsfs2008 was relatively minor compared to the current changeover from SPECsfs2008 to SPECsfs2014. To my knowledge, all they added was a CIFS version of the benchmark driver and made some modifications to the workload parameters to better reflect current field NAS use.
So the benchmark(s) didn’t change nearly as much and it still took a long time before we saw a lot of vendor submissions. So maybe the lack of vendor submissions should be expected at this point.
In the mean time, due to lack of any non-reference submissions, I have taken the opportunity to detail the benchmark changes (in December 2014’s StorInt), discuss some relatively obscure SPECsfs2008 metrics (in March 2015’s StorInt) and in this report discuss the new report information. There are probably a couple of other charts I can roll out on SPECsfs2008 data but I am running low on other “interesting” metrics. Hopefully, by the next time we report on SPECsfs activity in September, there will be some vendor submissions for SPECsfs2014 to review.
Until then, as always, suggestions on how to improve any of our performance analyses are welcomed. Additionally, if you are interested in more file performance details, we now provide a fuller version (Top 20 results) of some of these charts and a set of new NFSv3 and CIFS/SMB ChampionsCharts™ in our recently updated (April 2019), NAS Buying Guide available from our website. Also Top 20 versions of some of these charts are displayed in our recently updated (May 2019) SAN-NAS Buying guide also purchasable from our website.
[This performance dispatch was originally sent out to our newsletter subscribers in June of 2015. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Dispatches are posted to our website at least a quarter or more after they are sent to our subscribers. ]
This report was sent out to subscribers as part our free, monthly Storage Intelligence e-newsletter. If you are interested in receiving future storage performance analyses along with recent product announcement summaries, please use the QR code below right to signup for your own copy.
Silverton Consulting, Inc., is a U.S.-based Storage, Strategy & Systems consulting firm offering products and services to the data storage community