SCI 2012 May 29 Latest SPC-2 benchmark MBPS performance results analysis

In Data storage, ETERNUS DX440-S2, HPE, IBM, Infinite Storage 5500-SP, LDQ, LFP, MBPS, Oracle, P9500, SGI, SPC, SPC-2, Sun ZFS Storage 7420, SVC, Texas Memory Systems, Uncategorized, VOD, XIV, XIV Gen3 by AdministratorLeave a Comment

We return now to the preeminent block storage benchmark, the Storage Performance Council SPC results*.  There has been one new SPC-1 and four new SPC-2 submissions since our last report, one SPC-1, the SGI Infinite Storage 5500-SP and all the rest were SPC-2, the Fujitsu ETERNUS DX440-S2, the HP P9500 XP Disk Array, the IBM® XIV Gen3 and the Oracle Sun ZFS Storage 7420 Appliance. As the lone SPC-1 submission didn’t enter the top 10 in any chart, we focus this report on SPC-2 results.

SPC-2*results

We start our discussion with the top 10 MBPS™ performance for SPC-2.

SCISPC120529(001) (c) 2012 Silverton Consulting, All Rights Reserved

SCISPC120529(001) (c) 2012 Silverton Consulting, All Rights Reserved

Figure 1 Top 10 SPC-2* Composite MB/sec

Higher is better on the MB/sec or MBPS results.  Recall that MBPS is a composite score using 3 different workloads (file processing, database queries, and video). Here we can see all the new SPC-2 submissions with HP and Sun coming in at #1 and 2 respectively, the IBM at #7 and the latest Fujitsu at #10.

The main surprise here is that the IBM XIV (RAID1) came in so well sporting only 180-2TB 7200Rpm drives.  Most of the remaining top 10 used 256 or more 300GB SAS 10/15Krpm drives or SSDs (TMS).  This XIV run was submitted last December as an SPC-2/E submission, sorry I didn’t catch it earlier.  SPC-2/E is similar to the SPC-1/E as both run the full suite of SPC tests and as such, can then be ranked against the standard benchmark.

Next we turn to a different way to look at MBPS

SCISPC120529(002) (c) 2012 Silverton Consulting, All Rights Reserved

SCISPC120529(002) (c) 2012 Silverton Consulting, All Rights Reserved

Figure 2 SPC-2 MB/sec spider chart

I always enjoy looking beneath composite scores such as SPC-2 MBPS to see how storage performance varies for different components of the metric.  For example, it’s interesting that every single top 10 MBPS system does best with the large database query (LDQ) performance than the other two workloads.  On the other hand, large file processing (LFP) had the worst performance for 5 out of the top10 and video on demand (VOD) seems to vary all over the place, worse for the IBM XIV and the older IBM SVC 4.2 storage systems and almost equal to to best (LDQ) for TMS and the HP P9500 storage.  I assume all this has as much to do with SPC-2’s defined workloads as it has to do with individual storage system caching capabilities.  Potentially an interesting research project if some company was interested in funding it.

Next we turn to storage MBPS/drive spindle.

SCISPC120529(003) (c) 2012 Silverton Consulting, All Rights Reserved

SCISPC120529(003) (c) 2012 Silverton Consulting, All Rights Reserved

Figure 3 Top 10 MBPS/drive

As shown in Figure 3, MBPS/drive has a pretty narrow performance range and the only new showing here is the IBM XIV Gen3 (#9). Considering that even the slowest 2TB/7200rpm drive can sustain 155MB/sec, there is still a ways to go before we exhaust disk performance.

Once again, it’s somewhat surprising that it did so well with 2TB 7200Rpm disk drives.  Considering that even the 2TB/7200Rpm drive can sustain 155MB/sec there is still a ways to go here.  Remember that the Sun J4200 had no RAID protection whatsoever which is probably one reason it performed well here.  A couple of caveats are in order:

  • These are disk-only submissions, anything with SSDs was eliminated from consideration.
  • Storage systems with over 140GB per spindle were the only systems considered for this chart.

SPC-2 disk-only submissions range from 36GB to 2TB disk drives so we thought it best to eliminate anything below 140GB to give everyone a better chance at excelling.  Although, having a 2TB disk drive may force us to reconsider this exclusion. Especially since it performed so well.

Finally, we turn to a scatter plot for MBPS against number of drives

SCISPC120529(004) (c) 2012 Silverton Consulting, All Rights Reserved

SCISPC120529(004) (c) 2012 Silverton Consulting, All Rights Reserved

Figure 4 Scatter plot: MBPS vs. number of disk drives

We supply this chart to help answer the question as to whether subsystem MBPS performance is driven purely by spindle count.  An R**2 of 0.35 clearly indicates that SPC-2 MBPS rate is not correlated to disk count. For this chart we include all (36GB to 2TB) disk-only submissions.

The wider variance at higher drive counts and MBPS rates seems informative.  Consider the vertical pair of results at slightly less than 800 disk drives.  We consider the higher performer (IBM DS8800) to be the one obtaining more MPBS per spindle. It achieved ~9,750 MBPS vs. its brethren (IBM SVC 4.1 RAID5) at ~4500 MBPS.  The fact that the DS8800 system doubled throughput performance for the same number of drives indicates that it performed better.

Significance

It’s good to see some new SPC-2 results especially at the high-end MBPS performance level.  To have all four new submissions place in the Top 10 seems to say that storage is getting much better at SPC-2 workloads and probably says it’s time to change the test.

Contrary to SPC-1 results, SPC-2 results are not correlated very well with the number of disk drives. We have had a number of discussions over the years on how benchmarks can be gamed just by throwing more spindles or other hardware at the problem. At least for SPC-2 and disk spindles, this is clearly not the case.

As always, suggestions on how to improve any of our performance analyses are welcomed.  Additionally, if you are interested in more block performance details, we now provide a fuller version (top 30 results) of all these charts and more in SCI’s SAN Storage Buying Guide available from our website[1].

~~~~~

[This performance dispatch was originally sent out to our newsletter subscribers in May of 2012.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more, in-depth analysis of SAN storage system functionality, features and performance please see our recently revised (May 2019) SAN Storage Buying Guide available on our website.]

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.