SCI 2010 May 27 latest SPC-1 benchmark results analysis

In $/IOPS, DS8700, IBM, IOPS, LRT, SPC, SPC-1 by Administrator

We return now to that classic block storage benchmark, Storage Performance Council (SPC) results*.  We mostly discuss SPC-1 latest results below with a short discourse on an interesting SPC-1C result.

SPC-1*results

There have been five new SPC-1 results this past quarter – two Huwaei Symantec Oceanspace subsystems the S2600 and the S5600, two from Fujitsu the ETERNUS DX400 and DX8400 and the latest IBM 8700 (R5.1) with SSDs, SATA drives and Easy Tier automation.  It’s unclear whether IBM’s Easy Tier had sufficient runtime to effect performance optimization for any SPC-1 runs (see discussion below).  Nevertherless, none of these subsystems made it into the top 10 in IOPS™.  However, three of them did make it into the best LRT™ results discussed below.

(SCISPC10527-001) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISPC10527-001) (c) 2010 Silverton Consulting, All Rights Reserved

The new IBM DS8700 and the two Fujitsu systems showed up well in as #6, 8 & 9 in top 10 LRT results.  Recall that LRT measures the average response time during the 10% load factor run and as such, should correspond to the best response time from a relatively idle subsystem.  We have discussed the other Top 10 LRT subsystems in prior dispatches and do not cover them here#.

(SCISPC10527-002) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISPC10527-002) (c) 2010 Silverton Consulting, All Rights Reserved

Both Huawei Symantec subsystems showed up well in most of the $ based comparison and here one can see the $/IOPS metric where they came in at #6 & 8.  One caution here is that Huawei subsystem pricing was given in CNY which we converted into USD for comparison purposes at CNY ~6.8 to the $.  Nonetheless, they compare well in price performance as SPC now calls it.

In addition, it has been brought to my attention that the SPC-1/E benchmark runs are equivalent to the SPC-1 runs.  As such, we have added Xiotech’s recent SPC-1/E run for their Emprise 5000 with 146GB Huricane drives which now comes in at #7.

(SCISPC10527-003) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISPC10527-003) (c) 2010 Silverton Consulting, All Rights Reserved

In previous discussions we showed SPC-1 scatter plots for IOPS vs. Capacity and IOPS vs. $/GB.  In this report we now return to IOPS vs. LRT.  As one enhancement, we examined statistical trendlines (not shown) for this data but there does not appear to be any with a high correlation, so have left them out.

The two new Fujitsu systems, the DX440 and DX8400 show up well at the ~1.5msec line with  ~100K and ~170K IOPS respectively and the medium cost subsystem at ~1msec LRT with ~33K IOPS is the new IBM DS8700 with Easy Tier, SSD and SATA disks.

(SCISPC10527-004) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISPC10527-004) (c) 2010 Silverton Consulting, All Rights Reserved

We have a new winner for this chart as the low-end Huawei Symantec S2600 subsystem managed to crack 350 IOPS™ per drive.  Their higher-end subsystem was a respectable 258 IOPS™ per drive but didn’t break into this top ten.  Unclear why the low-end Oceanspace did so well, it was just using 48-146GB 15Krpm SAS drives in a RAID 1 configuration.

However we have also updated this chart to include SPC-1/E data and now show the Xiotech’s 146GB and 600GB drive SPC-1/E runs come in at #2 and #5.  Not bad for a 600GB disk drive.

You will recall that we have excluded pure SSD subsystems from this analysis as they tend to be off the chart, literally.  Not sure whether Easy Tier should or should not be ok here but as it came in at the middle of the pack, we felt including was justifiable.

(SCISPC10527-005) (c) 2010 IBM, All Rights Reserved

(SCISPC10527-005) (c) 2010 IBM, All Rights Reserved

As discussed above IBM was using SSD and SATA disks along with new automated storage tiering called Easy Tier.  The effect of Easy Tier& is to move “hot” extent data from SATA disk to SSD.  Hot is determined by subsystem activity monitoring over some time period.

One can see in this SPC-1 generated graph that I/O activity began ~15K IO/s and peaked out at ~50K IO/s before the SPC-1 driver dropped down to the requested workload (~33K IO/s).  The requested workload amount is chosen at the discretion of the vendor running the test but from our perspective it looks like it could have sustained 50K IO/s for the rest of the 24hr run.  The other thing of interest is the absolute lack of variability in the IO/s for the remainder of the run which probably says something about SPC-1’s working set size.

Other SPC results

(SCISPC10527-006) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISPC10527-006) (c) 2010 Silverton Consulting, All Rights Reserved

Another new SPC submission was for its SPC-1C, which as you may recall is a component level benchmark.  Oracle submitted a Sun F5100 Flash drive storage system which blew out all the other components by ~70X.  Please note that this chart shows 100% IOPS load on a logarithmic scale and without this one could barely see the other results.  The F5100 was SAS connected to a SPARC server running the benchmark.  In fact with over 300K IOPS™ the Oracle component storage would easily have qualified in the top 3 IOPS results for the normal SPC-1 if only it supported FC attachment.   Seems to be a “screamer” SAS-DAS storage.

Significance

There has been only one new SPC-2 submission this past quarter for the Fujitsu ETERNUS DX80 but it did not place in the top 10 in MBPS and so, we do not show any results for SPC-2.  As for the other SPC benchmarks, there have been no results this past quarter.

It seems like SPC might need to come up with a “pure SAS-SSD” benchmark.  I would think that there are other SAS-Flash storage vendors who might want to take on the Oracle F5100 juggernaut in a separate competition.

Also as automated storage tiering (like Easy Tier) goes mainstream it’s unclear how benchmarks should change to take advantage of these capabilities and how to better report on such capabilities.  The vast majority of current SPC-1 submissions only use one drive type.  Having multiple drive types and tiering automation certainly confounds any accurate performance comparisons.

As always if you have any suggestions on how we can improve our SPC or any performance analysis, please don’t hesitate to drop us a line.  Our contact information can be found in the footer of this page.

This performance dispatch was sent out to our newsletter subscribers in May of 2010.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

2010 May 27 SCI's latest analysis of SPC results

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.


* All results from www.storageperformance.org as of 27 May 2010

 

# See http://silvertonconsulting.com/cms1/news-4/

& From http://www.storageperformance.org/benchmark_results_files/SPC-1/IBM/A00092_IBM_DS8700_EasyTier-SSDs/a00092_IBM_DS8700_EasyTier-SSDs_SPC1_full-disclosure.pdf