SCI 2008 Feb 19 latest SPC-1 and SPC-2 performance results analysis

In $/IOPS, $/MBPS, CLARiion, CX3, Dell EMC, Hitachi Vantara, IOPS, LRT, MBPS, NetApp, SPC, SPC-1, SPC-2 by Administrator

After our last performance result dispatch proved so popular we decided to split out the SpecSFS results into a separate dispatch.  Look for the new SpecSFS performance results dispatch to be released in March.

SPC-1 results

(SCI080219-001) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-001) (c) 2008 Silverton Consulting, All Rights Reserved

The latest SPC-1 results (see Figure 1) show significant improvement across many performance metrics.  HDS released SPC-1 benchmark results last October and HP and Sun, HDS OEM partners followed releasing their own results on HDS USP-V products.  TMS published new SPC-1 benchmark with #1 performance at over 290K IOPS™ and a never before seen LRT™ of under 100 microseconds.  The other major new benchmark published since our last report was for the ETERNUS8000-M1100 and it came in at #8 in the top 10 IOPS™.

In addition to these new benchmarks, NetApp has published some interesting SPC results.  NetApp’s IOPS™ results are good for a mid-range product but even more interesting is that they also published results with SnapShots active.  This marks the first time that an SPC or for that matter any formal audited benchmark result has been published with an enhanced storage service features active.   Even more provocative, NetApp released parallel results on non-NetApp equipment, namely EMC CX3 with and without active SnapViews.  More about this below.

(SCI080219-002) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-002) (c) 2008 Silverton Consulting, All Rights Reserved

As for LRT™ results (see Figure 2), TMS now holds the first two positions for the fastest IO in the industry for their current and previous generation products, with the current product reporting in with a blistering 90 microsecond LRT™.  The next two slots are held by IBM D8300 products and then coming in at #5, the newly benchmarked ETERNUS8000-M1100.  EMC CX3 with SnapView active showed surprising LRT™ results that came in under 2 msec. and is the 11th fastest storage subsystem on record.

(SCI080219-003) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-003) (c) 2008 Silverton Consulting, All Rights Reserved

In the $/IOPS™ results (see Figure 3) TMS once again dominates with their latest TMS RamSan 400 coming in at less than $0.67/IO operation.  The $/IOPS™ metric is confounded by two factors that favor TMS, IOPS rate (291K) and subsystem cost (~$200K).  The fact that the TMS subsystem is only 137GB of storage is not a factor for this metric.  After TMS, DataCore and the rest of the top 5 all come in over $4/IO operation.

(SCI080219-004) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-004) (c) 2008 Silverton Consulting, All Rights Reserved

In contrast to the $/IOPS™ results, another way to look at IO value is to use the cost of a GB storage as a factor such as IOPS/$/GB (see Figure 4).  Here, TMS’s high priced storage ($1.4K/GB) places them at a significant disadvantage and out of the top 10.  The top 5 for this metric are IBM SVC 4.2, IBM DS4700, and then the 3 HDS USP-V variants.

SPC-2 results

(SCI080219-005) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-005) (c) 2008 Silverton Consulting, All Rights Reserved

As for SPC-2 results there have only been a few new results since our last report, namely the Sun StorageTek 2530 arrays with and without RAID5.  For the second SCI report in a row three generations of IBM’s SVC take the top three MPBS™ results (see Figure 5).  The ETERNUS8000 and IBM DS8300 round out the rest of the top 5.

Neither new Sun StorageTek result came in the top 20 for MBPS™ but for $/MBPS (see Figure 6) they came in as #1 & 2 cheapest bandwidth available.  The rest of the top 6 slots in this SCI created category are dominated by Sun StorageTek products (2540 RAID 5 & mirrored and 6140 RAID5 and mirrored).

Comments on the NetApp results

(SCI080219-006) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-006) (c) 2008 Silverton Consulting, All Rights Reserved

SPC allows members to submit audited results on any vendor’s equipment, but NetApp’s results are the first time SPC has done this.  EMC was notified the morning of the release and had only a few hours to react.  EMC is not a current member of the Storage Performance Consortium and now probably will never be one.  SCI believes the EMC CX3 results with SnapView activity deserves some further discussion.  Consider Table 1

Table 1 LRT™ results as NetApp reported&

LRT™ results Without SnapShot and SnapView With SnapShot and SnapView
NetApp FAS3040 2.85 msec. 3.45 msec.
NetApp reported EMC CX3-M40 4.34 msec. 1.86 msec.

Something’s wrong here.  While NetApp’s response time degraded with SnapShot enabled, EMC’s reported response time improved.  After informal talks with NetApp and the Storage Performance Council, SCI realizes that the results reported for the LRT™ metric did not have any SnapView active.   NetApp and SPC state this was to EMC’s benefit and showing up as the 11th fastest LRT™, it definitely helped.  However, it’s an unfair to other LRT™ results to do this.

All this presents an interesting problem – what is the proper way to benchmark storage subsystem with advanced features active?  Snapshot is a relatively straightforward feature but its use is dictated by a number of concerns and its operational profile can become quite complex to figure out.  For example, what’s the right frequency to use snapshot, when do you deactivate/delete a snapshot volume(s), how much storage should be snapshot, etc.  Operational profiles for snapshot use in backup are vastly different than snapshot use in testbed creation.  Are NetApp’s usage models appropriate?

In all fairness, the full disclosure reports for both NetApp’s FAS3040 and NetApp reported EMC CX-M40 results described how SnapShot and SnapView were used during the testing and they were different.  (NetApp issued SnapShots every 15 minutes on FAS3040 and issued SnapView every hour on EMC CX3-M40 and as stated above did not issue SnapView during the LRT™ testing.)  Once again, this was to EMC’s advantage, but how can they be truly compared unless SnapShot/SnapViews are used at the same frequency.

The benchmark business is fraught with vendors sparing no expense to field a system showing the best results.  Is it any wonder that aside from TMS, the other top 5 IOPS™ results were on storage subsystems costing more than $3 million each.  Although NetApp made an attempt to optimize EMC’s subsystem, is this what EMC would have done?  We will never know.  SCI believes the right answer here is to not allow one vendor to publish results for another and when publishing results with storage features.  When publishing benchmarks with advanced storage features operational profiles have to be established ahead of time and adhered to throughout the benchmark’s various phases and if that’s impossible the benchmark needs to be withdrawn.

This performance dispatch was sent out to our newsletter subscribers in February of 2008.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.A PDF version of this can be found at

SCI 2008 February 19 SPC Performance Results Update

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community


* SPC-1 and SPC-2 results are from the Storage Performance Council website http://www.storageperformance.org.

 

& Full disclosure reports are in file numbers A00057, A00058, A00059, and A00060, available at http://www.storageperformance.org as of 14 Feb 2008.