SCI 2011 May 27 Review of latest SPC performance results

May 272011

We return now to our favorite block storage benchmark, Storage Performance Council (SPC) results*. There have been three new SPC-1 submissions since our last report, TMS RamSan-630, IBM StorWize V7000 and Pillar Data Axiom 600 Series 3, one new SPC-1 also from TMS, their RamSan-630 (first all SSD system for this throughput benchmark), and an new SPC-1C (component benchmark) the new Seagate Pulsar-XT.2 SSD.

SPC-1*results

We start our discussion with the top 10 IO/sec, or as SPC calls it IOPS™

SCISPC110527-001 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 1 Top 10 SPC-1* IOPS

Higher is better here. The only new submission on this chart is the TMS RamSan-630 and came in as new number 1 at 400.5K IOPS. None of the other new submissions made into the top10 so the other systems here have all been discussed in previous dispatches. The new RamSan-630 had about 8TB of user storage using 20-640GiB SSDs. It also used 8Gb FC front and backend interfaces.

Next let’s turn to storage price performance and LRT™.

SCISPC110527-002 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 2 Top 10 LRT results

Lower is better on the LRT (least response time) results. The new RamSan continued TMS’s almost unique run with another sub 0.5msec LRT run (they came in with 0.22msec LRT). For a long time I never thought I would see sub-1 msec response times but I could never have imagined even 220 usec. let alone the all time winner TMS RamSan-040 at 90 usec. I would like a very deep dive into their code and hardware to understand this lot’s better, because whatever they’re doing deserves much wider dissemination.

SCISPC110527-003 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 3 Top 10 $/IOPS

I guess it’s official now, owning the top four slots, TMS SSDs provide the best $/IOPS around. The RamSan-630 8TB configuration came in at ~$413K and while not as cost effective ($/IOPS) as the RamSan-400 is capable of significantly more IOPS. Recall that the two SUN systems (#5&6) had no RAID protection, which would provide a significant cost advantage over the other, mostly RAID1 subsystems, but seriously increases risk of data loss.

SCISPC110527-004 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 4 Scatter plot: IOPS versus Number of Drives

We have added a new chart attempting to determine if drive counts lead to better IOPS. We have eliminated all SSD or NAND caching systems from this analysis. And the regression is pretty good. Given the regression equation above in order to match the 400.5K of IOPS for the number one TMS system one would need to field ~2240 disk drives. I have not eliminated older or slower drives from this but it clearly shows that we should be able to attain ~175 IOPS per disk drive.

The other interesting item here is that the regression seems to get worse as we get over 500 drives or more, indicating that systems capable of using these many drives effectively can seem to obtain more than ~175 IOPS per drive.

The new IBM StorWize V7000 and Pillar Data Axiom 600 Series 3 systems show up on this chart in the group between 50K and 100K IOPS.

SPC-2 Results

There was only one new submission in the SPC-2 arena this past quarter and that for TMS RamSan-630.

SCISPC110527-005 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 5 SPC-2 MBS results

Higher is better on MBPS chart. Recall that the SPC-2 MBPS is a composite score that averages storage performance for large file processing, large database query, and video on demand sequential workloads. The new TMS RamSan-630 came in at #4 at ~8300 MBPSon this chart and for some reason is the only SSD intensive system to be have a submission. Unclear why this is although throughput has not necessarily been a strongpoint for SSDs.

SCISPC110527-006 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 6 SPC2 MBPS spider chart

We have discussed this chart before because it shows how the three workloads in the MBPS composite compare by system. I find it interesting that the VOD workload is significantly different for four out of the top 5 systems versus the RamSan-630. Now, a majority of these workloads are heavily sequential but there is obviously something else going on with LFP and VOD. Given the above, I would say there must be a heavier write component to the LFP workload (relative to the VOD and LDQ workloads) and a heavier random component to the VOD workload (relative to the LFP and LDQ workloads).

Significance

It’s good to see the SPC-1 recent submissions by IBM, Pillar Data and TMS and the SPC-2 submission from TMS. Performance continues to be an important metric for block storage subsystems and random and sequential performance are complementary views of storage performance.

Although this time SSDs took center stage, we know how to do better if necessary with disks alone. Nonetheless, it seems on a price-performance basis, SSDs almost can’t be beat.

There seems to be a lack of SSD SPC-2 benchmarks. There doesn’t seem to be any real reason why SSDs couldn’t make a better showing here if they wanted. I keep seeing a NAND storage vendor’s multi-TV monitor showing literally 1000s of videos simultaneously which means it ought to be able to kill the VOD portion of the benchmark. Of course how SPC would handle a PCIe storage device is another question. Perhaps we need a SPC-2C benchmark.

As always, any suggestions on how to improve our performance analyses are welcomed. Additionally, if you are interested in more details, we now provide a fuller version of all these charts in SCI’s SAN Storage Briefing which can be purchased separately from our website[1].

[This performance dispatch was originally sent out to our newsletter subscribers in May of 2011. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.]

—–

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 26May2011

[1] Available at http://silvertonconsulting.com/cms1/products/

SCI 2011 Feb 21 Latest SPC-1 and SPC-2 performance results

Feb 272011

SPC-1*results

We start our discussion with the top 10 IO/sec, or as SPC calls it IOPS™

SCI20110227-001 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 1 Top 10 SPC-1* IOPS

Higher is better here. The only new submission on this chart is the TMS RamSan-630 and it once again stepped in as new number 1 at 400.5K IOPS. The remaining systems here have all been discussed in previous dispatches. The new RamSan-630 had about 8TB of user storage using 20-640GiB SSDs. It also used 8Gb FC front and backend interfaces. The RamSan-630 was priced at about $419K for .

Next let’s turn to storage price performance and $/IOPS.

SCI20110227-002 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 2 Top $/IOPS results

Lower is better on the $/IOPS. The only new submission on this chart is the IBM DS3524 Express Turbo (6.7TB) that came in at #9 with $3.26/IOPS using 4-Gb/s SAS connecitions, 4GB of cache, and RAID1 protection for 48-300GB 10Krpm SAS disk drives. TMS storage continues to dominate this chart with the top 3 slots at $0.67, $1.13 and $1.50 respectively. Recall that the two SUN systems (#4&5) had no RAID protection, which would provide a significant cost advantage over the other, mostly RAID1 subsystems, but seriously increases risk of data loss.

SCI20110227-003 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 3 Top IOPS/drive spindle

Higher is better on IOPS/drive chart. Once again, the new entry here is the IBM DS3524 Express Turbo (6.7TB) that came in at #8 with over 308 IOPS/drive spindle. Recall that this chart excludes subsystems using any amount of SSDs and as of this report, also eliminates any subsystem that uses NAND for cache. Similar to the $/IOPS results above, the Sun J4400 subsystem had no RAID protection and enjoys an IOPS/drive advantage because of this.

SPC-2 Results

We have not seen much activity in SPC-2 submissions of late and was beginning to think no one cared about sequential performance. Thankfully, IBM has seen the light and released 3 new SPC-2 benchmarks for all their current enterprise class storage. Unclear why only a limited number of vendors participate in the SPC-2 submissions, mostly IBM and Oracle Sun submissions with a smattering from other vendors.

SCI20110227-004 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 4 SPC-2 MBS results

Higher is better on MBPS chart. As one no doubt remembers, the SPC-2 MBPS is a composite score that averages storage performance for large file processing, large database query, and video on demand sequential workloads. The new IBM DS8800 came in at #1 with over 9700 MBPS, using 16-8GFC links, with RAID5 protection over 768-146GB disk drives. Also recall that the HP XP24000 is an OEM version of HDS USP-V, so the #2 and #3 slots are essentially the same subsystem.

SCI20110227-005 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 5 SPC2 MBPS/drive spindle

Higher is better on the MBPS/drive chart. This chart represents yet another way to view the overall MBPS performance but this time on a per spindle basis. The new IBM DS3524 Express Turbo broke in at #8 on this chart with 41.5 MBPS/drive using 4-6Gb/s SAS connections, 4GB of cache, and RAID 5 protection over 60-300GB 10Krpm SAS disk drives. Similar to the discussion on the SPC-1 IOPS/drive chart we eliminate subsystems using SSDs or NAND cache from this list. Recall that the Sun J4200 had no RAID protection and as such, enjoyed an MBPS/drive advantage.

Significance

It’s good to see the SPC-1 and SPC-2 recent submissions by IBM and others. Performance continues to be an important metric for block storage subsystems and random and sequential performance are complementary views of storage perfomance.

We are starting to see the emergence of high performing block storage subsystems based on a cluster of nodes (see Huawei 8- and 4-node SPC-1 results). This approach first emerged with 3PAR’s storage system (now owned by HP). Of course the other prominent cluster based storage subsystem is EMC’s VMAX but they do not submit SPC benchmark results.

We are still awaiting the latest HDS VSP benchmark results which hopefully, should take the block storage performance up another notch or two when published. Also NetApp’s high-end storage with flash cache has yet to submit any SPC results.

This performance dispatch was sent out to our newsletter subscribers in February of 2011. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2011 Feb 21 Latest SPC results analysis (PDF 769.2 KiB)

—-

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

* All results from www.storageperformance.org as of 21Feb2011

[1] Available at http://silvertonconsulting.com/cms1/products/

SCI 2010 Nov 18 Latest SPC results

Nov 182010

We return now to that classic block storage benchmark, Storage Performance Council (SPC) results*. There have been five new SPC-1 and -1/E submissions since our last report, namely two from Xiotech (Emprise 5000 with 300GB and 600GB drives), IBM Storwize V7000, Huawei Symantec S8100 4-node, and NetApp FAS3270A (SPC-1/E). You may recall that SPC-1/E is an energy monitored version of the SPC-1 test so we have been combining these two test results into our SPC-1 analysis

SPC-1*results

The chart that has changed most since our last report is the IOPs/drive results.

SCISPC101118-001 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 1 Top 10 SPC-1* IOPS/Drive spindle

Higher is better on the IOPS/drv chart. Recall that our IOPS/drive chart excludes drives under 146GB and eliminates systems that use SSDs or Flash Cache. The #1 and 6, (new Xiotech 300GB and 600GB, respectively) results on this chart are new for this quarter. On a purely IOPs per drive perspective, Xiotech seems to be doing quite well with 5 out of the top 10. Obtaining ~400 SPC-1 IOPS/drive out of a 15Krpm/300GB disk drive seems pretty good to me, considering the raw drive probably can do about ~286 average random read seeks per second (e.g. Cheetah 15K.5/300GB drive with an average 3.5msec read seek) and this doesn’t account for rotational latency or data transfer time.

Silverton Consulting, Inc. StorInt™ Dispatch

SCISPC101118-002 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 2 Top LRT™ results

Lower is better on the LRT. The top Least Response Time (LRT) chart also has new entries coming in at #5 and #10 for the new NetApp FAS3270 and Xiotech Emprise 5000/300GB drive subsystems. As the NetApp system has 1TB of Flash Cache, it represents an interesting mid-level performance between the all SSD systems to the left and the mostly [except HP EVA 6400 and IBM DS8700 (R5.1) w/SSDs] all disk systems to the right of them. It stands to reason that SSD and Flash Cache systems would perform well on LRT but the three all disk drive systems showing up here seem to have some secrets to tell as well on how to perform faster.

SCISPC101118-003 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 3 Top IOPS/$/GB

Higher is better on IOPS/$/GB which is an attempt to normalize performance. The other new winner was the recent Huawei submission in the IOPS/$/GB category. Huawei’s system pricing is listed in CNY and we use a conversion rate based on the day of our analysis report (0.1507$:1CNY as of 11/10/2010). Although conversion rates may shift over time, given Huawei’s low cost (CNY 7,847,296) and its high IOPS rate (~160K) it’s liable to be in the top 10 IOPS/$/GB for quite some time.

SCISPC101118-004 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 4 SPC-1/E IOPS per Watt under varying workloads

Higher is better on IOPS/Watt chart. The SPC-1/E benchmark submissions all report on IOPs/Watt measurements taken at distinct IO workloads. At this time these four represent the only available submissions for SPC-1/E.

As discussed previously[1] the three SPC-1/E workloads (nominal, medium and high) all measure different IO intensity and idle time. Also realize that the HP EVA 6400 only has SSDs.

One can see the impact of NetApp’s Flash Cache and HP’s SSDs at both the medium and high workloads as compared to the Xiotech systems (which only have disks). Moreover, we mentioned last time that the significant IOPS/Watt differences for the nominal workload seem entirely due to the SSDs & Flash Cache drawing similar power across all workloads but delivering correspondingly less IOPS at nominal with its high proportion of idle time. In contrast, the Xiotech all disk systems seem to draw less power when little used or idle.

Significance

Other performance charts for SPC-1 and SPC-2 were unchanged except for some of the SPC-1 scatter plots and have been discussed in previous reports. There have been no new SPC-2 benchmark submissions over the past two quarters, which seem to have been slowing down of late – unsure why.

There was one additional SPC-1C submission from Oracle but I am having some difficulty comparing a component level performance with Oracle’s 3-Flash Accelerator (PCIe) cards against single and multiple drive submissions. Perhaps I will figure this out for next time.

NetApp seemed to have refreshed their entire mid-range and enterprise product lines this past week and the FAS3270 is the top of their mid-range line. I was disappointed not to see anything on the HDS VSP subystem, also new as of last month, but maybe they’re too busy shipping systems to benchmark one. We will certainly let you know as these products show up in future SPC analyses.

As always, any suggestions on how to improve our SPC analysis are welcomed. Additionally, we now provide a fuller version of all these charts included in SCI’s SAN Storage Briefing which can be purchased separately on our website[2].

This performance dispatch was sent out to our newsletter subscribers in November of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCIs 2010 Nov 18 Latest SPC results analysis (PDF 1.6 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

* All results from www.storageperformance.org as of 10 Nov 2010

[1] Available at http://silvertonconsulting.com/cms1/dispatches/

[2] Available at http://silvertonconsulting.com/cms1/products/

SCI 2010 Aug 27 latest SPC benchmark results analysis with storage energy use

Aug 272010

We return now to that classic block storage benchmark, Storage Performance Council (SPC) results*. There have been only two new SPC submissions since our last report, namely the Oracle Sun Storage 6780 and HP StorageWorks 6400 EVA (with SSD) for SPC-1 and SPC-1/E respectively. You may recall that SPC-1/E is an energy monitored version of the SPC-1 test so we have been combing the two test results into one series

SPC-1*results

The chart that has changed most since our last report is LRT results.

(SCISPC10827-001) (c) 2010 Silverton Consulting, All Rights Reserved

The new HP 6400 EVA with SSDs showed up at #8 on this chart. Not as good as the TMS SSD runs but not bad for a mid-range subsystem. The EVA supported RAID 5 and had 8-73GB SSDs which put it on the small side. Nonetheless, the EVA SSD subsystem was priced around ~$121K US, which puts it at ~$350/GB pricy for storage today but this is SSD.

(SCISPC10827-002) (c) 2010 Silverton Consulting, All Rights Reserved

This chart also changed with the addition of the Oracle Sun 6780 with 146GB/15Krpm FC drives but I would defy anyone from showing us what has changed. For the discerning among you, the Oracle subsystem shows up at around 62K IOPS with an LRT of ~1.8msec and costing ~$43K US.

(SCISPC10827-003) (c) 2010 Silverton Consulting, All Rights Reserved

The SPC-1/E benchmark submissions report on IOPs/Watt measurements taken at Nominal, Medium and High daily workloads. As one can see the HP EVA with SSDs did fairly well at the Medium and High workloads but as it’s SSD based storage we would expect it to. But the Nominal workload IOPs/W for the SSD subsystem is worthy of discussion.

SPC defines moderate and heavy workloads at 50 and 80% of maximum reported performance respectively and then defines

Nominal daily workload as 16 hours of idle time and 8 hours of moderate workload,
Medium daily workload as 6 hours of idle time, 14 hours of moderate workload and 4 hours of heavy workload, and
High daily workload as 0 hours of idle time, 6 hours of moderate workload, and 18 hours of heavy workload.

As one can see the subsystem will determine the levels of IOPs used to determine nominal, medium, and heavy. Curent results show two things about the HP EVA SSDs

The idle power consumption of HP EVA SSDs is pretty high compared to the Xiotech’s systems.
The relative lack of power difference between nominal (465.0w) and high (471.9w) workloads for SSDs punishes them in the nominal usage scenario.

(SCISPC10827-004) (c) 2010 Silverton Consulting, All Rights Reserved

For the three current SPC-1/E benchmarks (Xiotech Emprise 5000 with 146GB drives, Xiotech Emprise 5000 with 600GB drives and the new HP StorageWorks EVA with SSDs) we show a new bubble chart depicting power and IOPS for medium and high workloads. The smallest bubble pair shown above is the Xiotech with 146GB, the middle size bubble pair is the Xiotech with 600GB and the largest bubble pair is the HP EVA with SSDs.

So why don’t we see a significant increase in power consumption with the SSDs. We have discussed this before but SSD power consumption doesn’t increase much as you drive it harder. So the 50% IOPS rate consumed ~469 watts and the 80% IOPS rate consumed about ~473 watts. Unclear why the SSD subsystem actually consumed more power than the other two disk drive subsystems but realize the EVA was a fully configured storage subsystem with FC attached SSDs and 8GB of cache (not to say the Emprise subsystems were not).

Significance

The other charts for SPC-1 and SPC-2 were unchanged and have all been covered in prior reports. SPC benchmark submissions have been slowing down of late. Unsure but this might be indicative of new products coming out, delaying submissions.

SSD storage energy profiles are certainly different than hard drives. I would have thought that any SSD subsystem would consume less power than a comparable hard drive subsystem. But that certainly wasn’t the case for the Xiotech vs. HP EVA. Nonetheless, the EVA SSD subsystem did put up some interesting response times and on a iops/Watt basis it did very well in the medium and high daily workloads.

We can only hope for more SPC-1/E test submissions. It’s exactly the same as the SPC-1 testing except for the power monitoring and it would supply power consumption metrics as well as performance with the same submission. Makes sense to me, but I don’t have to pay for the monitoring equipment.

As always if you have any suggestions on how we can improve our SPC or any performance analysis, please don’t hesitate to drop us a line. Our contact information can be found in the footer of this page.

This performance dispatch was sent out to our newsletter subscribers in August of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 Aug 27 Latest Storage Performance Council (SPC) results analyzed (PDF 837.3 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

* All results from www.storageperformance.org as of 27 Aug 2010

SCI 2010 May 27 latest SPC benchmark results analysis

May 272010

We return now to that classic block storage benchmark, Storage Performance Council (SPC) results*. We mostly discuss SPC-1 latest results below with a short discourse on an interesting SPC-1C result.

SPC-1*results

There have been five new SPC-1 results this past quarter – two Huwaei Symantec Oceanspace subsystems the S2600 and the S5600, two from Fujitsu the ETERNUS DX400 and DX8400 and the latest IBM 8700 (R5.1) with SSDs, SATA drives and Easy Tier automation. It’s unclear whether IBM’s Easy Tier had sufficient runtime to effect performance optimization for any SPC-1 runs (see discussion below). Nevertherless, none of these subsystems made it into the top 10 in IOPS™. However, three of them did make it into the best LRT™ results discussed below.

(SCISPC10527-001) (c) 2010 Silverton Consulting, All Rights Reserved

The new IBM DS8700 and the two Fujitsu systems showed up well in as #6, 8 & 9 in top 10 LRT results. Recall that LRT measures the average response time during the 10% load factor run and as such, should correspond to the best response time from a relatively idle subsystem. We have discussed the other Top 10 LRT subsystems in prior dispatches and do not cover them here#.

(SCISPC10527-002) (c) 2010 Silverton Consulting, All Rights Reserved

Both Huawei Symantec subsystems showed up well in most of the $ based comparison and here one can see the $/IOPS metric where they came in at #6 & 8. One caution here is that Huawei subsystem pricing was given in CNY which we converted into USD for comparison purposes at CNY ~6.8 to the $. Nonetheless, they compare well in price performance as SPC now calls it.

In addition, it has been brought to my attention that the SPC-1/E benchmark runs are equivalent to the SPC-1 runs. As such, we have added Xiotech’s recent SPC-1/E run for their Emprise 5000 with 146GB Huricane drives which now comes in at #7.

(SCISPC10527-003) (c) 2010 Silverton Consulting, All Rights Reserved

In previous discussions we showed SPC-1 scatter plots for IOPS vs. Capacity and IOPS vs. $/GB. In this report we now return to IOPS vs. LRT. As one enhancement, we examined statistical trendlines (not shown) for this data but there does not appear to be any with a high correlation, so have left them out.

The two new Fujitsu systems, the DX440 and DX8400 show up well at the ~1.5msec line with ~100K and ~170K IOPS respectively and the medium cost subsystem at ~1msec LRT with ~33K IOPS is the new IBM DS8700 with Easy Tier, SSD and SATA disks.

(SCISPC10527-004) (c) 2010 Silverton Consulting, All Rights Reserved

We have a new winner for this chart as the low-end Huawei Symantec S2600 subsystem managed to crack 350 IOPS™ per drive. Their higher-end subsystem was a respectable 258 IOPS™ per drive but didn’t break into this top ten. Unclear why the low-end Oceanspace did so well, it was just using 48-146GB 15Krpm SAS drives in a RAID 1 configuration.

However we have also updated this chart to include SPC-1/E data and now show the Xiotech’s 146GB and 600GB drive SPC-1/E runs come in at #2 and #5. Not bad for a 600GB disk drive.

You will recall that we have excluded pure SSD subsystems from this analysis as they tend to be off the chart, literally. Not sure whether Easy Tier should or should not be ok here but as it came in at the middle of the pack, we felt including was justifiable.

(SCISPC10527-005) (c) 2010 IBM, All Rights Reserved

As discussed above IBM was using SSD and SATA disks along with new automated storage tiering called Easy Tier. The effect of Easy Tier& is to move “hot” extent data from SATA disk to SSD. Hot is determined by subsystem activity monitoring over some time period.

One can see in this SPC-1 generated graph that I/O activity began ~15K IO/s and peaked out at ~50K IO/s before the SPC-1 driver dropped down to the requested workload (~33K IO/s). The requested workload amount is chosen at the discretion of the vendor running the test but from our perspective it looks like it could have sustained 50K IO/s for the rest of the 24hr run. The other thing of interest is the absolute lack of variability in the IO/s for the remainder of the run which probably says something about SPC-1’s working set size.

Other SPC results

(SCISPC10527-006) (c) 2010 Silverton Consulting, All Rights Reserved

Another new SPC submission was for its SPC-1C, which as you may recall is a component level benchmark. Oracle submitted a Sun F5100 Flash drive storage system which blew out all the other components by ~70X. Please note that this chart shows 100% IOPS load on a logarithmic scale and without this one could barely see the other results. The F5100 was SAS connected to a SPARC server running the benchmark. In fact with over 300K IOPS™ the Oracle component storage would easily have qualified in the top 3 IOPS results for the normal SPC-1 if only it supported FC attachment. Seems to be a “screamer” SAS-DAS storage.

Significance

There has been only one new SPC-2 submission this past quarter for the Fujitsu ETERNUS DX80 but it did not place in the top 10 in MBPS and so, we do not show any results for SPC-2. As for the other SPC benchmarks, there have been no results this past quarter.

It seems like SPC might need to come up with a “pure SAS-SSD” benchmark. I would think that there are other SAS-Flash storage vendors who might want to take on the Oracle F5100 juggernaut in a separate competition.

Also as automated storage tiering (like Easy Tier) goes mainstream it’s unclear how benchmarks should change to take advantage of these capabilities and how to better report on such capabilities. The vast majority of current SPC-1 submissions only use one drive type. Having multiple drive types and tiering automation certainly confounds any accurate performance comparisons.

This performance dispatch was sent out to our newsletter subscribers in May of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

2010 May 27 SCI's latest analysis of SPC results (PDF 1.2 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

* All results from www.storageperformance.org as of 27 May 2010

# See http://silvertonconsulting.com/cms1/dispatches/

& From http://www.storageperformance.org/benchmark_results_files/SPC-1/IBM/A00092_IBM_DS8700_EasyTier-SSDs/a00092_IBM_DS8700_EasyTier-SSDs_SPC1_full-disclosure.pdf

SCI 2009 Nov 19 latest SPC benchmark results analysis

Nov 192009

We once again return to the classic block storage benchmark, the latest Storage Performance Council (SPC) results*. Also we report on the new SPC-1/E energy usage benchmark for the first time.

SPC-1*results

There have been four new SPC-1 results this past quarter, IBM Power 595 with SSDs, TMS RamSan-620 with SSDs, Sun Storage 6180 and Fujitsu ETERNUS DX80 (rebadged 8000) storage subsystems. Both SSD subsystems made it into the top 10 on a number of charts. Let’s start with IOPS™.

(SCISPC091119-001) (c) 2009 Silverton Consulting, All Rights Reserved

The top IOPS storage subsystem now stands as an IBM Power 595 server using SSDs and comes in at ~301K IOPS. TMS’s RamSan-620 at number 5, hit almost 255K IOPS. All the remaining, non-SSD, top-10 IOPS results save one (IBM SVC 3.1), had over 1000 drives. In contrast, the TMS RamSan-620 used only 20 SSDs. Not sure what the IBM Power 595 is doing in a storage subsystem benchmark but for SAS attached SSD storage, it’s a screamer.

(SCISPC091119-002) (c) 2009 Silverton Consulting, All Rights Reserved

Both the RamSan-620 and the IBM Power 595 tied for 3^rd at 0.5 msec LRT. All the rotating disk results range from 0.9 to 1.7 msec LRT. It’s almost inconceivable that the TMS RamSan 400 hit a 0.1 msec LRT and its counterpart, the TMS RamSan-320 hit only 0.2 msec but both have been reported before. What’s somewhat surprising is that the FC attached SSDs (TMS) and the SAS attached SSDs (IBM) perform equally well in LRT results, probably indicating that LRT performance does not always depend on drive interface.

(SCISPC091119-003) (c) 2009 Silverton Consulting, All Rights Reserved

Recall that in our last report we now restrict the IOPS/drive chart to only those subsystems using 140GB drives or larger. To that we now must add that results for SSDs are also excluded along with other memory subsystems. We would need a log scale to include the latest SSD results here, as the TMS RamSan-620 hit over 12.7K IOPS/Drive and the IBM Power 595 hit over 3.5K IOPS/Drive.

In contrast, both the Sun 6180 and Fujitsu’s DX80 made it into the top 10 for IOPS/drive at 326 and 300 IOPS/drive respectively. Also the Sun 6180 and IBM’s DS5020 Express perform exactly alike and seem to represent almost the same storage subsystem (OEMed probably from LSI, see also SPC-2 results below).

(SCISPC091119-004) (c) 2009 Silverton Consulting, All Rights Reserved

Readers will recall that this chart used to show a minimum LRT of 1.5 msec. With the addition of the TMS RamSan-620 we have had to rescale the chart below 1.5 msec. The other two additions to this chart were the Sun 6180 and the Fujitsu DX80, although they are difficult to discern in the crowd around 25K IOPs and 2.0 msec LRT. This chart always seems to tell us that subsystem price is not the lone factor in determining SPC-1 performance.

SPC-1/E

There have been no new SPC-1C or SPC-1C/E benchmarks this last quarter but a new benchmark has been released for subsystem wide energy use, the SPC-1/E. Xiotech has released results for their Emprise 5000 system with both 146GB/15Krpm and 600GB/10Krpm drives.

(SCISPC091119-005) (c) 2009 Silverton Consulting, All Rights Reserved

Xiotech used the same number of drives in each case (20), probably why the 600GB drive subsystem cost so much more (note bubble size). But more significant is that even with over 4 times the storage capacity the subsystem running the newer drives operates at ~26% less power. In all honesty the new 600GB drives operate slower, at 10Krpm than the 15Krpm 146GB drives. However, peak performance dropped only 14% from 6962 to 6057 IOPS and as such, seems a viable tradeoff.

SPC-2 results

There were eight new SPC-2 results submitted this last quarter, Sun 6180 and IBM DS5020 Express at RAID5 and RAID6 and Sun 6780 and IBM DS5300 with 8GFC at RAID5 and RAID6. Similar to the discussions above (see IOPS/drive), these two sets of subsystems perform exactly alike, i.e., the Sun 6180 equals the IBM DS5020 Express and the Sun 6780 equals the IBM DS5300 in performance, and so seem to be two of the same subsystems OEMed from the same vendor (probably LSI).

(SCISPC091119-006) (c) 2009 Silverton Consulting, All Rights Reserved

Actually the Top 11 are shown as the old (4GFC) Sun 6780/IBM 5300 results are tied for last place here. One can see the new (8GFC) Sun 6780/IBM DS5300 showing up at positions 5 through 7. We would have thought the 8GFC might make more of a difference with the SPC-2 throughput oriented testing but it only seemed to boost MBPS by ~17% (for RAID5). From our perspective, the sad part about this chart is that there really are only four subsystems represented here the HDS and it’s OEM, the two (LSI) OEMs, and the IBM SVC.

Significance

Power use continues to gain more interest. We again applaud SPC for providing yet another new energy benchmark. The other items of note from these results are that SSDs perform well whether FC or SAS attached and that a subsystem with only 20 SSDs (TMS RamSan-620) can easily break into the top 10 IOPS chart.

This performance dispatch was sent out to our newsletter subscribers in November of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this is available at

SCI 2009 Nov 19 Update to SPC benchmark results (PDF 938.6 KiB)

As always we welcome any feedback on how to do this better. So, if you have any comments please don’t hesitate to contact us.

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

* All results from www.storageperformance.org as of 27 August 2009

SCI 2009 Aug 27 latest SPC benchmark results analysis

Aug 272009

We once again return to the classic block storage benchmark, the latest Storage Performance Council (SPC) results*. Also we report on the new SPC-1C/E energy usage benchmark for the first time.

SPC-1*results

There have been only two new SPC-1 results this past quarter, one for IBM DS5300 with encryption and the other for IBM DS5020 Express. Neither of these made the top 10 in IOPS™, $/IOPS™ or IOPS™/$/GB so these charts can be found in prior dispatches#. However, one of these products did make the top 10 for LRT and both products made the top 10 for IOPS/drive (see below).

(SCISPC090827-001) (c) 2009 Silverton Consulting, All Rights Reserved

Actually this shows the top 12 as there is currently a three-way tie for 10^th place. Both the IBM DS5300 with and without full drive encryption (FDE) had the same LRT at 1.8msec. From my perspective, the good news is, full drive encryption seems to have a negligible (un-measurable) affect on response time. Considering also that FDE has little to no impact on IOPS performance or cost, it should easily become widely adopted in this class of subsystems.

(SCISPC090827-002) (c) 2009 Silverton Consulting, All Rights Reserved

Figure 2 Top 10 IOPS/drive results

Actually this shows the top 11 as there is a tie for 9^th place. The latest IBM DS5020 Express (channel ready product) managed to be the top subsystem here with an average IOPS per drive of 326. Also the DS5300 with encryption made it into the top 10 matching the performance of the DS5300 without encryption. As can be seen above, most products that do well here are midrange and not the high-end subsystems.

Recall that in our last report we now restrict this chart to only those subsystems using 140GB drives or larger. Also, this metric is for only those subsystems using rotating media.

(SCISPC090827-003) (c) 2009 Silverton Consulting, All Rights Reserved

We return to the IOPs versus LRT bubble chart we have shown previously. The one thing always missing from a pure performance analysis is subsystem cost, added here as bubble size. To keep this interesting we capped the subsystems shown to a $100/GB maximum to eliminate the higher priced subsystems.

From an end-user perspective it’s interesting to note that one can obtain a reasonably performing subsystem (~100K IOPS with LRT <2msec) for about 1/4^th or less the price of truly high performance subsystem. Also, similarly priced or even more expensive subsystems can have much worse performance on an IOPS and/or a LRT basis indicating that price isn’t always the best factor to predict subsystem performance.

SPC-1C/E

SPC has a new benchmark category that displays energy usage for storage subsystems. So far there have only been two results submitted, IBM System Storage EXP12S with SSDs and Seagate Savvio 10K.3 ST93000603S and show some interesting results. Both subsystems use a SAS interface with no RAID protection.

(SCISPC090827-004) (c) 2009 Silverton Consulting, All Rights Reserved

As can be inferred from the above, the big bubbles are an IBM SSD system and the small bubbles are the Seagate Savvio 10Krpm drive system. The Blue bubbles show the “nominal” energy use and IOPS that are obtained from the two systems. Yellow bubbles show similar data at “peak” IOPS.

Nominal energy use is an aggregate workload constructed by SPC to show typical energy use and IOP/second obtainable from subsystems. This data is summarized and reported directly off the SPC-1C/E report. Peak IOPs & energy use took some hunting, but represents the max IOPS and the energy consumption at that IO rate.

I suppose it’s no surprise that the rotating media would consume more power for higher IO operations per second. However, the difference in power consumption for the IBM’s SSD subsystem at nominal versus peak IO rates is almost negligible (less than 2%). We are not sure which is the more fair comparison so we have elected to show both.

We do not show idle subsystem comparison but both have similar energy consumption. As such, rotating media has a much more slanted energy profile than SSD storage.

SPC-2, SPC-1C and SPC-2C results

There were no new SPC-2, SPC-1C and SPC-2C benchmarks released for this update and as such, can be found in prior reports.

Significance

Power use is starting to take on more importance. We should all applaud SPC for instituting a new energy use benchmark. As more results show up, it should become much more interesting but having SSDs vs. a 10Krpm subsystem as starters works well.

The fact that drive encryption seems to have no impact on performance or subsystem cost seems counterintuitive. By all accounts, there should have been at least some cost differential and certainly some performance degradation. Once more my assumptions are overturned by reality. Drive encryption seems to be here and deserves higher adoption if these results can hold up.

As always we welcome any feedback on how to do this better. So, if you have any comments please don’t hesitate to contact us.

This performance dispatch was sent out to our newsletter subscribers in August of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this is available at

SCI 2009 August 27 Update to SPC performance results (PDF 657.8 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 27 August 2009

# Prior SPC performance dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2009 May 27 latest SPC benchmark results analysis

May 272009

We once again return to discuss the latest Storage Performance Council (SPC) results*.

SPC-1*results

There have been only five new SPC-1 results these past quarter, one for IBM DS5300, one for 3PAR InServ F400, and three for HDS AMS2500, AMS2300, and AMS2100. None of these made the top 10 in IOPS™ or $/IOPS™ so these charts can be found in prior dispatches#. However, two of these products did make the top 10 for IOPS/$/GB (see Figure 1 below).

(SCISPC090527-001) (c) 2009 Silverton Consulting, All Rights Reserved

The two new subsystems here are the 3PAR InServ F400 and the HDS AMS2500, both of which would be considered mid-class storage subsystems. We calculate this metric as = IOPS / ($/GB) and created this metric as another way to factor in performance against cost and capacity. Although none of these metrics can tell the complete story.

(SCISPC090527-002) (c) 2009 Silverton Consulting, All Rights Reserved

As for response time, the latest IBM DS5300 managed to break into the top 10 at an average least response time (LRT) of 1.77 msec. I have been told this metric is not as important anymore but given all the interest in SSDs these days I find that hard to reconcile.

(SCISPC090527-003) (c) 2009 Silverton Consulting, All Rights Reserved

We have redone the IO operations/second/Drive (IOPS/Drv) chart so that it now only include drives over 140GB as the smaller drives held too great an advantage here. All of the latest benchmark results show up in this Top 10 IOPS/Drv with HDS holding the number 4, 7, and 10 spots, IBM DS5300 at number 8, and 3PAR F400 at number 9. Also, as mentioned in a prior report, both the J4400 and J4200 results had no RAID protection so may not be suitable comparisons for normal customer environments.

Previously this chart had an error for the ETERNUS2000 that caused us to report double its actual IOPS/Drv rate. We have fixed that error and it now shows the correct IOPS/Drive for the ETERNUS2000.

(SCISPC090527-004) (c) 2009 Silverton Consulting, All Rights Reserved

Figure 4 SPC-1 Bubble chart of IOPS against LRT, bubble size proportional to subsystem price

Always looking for an interesting cut on SPC-1 data we came up with this new bubble chart. It shows a scatter plot view of subsystem performance with the x-axis as LRT and the y-axis as IOPS. The one thing always missing from a pure performance analysis is subsystem cost, added here as bubble size. To keep this interesting we capped the subsystems shown on the chart to a $100/GB maximum to eliminate the higher priced subsystems.

From an end-user perspective it’s interesting to note that one can obtain a reasonably performing subsystem (~100K IOPS with LRT <2msec) for about 1/4^th or less the price of truly high performance subsystem. Also, similarly priced or even more expensive subsystems can have much worse performance on an IOPS and/or a LRT basis indicating that pricing isn’t always the best factor in subsystem selection.

SPC-2, SPC-1C and SPC-2C results

There were no new SPC-2, SPC-1C and SPC-2C benchmarks released for this update and as such we stand with our last report SPC StorInt Dispatch&.

Significance

As we show above midrange subsystems can be high performers when configured properly. Although higher end subsystems dominate the Top 10 IOPS chart, the midrange subsystems discussed here have all managed to do well on most of the other performance metrics.

Over the years we have tried to come up with ways to compare performance to price and have used both $/IOPS and IOPS/$/GB (and others on occasion) as attempts to further this analysis. In the end we find that both of these metrics, although interesting on their own, leaves something out. The new bubble chart (see Figure 4 above) is our latest attempt to incorporate pricing information with subsystem performance. Hopefully, the reader will view this as a worthy addition to our SPC analysis. As always we welcome your feedback on how to do this better.

This performance dispatch was sent out to our newsletter subscribers in May of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 May 27 Update to SPC benchmark results (PDF 616.1 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 23 February 2009

# Prior SPC performance dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

& Available at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2009 Feb 24 latest SPC benchmark results analysis

Feb 242009

We once again return to discuss the latest Storage Performance Council (SPC) results*. Also, SPC has recently introduced new versions of its benchmarks to measure storage component performance. We report on these new results below.

SPC-1*results

There have been only two new SPC-1 results these past three months one for Pillar Data Axiom 600 and the other for Sun Storage 6780. Neither of these two made the top 10 in IOPS™, LRT™, or $/IOPS™ so these charts can be found in prior dispatches#. However, both of these products did make the top 10 for IOPS/$/GB (see Figure 1 below)

(SCISPC090224-001) (c) 2009 Silverton Consulting, All Rights Reserved

The big two players here from last report are 3PAR and SVC 4.3. Again, we calculate this metric as = IOPS / ($/GB) and created this metric as another way to factor in performance against cost and capacity.

On the other hand, as has been pointed out to SCI, this metric may unfairly advantage big, monolithic subsystems at the expense of smaller subsystems. The monolithic subsystems generate such high IOPS counts that they’re relatively expensive $/GB doesn’t impact their ranking on this chart. In contrast, smaller

subsystems, such as Xiotech’s Emprise may be capable of putting up high IOPS rates by aggregating a number of smaller subsystems but in their current instantiation, their relatively modest $/GB doesn’t compensate for the resultant IOPS and hence they cannot compete on this chart. In such a configuration, even when taking additional switch port costs into account, may still be significantly less costly than the systems shown here.

Again it would seem the FAS3170, Sun Storage 6780 and Pillar Axiom 600 look out of place here with these multi-million $ subsystems (3PAR, IBM SVC 4.3&4.2, Sun T9990V, HDS USP-V, and HP XP2400) but seems to provide relatively good performance for its price and capacity. Also, once again the Sun J4400 has no RAID protection whatsoever and probably does not belong here.

SPC-2 results

There were also two new SPC-2 benchmarks recorded for this update, both from Sun, one the SUN 6780 with RAID5 and the other the same system with RAID6. Both these systems are new entries in the top 10 MBPS™

(SCISPC090224-002) (c) 2009 Silverton Consulting, All Rights Reserved

There was a slight degradation in performance for the RAID6 over the RAID5 version of the Sun Storage 6780 of roughly 3%. All the other systems were reported on in prior SPC performance dispatches.

(SCISPC090224-003) (c) 2009 Silverton Consulting, All Rights Reserved

With this report SCI is introducing a new way to show subsystem performance for each of the three workloads that constitute the composite MPBS™ score. Here one can see some variability in the scale of workloads each vendor’s product can attain. There seems to be some interesting difference between the LDQ (large database query) workloads and the other two LFP (large file processing) and VOD (video on demand).

It’s unclear why the HP XP24000, HDS USP-V and the SVC 4.2 do so well on the LDQ workload as compared to the other two workloads. SCI believes the secret lies somewhere in their respective caching algorithms being optimized for LDQ and not as well optimized for the other two workloads. This may say that the other products such as the Fujitsu ETERNUS8000, Sun Storage 6780 and IBM DS5300 could improve their LDQ performance if they tweaked their caching algorithms somewhat.

New SPC-1C and -2C Results

SPC has been busy and have created new versions of their SPC-1 and SPC-2 benchmarks specifically to measure storage component performance. Currently there are not many released results. Also, SCI is having some difficulty trying to compare a 24 drive RAID subsystem against a single drive SATA system but other than that we believe the results are worthy of elaboration.

SPC-1C Results

SPC-1 results have always reported details for 10%, 50%, 80%, 90%, 95%, and 100% load levels with IOPS counts and response times in msec. 100% is based on exceeding a pre-determined response time cap, which SCI believes to be greater than 30msec.

(SCISPC090224-004) (c) 2009 Silverton Consulting, All Rights Reserved

For the single drive results instead of reporting IOPS™ which is at the 100% level SCI decided to use the 50% load level and plot error bars up to 80% load and down to 10%. At the 100% load point many of these single drives are producing response times just shy of the 30msec cut off. Unclear whether anyone would run these drives at that slow a response time. Thus, SCI decided to show the 50% load level with error bars. Somewhere around 80% load, the drive busy-ness takes over and how it manages its seek queue drives performance rather than seek speed.

All of these devices are running 7200RPM and have 1TB of storage. The first drive listed uses a SAS interface, all the others use SATA. LSI SAS3041X-R HBAs were used for all these runs. A couple of caveats here

Seagate sponsored all of these results. But the results as reported were from benchmark runs executed at the SPC lab with sponsored supplied hardware.
There are three other SPC-1C results, for 12, 24, and 15 drive RAID systems that don’t seem comparable to these results so they have been left out of this analysis.

(SCISPC090224-005) (c) 2009 Silverton Consulting, All Rights Reserved

Many end users are more interested in what a storage subsystem could do at a constant 10msec response time. As such, we introduce another new chart for the SPC-1C that depicts the expected IOPS at an average response time to 10msec. For most drives, the IOPS shown in the chart is an interpolated value. However for the Samsung Spinpoint F1 HE103UJ the value shown represents what they actually achieved at the 100% workload value.

For some reason both the Samsung devices seem to do well at 10msec response time while the others fall off. This could possibly be due to just different seek optimization profiles targeted at varying workload levels. The 50% workload level is typically closer to the 10msec mark for all the other drives. For the other Samsung drive, the 10msec response time is closer to its 95% load.

SPC-2C results

(SCISPC090224-006) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, for the new SPC-2C results we show the various workloads in a radar chart format. Here we can obviously see the slowness of the VOD workload vis-à-vis the other workloads. Also there appears to be something amiss with the HDS Ultrastar drive during the LFP workload and the Samsung non-RAID class drive during the VOD workload. Possibly another artifact of varying seek optimization profiles.

Aside from the fact that these are 7200RPM drives, SCI would have assumed this chart would have looked closer to the subsystem SPC-2 chart shown earlier. Obviously, subsystem caching and data striping has an impact and can radically alter subsystem performance in comparison to drive performance on the same workloads. However, the difference between all these drives for the LDQ vs. VOD workloads are significantly different than what was shown on the earlier chart. For example, a maximum of 30% difference in performance between LDQ and VOD workloads on the subsystem chart vs. a minimum of almost 50% difference in performance for the current chart seems disquieting and worthy of more research.

The same caveats apply to SPC-2C that applied to SPC-1C results (see above).

Significance

SCI is always interested in understanding subsystem performance. The new SPC-1C and -2C benchmark results open up a new way to judge storage components rather than storage subsystems. Seagate has taken a shot at 7200RPM drives. It would be even more interesting to show 10K and 15KRPM drives, as well as HBA’s.

As much as we like the new SPC-1C and -2C benchmarks, some method needs to be implemented to insure comparable benchmarks are run. It’s unclear to SCI how to compare a 12, 15, and 24 drive storage subsystem against a single drive subsystem other than on a per drive basis. Of course this problem also occurs with the SPC-1 and SPC-2 benchmarks as well but at least there we are talking about complete storage subsystems rather than storage components. Perhaps as more multi-drive SPC-1C and -2C results are released we can use the per drive basis as a comparison.

Finally, SPC has once again allowed a sponsor to supply results for other vendor products (Seagate for SPC-1C and -2C results). SCI believes that while this may be expedient, it may hurt the long run value of SPC benchmarks. Because some non-sponsoring vendors feel slighted when not getting an opportunity to review, submit their own benchmark results, and/or reject the results entirely. Unclear how to solve this dilemma, given SPC’s stated policy but perhaps allowing the non-SPC member vendors to review and dispute benchmark results ahead of publication may be a necessary first step.

This performance dispatch was sent out to our newsletter subscribers in Febuary of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 February 24 Update to SPC performance results (PDF 823.1 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 23 February 2009

# Prior SPC performance dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2008 Aug 25 latest SPC-1 and SPC-2 performance results analysis

Aug 252008

We once again return to our quarterly SPC results and as such we report on the latest benchmark results below.

SPC-1*results

(SCISPC080825-001) (c) 2008 Silverton Consulting, All Rights Reserved

There have been only a few new SPC-1 results these past three months. Mostly Sun has released results for their J4200 and J4400 low-end storage systems and NetApp has released results for FAS3170. None of these benchmark results impacted the top IOPS or LRT results so these unchanged results can be found in the appendix.

However, the new benchmarks did impact some of the other charts SCI maintains for SPC-1 data. We start with a standard SPC-1 metric $/IOPS™. As can be seen below the new Sun products (J4200 and J4400) show up as number 3 and 4 in the top 10. One critical caveat is that both the J4200 and J4400 have no RAID protection and as such have very cheap but unprotected storage. However, RAID protection would not necessarily impact this chart as its mainly storage cost divided by maximum attainable IOPs but cheaper storage does have a significant bearing on this metric.

Next we turn to our own metric IOPS/drive, which shows both the new Sun J4400 and the FAS3170 results (see Figure 2 below) in the top 10. As has been pointed out recently to SCI the Fujitsu ETERNUS3000-100 results benefited from short-stroking their disks (roughly 50%), and as such their results although technically correct are not comparable to the other results which don’t short stroke.

(SCISPC080825-002) (c) 2008 Silverton Consulting, All Rights Reserved

As for new results, again the unprotected storage in the J4400 has an advantage when it comes to this chart as they use less drives to configure their storage as SAS drives. But one real surprise is the new FAS3170 result at number 10 that uses FC/AL to support the 224 drives in their configuration. We are also surprised the J4200 didn’t make the top 10 as it was also configured without RAID protected storage but it just missed the cut at number 11.

(SCISPC080825-003) (c) 2008 Silverton Consulting, All Rights Reserved

Finally, we turn to our IOPS™/$/GB metric. We calculate this metric as = IOPS / ($/GB). We created this metric as another way to factor in performance against cost and capacity. Once again the Sun J4400 has no RAID protection so its very cheap storage for its performance. But the real surprise here is the FAS3170, which takes the second slot and is the only non-storage virtualization product in the top 5.

This metric deserves some discussion. We have often felt that $/IOPS (or IOPS/$) was somewhat fabricated because subsystem cost is essentially unverifiable. Some vendors supply pricing data that is independently audited and verified. SCI is not aware whether this is standard SPC practice or whether this is done at vendor discretion. As such we are concerned that the pricing data may not be entirely accurate.

Also another metric we once considered was IOPS/GB but that rewards storage using smaller drives and some of the oldest benchmarks would have an un-natural advantage here. As such we began to use IOPS/$/GB. It’s perhaps not as easy to understand as $/IOPS or IOPS/GB but it provides sort of a rough performance metric against the price of storage. This metric is harder to game as it factors the baseline storage price ($/GB) against storage performance.

The chart would say that the FAS3170 has relatively good performance for the price of its storage and the same would go for Sun’s J4400 but for the fact its unprotected storage and no one would buy unprotected storage for real data.

SPC-2 results

There are only two new SPC-2 benchmarks recorded for this update, both from Sun, the J4400 and J4200 arrays. We rarely discuss non-MPBS™ SPC-2 metrics because of the high correlation between MPBS™ and these other metrics but the J4400 results has forced a reexamination of this practice. For example, although the J4400 would rank number #13 on the top MPBS performance, it ranks #11 on LFP (Large File Processing) and #12 on LDQ (Large Database Query and #29 on VOD (Video On Demand) workloads. This is very unusual because until the J4200 and J4400 results no other subsystem has had such different rankings from the aggregated MPBS data. It almost seems like a problem in the benchmark run or a serious sub-optimization in the storage subsystems.

(SCISPC080825-004) (c) 2008 Silverton Consulting, All Rights Reserved

As for new SPC-2 MBPS™ results (see figure 5), there are no changes to the top 10 MPBS list and as such we have relegated these unchanged results to Appendix A. However, akin to the SPC-1 results above, we have analyzed the MBPS results from a drive perspective and present our analysis of the Top 10 MBPS normalized on a per drive basis (See Figure 4) below.

Sun’s J4200 and J4400 using 1TB SATA drives have pretty good throughput. It is somewhat surprising that 7200 RPM SATA drives could do so well in this test but as it’s essentially big block sequential once you are past the seek time, spinning data off the disk sequentially can be done quickly, even with slow drives.

Another place where the Sun J4200 and J4400 do well is in the $/MBPS™ results (see Figure 5 above) once again both of these results are for unprotected storage and 1TB SATA drives don’t have the best reliability in the world. It’s unlikely that anyone would use this storage for any data that they want to keep around for long, but it’s undeniably cheap storage.

This performance dispatch was sent out to our newsletter subscribers in August of 2008. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2008 August 25 New SPC performance results analysis (PDF 1.7 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

Appendix A Unchanged SPC-1&2 Results

(SCISPC080825-006) (c) 2008 Silverton Consulting, All Rights Reserved

Top 10 IOPS™ results (see Figure 6) have not changed since last time and continue to be dominated by TMS RamSan400, IBM SVC4.2 and OEM and native versions of HDS USP-V.

(SCISPC080825-007) (c) 2008 Silverton Consulting, All Rights Reserved

Top 10 SPC-1 LRT results have not changed from last time (see Figure 7) and are dominated by the showings of TMS RamSan and IBM DS8300.

Top 10 SPC-2 MBPS™ results (see Figure also did not change for this update and are dominated by IBM SVC, Fujitsu ETERNUS 8000 and IBM DS8300 (See Figure subsystems

(SCISPC080825-008) (c) 2008 Silverton Consulting, All Rights Reserved

* All results from www.storageperformance.org as of 25 August 2008

Older Entries

SPC-1*results

SPC-2 Results

Significance

SPC-1*results

SPC-2 Results

Significance

SPC-1*results

Significance

SPC-1*results

Significance

SPC-1*results

Significance

SPC-1*results

SPC-1/E

SPC-2 results

Significance

SPC-1*results

SPC-1C/E

SPC-2, SPC-1C and SPC-2C results

Significance

SPC-1*results

SPC-2, SPC-1C and SPC-2C results

SPC-1*results

SPC-2 results

New SPC-1C and -2C Results

SPC-1C Results

SPC-2C results

Significance

SPC-1*results

NEWS and EVENTS

TOP DOWNLOADS

TWITTER FEED...

POPULAR POSTS

NEWSLETTER

TRANSLATE THIS PAGE

Dispatch Tag Cloud

Dispatch categories

Dispatch archives