We once again return to our quarterly SPC results and as such we report on the latest benchmark results below.
There have been only a few new SPC-1 results these past three months. Mostly Sun has released results for their J4200 and J4400 low-end storage systems and NetApp has released results for FAS3170. None of these benchmark results impacted the top IOPS or LRT results so these unchanged results can be found in the appendix.
However, the new benchmarks did impact some of the other charts SCI maintains for SPC-1 data. We start with a standard SPC-1 metric $/IOPS™. As can be seen below the new Sun products (J4200 and J4400) show up as number 3 and 4 in the top 10. One critical caveat is that both the J4200 and J4400 have no RAID protection and as such have very cheap but unprotected storage. However, RAID protection would not necessarily impact this chart as its mainly storage cost divided by maximum attainable IOPs but cheaper storage does have a significant bearing on this metric.
Next we turn to our own metric IOPS/drive, which shows both the new Sun J4400 and the FAS3170 results (see Figure 2 below) in the top 10. As has been pointed out recently to SCI the Fujitsu ETERNUS3000-100 results benefited from short-stroking their disks (roughly 50%), and as such their results although technically correct are not comparable to the other results which don’t short stroke.
As for new results, again the unprotected storage in the J4400 has an advantage when it comes to this chart as they use less drives to configure their storage as SAS drives. But one real surprise is the new FAS3170 result at number 10 that uses FC/AL to support the 224 drives in their configuration. We are also surprised the J4200 didn’t make the top 10 as it was also configured without RAID protected storage but it just missed the cut at number 11.
Finally, we turn to our IOPS™/$/GB metric. We calculate this metric as = IOPS / ($/GB). We created this metric as another way to factor in performance against cost and capacity. Once again the Sun J4400 has no RAID protection so its very cheap storage for its performance. But the real surprise here is the FAS3170, which takes the second slot and is the only non-storage virtualization product in the top 5.
This metric deserves some discussion. We have often felt that $/IOPS (or IOPS/$) was somewhat fabricated because subsystem cost is essentially unverifiable. Some vendors supply pricing data that is independently audited and verified. SCI is not aware whether this is standard SPC practice or whether this is done at vendor discretion. As such we are concerned that the pricing data may not be entirely accurate.
Also another metric we once considered was IOPS/GB but that rewards storage using smaller drives and some of the oldest benchmarks would have an un-natural advantage here. As such we began to use IOPS/$/GB. It’s perhaps not as easy to understand as $/IOPS or IOPS/GB but it provides sort of a rough performance metric against the price of storage. This metric is harder to game as it factors the baseline storage price ($/GB) against storage performance.
The chart would say that the FAS3170 has relatively good performance for the price of its storage and the same would go for Sun’s J4400 but for the fact its unprotected storage and no one would buy unprotected storage for real data.
There are only two new SPC-2 benchmarks recorded for this update, both from Sun, the J4400 and J4200 arrays. We rarely discuss non-MPBS™ SPC-2 metrics because of the high correlation between MPBS™ and these other metrics but the J4400 results has forced a reexamination of this practice. For example, although the J4400 would rank number #13 on the top MPBS performance, it ranks #11 on LFP (Large File Processing) and #12 on LDQ (Large Database Query and #29 on VOD (Video On Demand) workloads. This is very unusual because until the J4200 and J4400 results no other subsystem has had such different rankings from the aggregated MPBS data. It almost seems like a problem in the benchmark run or a serious sub-optimization in the storage subsystems.
As for new SPC-2 MBPS™ results (see figure 5), there are no changes to the top 10 MPBS list and as such we have relegated these unchanged results to Appendix A. However, akin to the SPC-1 results above, we have analyzed the MBPS results from a drive perspective and present our analysis of the Top 10 MBPS normalized on a per drive basis (See Figure 4) below.
Sun’s J4200 and J4400 using 1TB SATA drives have pretty good throughput. It is somewhat surprising that 7200 RPM SATA drives could do so well in this test but as it’s essentially big block sequential once you are past the seek time, spinning data off the disk sequentially can be done quickly, even with slow drives.
Another place where the Sun J4200 and J4400 do well is in the $/MBPS™ results (see Figure 5 above) once again both of these results are for unprotected storage and 1TB SATA drives don’t have the best reliability in the world. It’s unlikely that anyone would use this storage for any data that they want to keep around for long, but it’s undeniably cheap storage.
This performance dispatch was sent out to our newsletter subscribers in August of 2008. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.
A PDF version of this can be found atSCI 2008 August 25 New SPC performance results analysis
Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community
Appendix A Unchanged SPC-1&2 Results
Top 10 IOPS™ results (see Figure 6) have not changed since last time and continue to be dominated by TMS RamSan400, IBM SVC4.2 and OEM and native versions of HDS USP-V.
Top 10 SPC-1 LRT results have not changed from last time (see Figure 7) and are dominated by the showings of TMS RamSan and IBM DS8300.
Top 10 SPC-2 MBPS™ results (see Figure 8) also did not change for this update and are dominated by IBM SVC, Fujitsu ETERNUS 8000 and IBM DS8300 (See Figure 8) subsystems