Latest SPC-1 results – IOPS vs drive counts – chart-of-the-month

Scatter plot of SPC-1  IOPS against Spindle count, with linear regression line showing Y=186.18X + 10227 with R**2=0.96064
(SCISPC111122-004) (c) 2011 Silverton Consulting, All Rights Reserved

[As promised, I am trying to get up-to-date on my performance charts from our monthly newsletters. This one brings us current up through November.]

The above chart plots Storage Performance Council SPC-1 IOPS against spindle count.  On this chart, we have eliminated any SSD systems, systems with drives smaller than 140 GB and any systems with multiple drive sizes.

Alas, the regression coefficient (R**2) of 0.96 tells us that SPC-1 IOPS performance is mainly driven by drive count.  But what’s more interesting here is that as drive counts get higher than say 1000, the variance surrounding the linear regression line widens – implying that system sophistication starts to matter more.

Processing power matters

For instance, if you look at the three systems centered around 2000 drives, they are (from lowest to highest IOPS) 4-node IBM SVC 5.1, 6-node IBM SVC 5.1 and an 8-node HP 3PAR V800 storage system.  This tells us that the more processing (nodes) you throw at an IOPS workload given similar spindle counts, the more efficient it can be.

System sophistication can matter too

The other interesting facet on this chart comes from examining the three systems centered around 250K IOPS that span from ~1150 to ~1500 drives.

  • The 1156 drive system is the latest HDS VSP 8-VSD (virtual storage directors, or processing nodes) running with dynamically (thinly) provisioned volumes – which is the first and only SPC-1 submission using thin provisioning.
  • The 1280 drive system is a (now HP) 3PAR T800 8-node system.
  • The 1536 drive system is an IBM SVC 4.3 8-node storage system.

One would think that thin provisioning would degrade storage performance and maybe it did but without a non-dynamically provisioned HDS VSP benchmark to compare against, it’s hard to tell.  However, the fact that the HDS-VSP performed as well as the other systems did with much lower drive counts seems to tell us that thin provisioning potentially uses hard drives more efficiently than fat provisioning, the 8-VSD HDS VSP is more effective than an 8-node IBM SVC 4.3 and an 8-node (HP) 3PAR T800 systems, or perhaps some combination of these.

~~~~

The full SPC performance report went out to our newsletter subscriber’s last November.  [The one change to this chart from the full report is the date in the chart’s title was wrong and is fixed here].  A copy of the full report will be up on the dispatches page of our website sometime this month (if all goes well). However, you can get performance information now and subscribe to future newsletters to receive these reports even earlier by just sending us an email or using the signup form above right.

For a more extensive discussion of block or SAN storage performance covering SPC-1&-2 (top 30) and ESRP (top 20) results please consider purchasing our recently updated SAN Storage Buying Guide available on our website.

As always, we welcome any suggestions on how to improve our analysis of SPC results or any of our other storage system performance discussions.

Comments?

ESRP v3 (Exchange 2010) log playback results, 1Kmbox&under – chart-of-the-month

(SCIESRP111029-003) (c) 2011 Silverton Consulting, All Rights Reserved
(SCIESRP111029-003) (c) 2011 Silverton Consulting, All Rights Reserved

The above chart is from our last Exchange [2010] Solution Review Program (ESRP) performance dispatch released in our October newsletter (sign-up upper right).  The 1K mailbox and under category for ESRP represents Exchange storage solutions for SMB data centers.

As one can see from the above the NetApp FAS2040 has done well but an almost matching result came in from the HP P2000 G3 MSA system.  What’s not obvious here is that the FAS2040 had 8 disks and the P2000 had 78 so there was quite a difference in the spindle counts. The #3&4 runs from EMC VNXe3100 also posted respectable results (within 1sec of top performer) and only had 5 and 7 disks respectively, so they were much more inline with the FAS2040 run.  The median number of drives for this category is 8 drives which probably makes sense for SMB storage solutions.

Why log playback

I have come to prefer a few metrics in the Exchange 2010 arena that seem to me to capture a larger part of the information available from an ESRP report.  The Log Playback metric is one of them that seems to me to fit the bill nicely.  Specifically:

  • It doesn’t depend on the Jetstress IO/rate parameter that impacts the database transfers per second rate.  The log playback is just the average time it takes to playback a 1MB log file against a database.
  • It is probably a clear indicator of how well a storage system (configured matching the ESRP) can support DAG log processing.

In addition, I believe Log Playback is a great stand-in for any randomized database transaction processing. Now I know that Exchange is not necessarily a pure relational database but it does have a significant component of indexes, tables, and sequentiality to it.

My problem is that there doesn’t seem to be any other real database performance benchmark out there for storage.  I know that TPC has a number of benchmarks tailored to database transaction activity but these seem to be more a measure of the database server than the storage.  SPC-2 has some database oriented queries but it’s generally focused on through put and doesn’t really represent randomized database activity and for other reasons it’s not as highly used as SPC-1 or ESRP so there is not as much data to report on.

That leaves ESRP.  For whatever reason (probably the popularity of Exchange), almost everyone submits for ESRP. Which makes it ripe for product comparisons.

Also, there are a number of other good metrics in ESRP results that I feel have general applicability outside Exchange as well.  I will be reporting on them in future posts.

~~~~

Comments?

Sorry, I haven’t been keeping up with our chart-of-the-month posts, but I promise to do better in the future.  I plan to be back in synch with our newsletter dispatches before month end.

The full ESRP performance report for the 1K and under mailbox category went out to our newsletter subscriber’s last October.  A copy of the full report will be up on the dispatches page of our website sometime this month (if all goes well). However, you can get performance information now and subscribe to future newsletters to receive these reports even earlier by just sending us an email or using the signup form above right.

For a more extensive discussion of block storage performance in ESRP (top 20) and SPC-1&-2 (top 30) results please consider purchasing our recently updated SAN Storage Buying Guide available on our website.

As always, we welcome any suggestions on how to improve our analysis of ESRP results or any of our other storage system performance discussions.

 

Latest SPECsfs2008 benchmarks analysis, CIFS vs NFS corrected – chart of the month

(SCISFS110929-001) (c) 2011 Silverton Consulting, All Rights Reserved
(SCISFS110929-001) (c) 2011 Silverton Consulting, All Rights Reserved

We made a mistake in our last post discussing CIFS vs. NFS results using SPECsfs2008 benchmarks by including some storage systems that had SSDs in this analysis. All of our other per spindle/disk drive analyses exclude SSDs and NAND cache because they skew per drive results so much.  We have corrected this in the above chart which includes all the SPECsfs2008 results, up to the end of last month.

However, even with the corrections the results stand pretty much the way they were. CIFS is doing more throughput per disk drive spindle than NFS for all benchmark results not using SSDs or Flash Cache.

Dropping SSD results changed the linear regression equation. Specificall,  the R**2 for CIFS and NFS dropped from 0.99 to 0.98 and from 0.92 to 0.82 and the B coefficient dropped from 463 to 405 and from 296 to 258 respectively.

I would be remiss if I didn’t discuss a few caveats with this analysis.

  • Now there are even less results in both CIFS and NFS groups, down to 15 for CIFS and 38 for NFS.   For any sort of correlation comparison, more results would have better statistical significance.
  • In the NFS data, we include some NAS systems which have lots of DRAM cache (almost ~0.5TB).  We should probably exclude these as well, which might drop the NFS line down some more (at least lower the B value).
  • There are not a lot of enterprise level CIFS systems in current SPECsfs resuslts, with or without SSD or NAND caching.  Most CIFS benchmarks are from midrange or lower filers.  Unclear why these would do much better on a per spindle basis than a wider sample of NFS systems, but they obviously do.

All that aside, it seems crystal clear here, that CIFS provides more throughput per spindle.

In contrast, we have shown in the past posts how for the limited number of systems that submitted benchmarks with both CIFS and NFS typically show roughly equivalent throughput results for CIFS and NFS. (See my other previous post on this aspect of the CIFS vs. NFS discussion).

Also, in our last post we discussed some of the criticism leveled against this analysis and provided our view to refute these issues. Mostly their concerns are due to the major differences between CIFS state-full protocol and NFS stateless protocol.

But from my perspective it’s all about the data.  How quickly can I read a file, how fast can I create a file.  Given similar storage systems, with similar SW, cache and hard disk drives, it’s now clear to me that CIFS provides faster access to data than NFS does, at least on a per spindle basis.

Nevertheless, more data may invalidate these results, so stay tuned.

—-

Why this is should probably be subject for another post but it may have a lot to do with the fact that it is stateless….

Comments?

SCI’s latest SPC-2 performance results analysis – chart-of-the-month

SCISPC110822-002 (c) 2011 Silverton Consulting, All Rights Reserved
SCISPC110822-002 (c) 2011 Silverton Consulting, All Rights Reserved

There really wasn’t that many new submissions for the Storage Performance Council SPC-1 or SPC-2 benchmarks this past quarter (just the new Fujitsu DX80S2 SPC-2 run) so we thought it time to roll out a new chart.

The chart above shows a scatter plot of the number of disk drives in a submission vs. the MB/sec attained for the Large Database Query (LDQ) component of an SPC-2 benchmark.

As one who follows this blog and our twitter feed knows we continue to have an ongoing, long running discussion on how I/O benchmarks such as this are mostly just a measure of how much hardware (disks and controllers) are thrown at them.  We added a linear regression line to the above chart to evaluate the validity of that claim and as clearly shown above, disk drive count is NOT highly correlated with SPC-2 performance.

We necessarily exclude from this analysis any system results that used NAND based caching or SSD devices so as to focus specifically on disk drive count relevance.   There are not a lot of these in SPC-2 results but there are enough to make this look even worse.

We chose to only display the LDQ segment of the SPC-2 benchmark because it has the best correlation or highest R**2 at 0.41 between workload and disk count. The aggregate MBPS as well as the other components of the SPC-2 benchmark include video on demand (VOD) and large file processing (LFP) both of which had R**2’s of less than 0.36.

For instance, just look at the vertical centered around 775 disk drives.  There are two systems that show up here, one doing ~ 6000 MBPS and the other doing ~11,500 MBPS – quite a difference.  The fact that these are two different storage architectures from the same vendor is even more informative??

Why is the overall correlation so poor?

One can only speculate but there must be something about system sophistication at work in SPC-2 results.  It’s probably tied to better caching, better data layout on disk, and better IO latency but it’s only an educated guess.  For example,

  • Most of the SPC-2 workload is sequential in nature.  How a storage system detects sequentiality in a seemingly random IO mix is an art form and what a system does armed with that knowledge is probably more of a science.
  • In the old days of big, expensive CKD DASD, sequential data was all laid out in consecutively (barring lacing) around a track and up a cylinder.  These days of zoned FBA disks one can only hope that sequential data resides in laced sectors, along consecutive tracks on the media, minimizing any head seek activity.  Another approach,  popular this last decade, has been to throw more disks at the problem, resulting in many more seeking heads to handle the workload and who care where the data lies.
  • IO latency is another factor.  We have discussed this before (see Storage throughput vs IO response time and why it matters. But one key to systems throughput is how quickly data gets out of cache and into the hands of servers. Of course the other part to this, is how fast does the storage system get the data from sitting on disk into cache.

Systems that do these better will perform better on SPC-2 like benchmarks that focus on raw sequential throughput.

Comments?

—–

The full SPC performance report went out to our newsletter subscribers last month.  A copy of the full report will be up on the dispatches page of our website later next month. However, you can get this information now and subscribe to future newsletters to receive these reports even earlier by just sending us an email or using the signup form above right.

As always, we welcome any suggestions on how to improve our analysis of SPC results or any of our other storage system performance discussions.

 

 

CIFS vs. NFS the saga continues, recent SPECsfs2008 results- chart of the month


SCISFS110628-004A (c) 2011 Silverton Consulting, Inc., All Rights Reserved
SCISFS110628-004A (c) 2011 Silverton Consulting, Inc., All Rights Reserved

When last we discussed this topic, the tides had turned and the then current SPECsfs 2008 results had shown that any advantage that CIFS had over NFS was an illusion.

Well there has been more activity for both CIFS and NFS protocols since our last discussion and it showed, once again that CIFS was faster than NFS but rather than going down that same path again, I decided to try something different.

As a result, we published the above chart which places all NFS and CIFS disk only submissions in stark contrast.

This chart was originally an attempt to refute many analysts contention that storage benchmarks are more of a contest as to who has thrown more disks at the problem rather than some objective truth about the performance of one product or another.

But a curious thought occurred to me as I was looking at these charts for CIFS and NFS last month. What if I plotted both results on the same chart?  Wouldn’t such a chart provide some additional rationale to our discussion on CIFS vs. NFS.

Sure enough, it did.

From my perspective this chart proves that CIFS is faster than NFS.  But, maybe a couple of points might clarify my analysis:

  1. I have tried to eliminate any use of SSDs or NAND caching from this chart as they just confound the issue.  Also, all disk-based, NFS and CIFS benchmarks are represented on the above charts, not just those that have submitted both CIFS and NFS results on the same hardware.
  2. There is an industry wide view that CIFS and NFS are impossible to compare because one is state-full (CIFS) and the other state-less (NFS).  I happen to think this is wrong.  Most users just want to know which is faster and/or better.  It would be easier to do analyze this if SPECsfs2008 reported data transfer rates rather than operations/second rates but they don’t.
  3. As such, one potential problem with comparing the two on the above chart is that the percentage of “real” data transfers represented by “operations per second” may be different.  Ok, this would need to be normalized if they were a large difference between CIFS and NFS.  But when examining the SPECsfs2008 user’s guide spec., one sees that NFS read and write data ops is 28.0% of all operations and CIFS read and write data ops is 29.1% of all operations.  As they aren’t that different, the above chart should correlate well to the number of data operations done by each separate protocol. If anything, normalization would show an even larger advantage for CIFS, not less.
  4. Another potential concern one needs to consider is the difference in the average data transfer size between the protocols.  The user guide doesn’t discriminate between access transfer rates for NFS or CIFS, so we assume it’s the same for the two protocols. Given that assumption, then the above chart provides a reasonable correlation to the protocols relative data transfer rates.
  5. The one real concern on this chart is the limited amount of CIFS disk benchmarks.  At this time there are about 20 CIFS disk benchmarks vs. 40 NFS disk benchmarks. So the data is pretty slim for CIFS, nonetheless, 20 is almost enough to make this statistically significant.  So with more data the advantage may change slightly but I don’t think it will ever shift back to NFS.

Ok, now that I have all the provisos dealt with, what’s the chart really telling me.

One has to look at the linear regression equations to understand this but, CIFS does ~463.0 operations/second per disk drive and NFS does ~296.5 operations/second per disk drive.  What this says is, for all things being equal, i.e., the same hardware and disk drive count, CIFS does about 1.6X (463.0/296.5) more operations per second than NFS and correspondingly, CIFS provides ~1.6X more data per second than NFS does.

Comments?

—–

The full SPECsfs 2008 report went out to our newsletter subscribers last month. The above chart has been modified somewhat from a plot in our published report, but the data is the same (forced the linear equations to have an intercept of 0 to eliminate the constant, displayed the R**2 for CIFS, and fixed the vertical axis title).

A copy of the full SPECsfs report will be up on the dispatches page of our website later next month. However, you can get this information now and subscribe to future newsletters to receive these reports even earlier by just emailing us at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter or using the signup form above and to the right.

As always, we welcome any suggestions on how to improve our analysis of SPECsfs or any of our other storage system performance discussions.

 

 

SCI May 2011, Latest SPC-1 results IOPS vs. drive count – chart of the month

SCISPC110527-004 (c) 2011 Silverton Consulting, Inc., All Rights Reserved
SCISPC110527-004 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

The above chart is from our May Storage Intelligence newsletter dispatch on system performance and shows the latest Storage Performance Council SPC-1 benchmark results in a scatter plot with IO/sec [or IOPS(tm)] on the vertical axis and number of disk drives on the horizontal axis.  We have tried to remove all results that used NAND flash as a cache or SSDs. Also this displays only results below a $100/GB.

One negative view of benchmarks such as SPC-1 is that published results are almost entirely due to the hardware thrown at it or in this case, the number of disk drives (or SSDs) in the system configuration.  An R**2 of 0.93 shows a pretty good correlation of IOPS performance against disk drive count and would seem to bear this view out, but is an incorrect interpretation of the results.

Just look at the wide variation beyond the 500 disk drive count versus below that where there are only a few outliers with a much narrower variance. As such, we would have to say that at some point (below 500 drives), most storage systems can seem to attain a reasonable rate of IOPS as a function of the number of spindles present, but after that point the relationship starts to break down.  There are certainly storage systems at the over 500 drive level that perform much better than average for their drive configuration and some that perform worse.

For example, consider the triangle formed by the three best performing (IOPS) results on this chart.  The one at 300K IOPS with ~1150 disk drives is from Huawei Symantec and is their 8-node Oceanspace S8100 storage system whereas the other system with similar IOPS performance at ~315K IOPS used ~2050 disk drives and is a 4-node, IBM SVC (5.1) system with DS8700 backend storage.   In contrast, the highest performer on this chart at ~380K IOPS, also had ~2050 disk drives and is a 6-node IBM SVC (5.1) with DS8700 backend storage.

Given the above analysis there seems to be much more to system performance than merely disk drive count, at least at the over 500 disk count level.

—-

The full performance dispatch will be up on our website after the middle of next month but if you are interested in viewing this today, please sign up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we’ll send you the current issue.  If you need a more analysis of SAN storage performance please consider purchasing SCI’s SAN Storage Briefing.

As always, we welcome all constructive suggestions on how to improve any of our storage performance analyses.

Comments?

 

Recent ESRP v3.0 (Exchange 2010) performance results – chart of the month

SCIESRP110127-003 (c) 2011 Silverton Consulting, Inc., All Rights Reserved
SCIESRP110127-003 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

We return to our monthly examination of storage performance, and this month’s topic is Exchange 2010 performance comparing results from the latest ESRP v3.0 (Exchange Solution Review Program).  This latest batch is for the 1K-and-under mailbox category and the log playback chart above represents the time it takes to process a 1MB log file and apply this to the mailbox database(s).  Data for this report is taken from Microsoft’s ESRP v3.0 website published results and this chart is from our latest storage intelligence performance dispatch sent out in our free monthly newsletter.

Smaller is better on the log playback chart.  As one can see it takes just under 2.4 seconds for the EMC Celerra NX4 to process a 1MB log file whereas it takes over 7.5 seconds on an EMC Iomega IX12-300r storage subsystem.  To provide some perspective in the next larger category, for storage supporting from 1K-to-5K mailboxes,  the top 10 log playback times range from ~0.3 to ~4.5 seconds and as such, the Celerra NX4 system and the other top four subsystems here would be in the top 10 log playback times for that category as well.

Why log playback

I have come to believe that log playback is an important metric in Exchange performance, for mainly one reason, it’s harder to game using Jetstress paramaterization.   For instance, with Jetstress one must specify how much IO activity is generated on a per mailbox basis, thus generating more or less requests for email database storage. Such specifications will easily confound storage performance metrics such as database accesses/second when comparing storage. But with log playback, that parameter is immaterial and every system has the same 1MB sized log file to process as fast as passible, i.e., it has to be read and applied to the configured Exchange database(s).

One can certainly still use a higher performing storage system, and/or throw SSD, more drives or more cache at the problem to gain better storage performance but that also works for any other ESRP performance metric.  But with log playback, Jetstress parameters are significantly less of a determinant of storage performance.

In the past I have favored database access latency charts for posts on Microsoft Exchange performance but there appears to be much disagreement as to the efficacy of that metric in comparing storage performance (e.g., see the 30+ comments on one previous ESRP post).  I still feel that latency is an important metric and one that doesn’t highly depend on Jetstress IO/sec/mailbox parameter but log playback is even more immune to that parm and so, should be less disputable.

Where are all the other subsystems?

You may notice that there are less than 10 subsystems on the chart. These six are the only subsystems that have published results in this 1K-and-under mailbox category.  One hopes that the next time we review this category there will be more subsystem submissions available to discuss here.  Please understand, ESRP v3.0 is only a little over a year old when our briefing came out.

—-

The full performance dispatch will be up on our website after month end but if one needs to see it sooner, please sign up for our free monthly newsletter (see subscription widget, above right) or subscribe by email and we’ll send you the current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of block storage performance please consider purchasing SCI’s SAN StorInt Briefing also available from our website.

As always, we welcome any constructive suggestions on how to improve any of our storage performance analyses.

Comments?

SCI’s latest SPECsfs2008 NFS ops vs. system size – chart of the month

(c) 2011 Silverton Consulting, Inc., All Rights Reserved
(c) 2011 Silverton Consulting, Inc., All Rights Reserved

We return to our periodic discussion of storage system performance, this time on SPECsfs(r)2008 benchmark results for NFS file serving ops per second vs. system reported memory size.  For some obscure reason, I was very intrigued with this chart.

Chart description

We have broken the data out to show those system that only used DRAM in system memory with only hard disk drives and those systems that also included either NAND cache in system memory or SSDs.  Current SPECsfs2008 results show 33 systems with DRAM and disk drives and only 6 using SSDs or NAND cache for NFS results.

The horizontal axis shows system memory size for the systems under test and doesn’t include SSD capacity (considered drive capacity by SPECsfs2008) size but does include NAND cache size (considered system memory by SPECsfs2008).  The vertical axis shows maximum NFS throughput operations per second attained by the storage.  The two lines are Excel generated linear regressions across the two sets of data (DRAM-Disk only systems and SSD or NAND caching systems).

Discussion of results

Given the limited data we probably can’t conclude much from the SSD-NAND linear regression line other than it’s different and somewhat less than what can be gained on average from a system using DRAM and disk only.  Also the regression coefficient (R**2) of either linear regression is not that great (~0.62 for DRAM-Disk only and ~0.69 for SSD or NAND use) which might be stretching any real discussion based on statistical normalcy. Nevertheless, one question that emerges is why would SSD or NAND use not generate an relatively equivalent amount of NFS throughput as systems with DRAM only?

Obviously, NAND-SSDs access times are not as fast as DRAM.  Thus, if I had 800GB of DRAM, I could potentially access data faster than if I had 800GB of NAND cache or SSDs, all things being equal.   However, when talking about access time frames at the sub-msec level one would think it wouldn’t make that much of a difference, but apparently it does.

Also, given the limited data on SSDs vs. NAND cache use, it’s hard to make any distinction between these two types of systems but one can hope that as more data comes in, we can answer this as well.  Another question is whether SSD capacity should be considered system memory or drive capacity for benchmark purposes.  SPECsfs2008 states that SSD size should be considered drive capacity but that’s open to some debate which more data could help resolve.

Finally, missing from SPECsfs2008 reports is any statement of system cost.  As such, it’s impossible to add any measure of relative cost effectiveness factor to this discussion. However, given the current price differential (on $/GB) between DRAM and NAND memory or SSD capacity, one could probably conclude that on a cost effectiveness basis the relative advantages of DRAM only systems might diminish.  But without system cost information that’s difficult to quantify.

—-

The full performance dispatch will be up on our website after month end but if one is interested in seeing it sooner sign up for our free monthly newsletter (see subscription widget, above right) or subscribe by email and we will send the current issue along with download instructions for this and other reports.  If you need an even more in-depth analysis of NAS system performance please consider purchasing SCI’s NAS Buying Guide also available from our website.

As always, we welcome any constructive suggestions on how to improve any of our storage performance analysis.

Comments?