Analyzing SPECsfs2008 flash use in NFS performance – chart-of-the-month

(SCISFS120316-002) (c) 2012 Silverton Consulting, All Rights Reserved
(SCISFS120316-002) (c) 2012 Silverton Consulting, All Rights Reserved

For some time now I have been using OPS/drive to measure storage system disk drive efficiency but have so far failed to come up with anything similar for flash or SSD use.  The problem with flash in storage is that it can be used as a cache or as a storage device.  Even when used as a storage device under automated storage tiering, SSD advantages can be difficult to pin down.

In my March newsletter as a first attempt to measure storage system flash efficiency I supplied a new chart shown above, which plots the top 10 NFS throughput ops/second/GB of NAND used in the SPECsfs2008 results.

What’s with Avere?

Something different has occurred with the (#1) Avere FXT 3500 44-node system in the chart.   The 44-node Avere system only used ~800GB of flash as a ZIL (ZFS intent log from the SPECsfs report).   However, the 44-node system also had ~7TB of DRAM across their 44-node system, most of which was used for file IO caching.  If we incorporated storage system memory size with flash GB in the above chart it would have dropped the Avere numbers by a factor of 9 while only dropping the others by a factor of ~2X which would still give the Avere a significant advantage but not quite so stunning.  Also, the Avere system frontends other NAS systems, (this one running ZFS) so it’s not quite the same as being a direct NAS storage system like the others on this chart.

The remainder of the chart (#2-10) belongs to NetApp and their FlashCache (or PAM) cards.  Even Oracles Sun ZFS Storage 7320 appliance did not come close to either the Avere FXT 3500 system or the NetApp storage on this chart.  But there were at least 10 other SPECsfs2008 NFS results using some form of flash but were not fast enough to rank on this chart.

Other measures of flash effectiveness

This metric still doesn’t quite capture flash efficiency.  I was discussing flash performance with another startup the other day and they suggested that SSD drive count might be a better  alternative.  With such a measure, it would take into consideration that each SSD has a only a certain performance level it can sustain, not unlike disk drives.

In that case Avere’s 44-node system had 4 drives, and each NetApp system had two FlashCache cards, representing 2-SSDs per NetApp node.  I try that next time to see if it’s  a better fit.

~~~~

The complete SPECsfs2008 performance report went out in SCI’s March newsletter.  But a copy of the report will be posted on our dispatches page sometime next month (if all goes well). However, you can get the SPECsfs performance analysis now and subscribe to future free newsletters by just sending us an email or using the signup form above right.

For a more extensive discussion of current NAS or file system storage performance covering SPECsfs2008 (Top 20) results and our new ChampionChart™ for NFS and CIFS storage systems, please see SCI’s NAS Buying Guide available from our website.

As always, we welcome any suggestions or comments on how to improve our analysis of SPECsfs2008 results or any of our other storage performance analyses.


Latest SPECsfs2008 results, over 1 million NFS ops/sec – chart-of-the-month

Column chart showing the top 10 NFS througput operations per second for SPECsfs2008
(SCISFS111221-001) (c) 2011 Silverton Consulting, All Rights Reserved

[We are still catching up on our charts for the past quarter but this one brings us up to date through last month]

There’s just something about a million SPECsfs2008(r) NFS throughput operations per second that kind of excites me (weird, I know).  Yes it takes over 44-nodes of Avere FXT 3500 with over 6TB of DRAM cache, 140-nodes of EMC Isilon S200 with almost 7TB of DRAM cache and 25TB of SSDs or at least 16-nodes of NetApp FAS6240 in Data ONTAP 8.1 cluster mode with 8TB of FlashCache to get to that level.

Nevertheless, a million NFS throughput operations is something worth celebrating.  It’s not often one achieves a 2X improvement in performance over a previous record.  Something significant has changed here.

The age of scale-out

We have reached a point where scaling systems out can provide linear performance improvements, at least up to a point.  For example, the EMC Isilon and NetApp FAS6240 had a close to linear speed up in performance as they added nodes indicating (to me at least) there may be more there if they just throw more storage nodes at the problem.  Although maybe they saw some drop off and didn’t wish to show the world or potentially the costs became prohibitive and they had to stop someplace.   On the other hand, Avere only benchmarked their 44-node system with their current hardware (FXT 3500), they must have figured winning the crown was enough.

However, I would like to point out that throwing just any hardware at these systems doesn’t necessary increase performance.  Previously (see my CIFS vs NFS corrected post), we had shown the linear regression for NFS throughput against spindle count and although the regression coefficient was good (~R**2 of 0.82), it wasn’t perfect. And of course we eliminated any SSDs from that prior analysis. (Probably should consider eliminating any system with more than a TB of DRAM as well – but this was before the 44-node Avere result was out).

Speaking of disk drives, the FAS6240 system nodes had 72-450GB 15Krpm disks, the Isilon nodes had 24-300GB 10Krpm disks and each Avere node had 15-600GB 7.2Krpm SAS disks.  However the Avere system also had a 4-Solaris ZFS file storage systems behind it each of which had another 22-3TB (7.2Krpm, I think) disks.  Given all that, the 16-node NetApp system, 140-node Isilon and the 44-node Avere systems had a total of 1152, 3360 and 748 disk drives respectively.   Of course, this doesn’t count the system disks for the Isilon and Avere systems nor any of the SSDs or FlashCache in the various configurations.

I would say with this round of SPECsfs2008 benchmarks scale-out NAS systems have come out.  It’s too bad that both NetApp and Avere didn’t release comparable CIFS benchmark results which would have helped in my perennial discussion on CIFS vs. NFS.

But there’s always next time.

~~~~

The full SPECsfs2008 performance report went out to our newsletter subscriber’s last December.  A copy of the full report will be up on the dispatches page of our site sometime later this month (if all goes well). However, you can see our full SPECsfs2008 performance analysis now and subscribe to our free monthly newsletter to receive future reports directly by just sending us an email or using the signup form above right.

For a more extensive discussion of file and NAS storage performance covering top 30 SPECsfs2008 results and NAS storage system features and functionality, please consider purchasing our NAS Buying Guide available from SCI’s website.

As always, we welcome any suggestions on how to improve our analysis of SPECsfs2008 results or any of our other storage system performance discussions.

Comments?

Latest SPECsfs2008 benchmarks analysis, CIFS vs NFS corrected – chart of the month

(SCISFS110929-001) (c) 2011 Silverton Consulting, All Rights Reserved
(SCISFS110929-001) (c) 2011 Silverton Consulting, All Rights Reserved

We made a mistake in our last post discussing CIFS vs. NFS results using SPECsfs2008 benchmarks by including some storage systems that had SSDs in this analysis. All of our other per spindle/disk drive analyses exclude SSDs and NAND cache because they skew per drive results so much.  We have corrected this in the above chart which includes all the SPECsfs2008 results, up to the end of last month.

However, even with the corrections the results stand pretty much the way they were. CIFS is doing more throughput per disk drive spindle than NFS for all benchmark results not using SSDs or Flash Cache.

Dropping SSD results changed the linear regression equation. Specificall,  the R**2 for CIFS and NFS dropped from 0.99 to 0.98 and from 0.92 to 0.82 and the B coefficient dropped from 463 to 405 and from 296 to 258 respectively.

I would be remiss if I didn’t discuss a few caveats with this analysis.

  • Now there are even less results in both CIFS and NFS groups, down to 15 for CIFS and 38 for NFS.   For any sort of correlation comparison, more results would have better statistical significance.
  • In the NFS data, we include some NAS systems which have lots of DRAM cache (almost ~0.5TB).  We should probably exclude these as well, which might drop the NFS line down some more (at least lower the B value).
  • There are not a lot of enterprise level CIFS systems in current SPECsfs resuslts, with or without SSD or NAND caching.  Most CIFS benchmarks are from midrange or lower filers.  Unclear why these would do much better on a per spindle basis than a wider sample of NFS systems, but they obviously do.

All that aside, it seems crystal clear here, that CIFS provides more throughput per spindle.

In contrast, we have shown in the past posts how for the limited number of systems that submitted benchmarks with both CIFS and NFS typically show roughly equivalent throughput results for CIFS and NFS. (See my other previous post on this aspect of the CIFS vs. NFS discussion).

Also, in our last post we discussed some of the criticism leveled against this analysis and provided our view to refute these issues. Mostly their concerns are due to the major differences between CIFS state-full protocol and NFS stateless protocol.

But from my perspective it’s all about the data.  How quickly can I read a file, how fast can I create a file.  Given similar storage systems, with similar SW, cache and hard disk drives, it’s now clear to me that CIFS provides faster access to data than NFS does, at least on a per spindle basis.

Nevertheless, more data may invalidate these results, so stay tuned.

—-

Why this is should probably be subject for another post but it may have a lot to do with the fact that it is stateless….

Comments?

CIFS vs. NFS the saga continues, recent SPECsfs2008 results- chart of the month


SCISFS110628-004A (c) 2011 Silverton Consulting, Inc., All Rights Reserved
SCISFS110628-004A (c) 2011 Silverton Consulting, Inc., All Rights Reserved

When last we discussed this topic, the tides had turned and the then current SPECsfs 2008 results had shown that any advantage that CIFS had over NFS was an illusion.

Well there has been more activity for both CIFS and NFS protocols since our last discussion and it showed, once again that CIFS was faster than NFS but rather than going down that same path again, I decided to try something different.

As a result, we published the above chart which places all NFS and CIFS disk only submissions in stark contrast.

This chart was originally an attempt to refute many analysts contention that storage benchmarks are more of a contest as to who has thrown more disks at the problem rather than some objective truth about the performance of one product or another.

But a curious thought occurred to me as I was looking at these charts for CIFS and NFS last month. What if I plotted both results on the same chart?  Wouldn’t such a chart provide some additional rationale to our discussion on CIFS vs. NFS.

Sure enough, it did.

From my perspective this chart proves that CIFS is faster than NFS.  But, maybe a couple of points might clarify my analysis:

  1. I have tried to eliminate any use of SSDs or NAND caching from this chart as they just confound the issue.  Also, all disk-based, NFS and CIFS benchmarks are represented on the above charts, not just those that have submitted both CIFS and NFS results on the same hardware.
  2. There is an industry wide view that CIFS and NFS are impossible to compare because one is state-full (CIFS) and the other state-less (NFS).  I happen to think this is wrong.  Most users just want to know which is faster and/or better.  It would be easier to do analyze this if SPECsfs2008 reported data transfer rates rather than operations/second rates but they don’t.
  3. As such, one potential problem with comparing the two on the above chart is that the percentage of “real” data transfers represented by “operations per second” may be different.  Ok, this would need to be normalized if they were a large difference between CIFS and NFS.  But when examining the SPECsfs2008 user’s guide spec., one sees that NFS read and write data ops is 28.0% of all operations and CIFS read and write data ops is 29.1% of all operations.  As they aren’t that different, the above chart should correlate well to the number of data operations done by each separate protocol. If anything, normalization would show an even larger advantage for CIFS, not less.
  4. Another potential concern one needs to consider is the difference in the average data transfer size between the protocols.  The user guide doesn’t discriminate between access transfer rates for NFS or CIFS, so we assume it’s the same for the two protocols. Given that assumption, then the above chart provides a reasonable correlation to the protocols relative data transfer rates.
  5. The one real concern on this chart is the limited amount of CIFS disk benchmarks.  At this time there are about 20 CIFS disk benchmarks vs. 40 NFS disk benchmarks. So the data is pretty slim for CIFS, nonetheless, 20 is almost enough to make this statistically significant.  So with more data the advantage may change slightly but I don’t think it will ever shift back to NFS.

Ok, now that I have all the provisos dealt with, what’s the chart really telling me.

One has to look at the linear regression equations to understand this but, CIFS does ~463.0 operations/second per disk drive and NFS does ~296.5 operations/second per disk drive.  What this says is, for all things being equal, i.e., the same hardware and disk drive count, CIFS does about 1.6X (463.0/296.5) more operations per second than NFS and correspondingly, CIFS provides ~1.6X more data per second than NFS does.

Comments?

—–

The full SPECsfs 2008 report went out to our newsletter subscribers last month. The above chart has been modified somewhat from a plot in our published report, but the data is the same (forced the linear equations to have an intercept of 0 to eliminate the constant, displayed the R**2 for CIFS, and fixed the vertical axis title).

A copy of the full SPECsfs report will be up on the dispatches page of our website later next month. However, you can get this information now and subscribe to future newsletters to receive these reports even earlier by just emailing us at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter or using the signup form above and to the right.

As always, we welcome any suggestions on how to improve our analysis of SPECsfs or any of our other storage system performance discussions.

 

 

Latest SPECsfs2008 results – chart of the month

(SCISFS110318-003) (c) 2011 Silverton Consulting, Inc., All Rights Reserved
(SCISFS110318-003) (c) 2011 Silverton Consulting, Inc., All Rights Reserved

The above chart comes from our last month’s newsletter on the lastest SPECsfs2008 file system performance benchmark results and depicts a scatter plot of system NFS throughput operations per second versus the number of disk drives in the system being tested.  We eliminate from this chart any system that makes use of Flash Cache/SSDS or any other performance use of NAND (See below on why SONAS was still included).

One constant complaint of benchmarks is that system vendors can just throw hardware at the problem to attain better results.   The scatter plot above is one attempt to get to the truth in that complaint.

The regression equation shows that NFS throughput operations per second = 193.68*(number of disk drives) + 23834. The regression coefficient (R**2) is 0.87 which is pretty good but not exactly perfect. So given these results, one would have to conclude there is some truth in the complaint but it doesn’t tell the whole story. (Regardless of how much it pains me to admit it).

A couple of other interesting things about the chart:

  • IBM released a new SONAS benchmark with 1975 disks, with 16 interface and 10 storage nodes to attain its 403K NFS ops/second. Now the SONAS had 512GB of NV Flash, which I assume is being used for redundancy purposes on writes and not as a speedup for read activity. Also the SONAS system complex had over 2.4TB of cache (includes the NV Flash).  So there was a lot of cache to throw at the problem.
  • HP BL860c results were from a system with 1480 drives, 4 nodes (blades) and ~800GB of cache to attain its 333KNFS ops/second.

(aside) Probably need to do a chart like this with amount of cache as the x variable (/aside)

In the same report we talked about the new #1 performing EMC VNX Gateway  that used 75TB of SAS-SSDs and 4 VNX5700’s as its backend. It was able to reach 497K NFS ops/sec.   It doesn’t show up on this chart because of its extensive use of SSDs.  But according to the equation above one would need to use ~2500 disk drives to attain similar performance without SSDS and I believe, a whole lot of cache.

—-

The full performance dispatch will be up on our website after the middle of next month (I promise) but if one is interested in seeing it sooner sign up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send the current issue along with download instructions for this and other reports.  If you need an even more in-depth analysis of NAS system performance please consider purchasing SCI’s NAS Buying Guide also available from our website.

As always, we welcome any constructive suggestions on how to improve any of our storage performance analysis.

Comments?

 

SCI’s latest SPECsfs2008 NFS ops vs. system size – chart of the month

(c) 2011 Silverton Consulting, Inc., All Rights Reserved
(c) 2011 Silverton Consulting, Inc., All Rights Reserved

We return to our periodic discussion of storage system performance, this time on SPECsfs(r)2008 benchmark results for NFS file serving ops per second vs. system reported memory size.  For some obscure reason, I was very intrigued with this chart.

Chart description

We have broken the data out to show those system that only used DRAM in system memory with only hard disk drives and those systems that also included either NAND cache in system memory or SSDs.  Current SPECsfs2008 results show 33 systems with DRAM and disk drives and only 6 using SSDs or NAND cache for NFS results.

The horizontal axis shows system memory size for the systems under test and doesn’t include SSD capacity (considered drive capacity by SPECsfs2008) size but does include NAND cache size (considered system memory by SPECsfs2008).  The vertical axis shows maximum NFS throughput operations per second attained by the storage.  The two lines are Excel generated linear regressions across the two sets of data (DRAM-Disk only systems and SSD or NAND caching systems).

Discussion of results

Given the limited data we probably can’t conclude much from the SSD-NAND linear regression line other than it’s different and somewhat less than what can be gained on average from a system using DRAM and disk only.  Also the regression coefficient (R**2) of either linear regression is not that great (~0.62 for DRAM-Disk only and ~0.69 for SSD or NAND use) which might be stretching any real discussion based on statistical normalcy. Nevertheless, one question that emerges is why would SSD or NAND use not generate an relatively equivalent amount of NFS throughput as systems with DRAM only?

Obviously, NAND-SSDs access times are not as fast as DRAM.  Thus, if I had 800GB of DRAM, I could potentially access data faster than if I had 800GB of NAND cache or SSDs, all things being equal.   However, when talking about access time frames at the sub-msec level one would think it wouldn’t make that much of a difference, but apparently it does.

Also, given the limited data on SSDs vs. NAND cache use, it’s hard to make any distinction between these two types of systems but one can hope that as more data comes in, we can answer this as well.  Another question is whether SSD capacity should be considered system memory or drive capacity for benchmark purposes.  SPECsfs2008 states that SSD size should be considered drive capacity but that’s open to some debate which more data could help resolve.

Finally, missing from SPECsfs2008 reports is any statement of system cost.  As such, it’s impossible to add any measure of relative cost effectiveness factor to this discussion. However, given the current price differential (on $/GB) between DRAM and NAND memory or SSD capacity, one could probably conclude that on a cost effectiveness basis the relative advantages of DRAM only systems might diminish.  But without system cost information that’s difficult to quantify.

—-

The full performance dispatch will be up on our website after month end but if one is interested in seeing it sooner sign up for our free monthly newsletter (see subscription widget, above right) or subscribe by email and we will send the current issue along with download instructions for this and other reports.  If you need an even more in-depth analysis of NAS system performance please consider purchasing SCI’s NAS Buying Guide also available from our website.

As always, we welcome any constructive suggestions on how to improve any of our storage performance analysis.

Comments?

Latest SPECsfs2008 results: NFSops vs. #disks – chart of the month

(c) 2010 Silverton Consulting, All Rights Reserved
(c) 2010 Silverton Consulting, All Rights Reserved

Above one can see a chart from our September SPECsfs2008 Performance Dispatch displaying the scatter plot of NFS Throughput Operations/Second vs. number of disk drives in the solution.  Over the last month or so there has been a lot of Twitter traffic on the theory that benchmark results such as this and Storage Performance Council‘s SPC-1&2 are mostly a measure of the number of disk drives in a system under test and have little relation to the actual effectiveness of a system.  I disagree.

As proof of my disagreement I offer the above chart.  On the chart we have drawn a linear regression line (supplied by Microsoft Excel) and displayed the resultant regression equation.  A couple of items to note on the chart:

  1. Regression Coefficient – Even though there are only 37 submissions which span anywhere from 1K to over 330K NFS throughput operations a second, we do not have a perfect correlation (R**2=~0.8 not 1.0) between #disks and NFS ops.
  2. Superior systems exist – Any of the storage systems above the linear regression line have superior effectiveness or utilization of their disk resources than systems below the line.

As one example, take a look at the two circled points on the chart.

  • The one above the line is from Avere Systems and is a 6-FXT 2500 node tiered NAS storage system which has internal disk cache (8-450GB SAS disks per node) and an external mass storage NFS server (24-1TB SATA disks) for data with each node having a system disk as well, totaling 79 disk drives in the solution.  The Avere system was able to attain ~131.5K NFS throughput ops/sec on SPECsfs2008.
  • The one below the line is from Exanet Ltd., (recently purchased by Dell) and is an 8-ExaStore node clusterd NAS system which has attached storage (576-146GB SAS disks) as well as mirrored boot disks (16-73GB disks) totaling 592 disks drives in the solution.  They were only able to attain ~119.6K NFS throughput ops/sec on the benchmark.

Now the two systems respective architectures were significantly different but if we just count the data drives alone, Avere Systems (with 72 data disks) was able to attain 1.8K NFS throughput ops per second per data disk spindle and Exanet (with 576 data disks) was able to attain only 0.2K NFS throughput ops per second per data disk spindle.  A 9X difference in per drive performance for the same benchmark.

As far as I am concerned this definitively disproves the contention that benchmark results are dictated by the number of disk drives in the solution.  Similar comparisons can be seen looking horizontally at any points with equivalent NFS throughput levels.

Rays reading: NAS system performance is driven by a number of factors and the number of disk drives is not the lone determinant of benchmark results.  Indeed, one can easily see differences in performance of almost 10X on a throughput ops per second per disk spindle for NFS storage without looking very hard.

We would contend that similar results can be seen for block and CIFS storage benchmarks as well which we will cover in future posts.

The full SPECsfs2008 performance report will go up on SCI’s website next month in our dispatches directory.  However, if you are interested in receiving this sooner, just subscribe by email to our free newsletter and we will send you the current issue with download instructions for this and other reports.

As always, we welcome any suggestions on how to improve our analysis of SPECsfs2008 performance information so please comment here or drop us a line.

SPECsfs2008 CIFS ORT performance – chart of the month

(c) 2010 Silverton Consulting Inc., All Rights Reserved
(c) 2010 Silverton Consulting Inc., All Rights Reserved

The above chart on SPECsfs(R) 2008 results was discussed in our latest performance dispatch that went out to SCI’s newsletter subscribers last month.  We have described Apple’s great CIFS ORT performance in previous newsletters but here I would like to talk about NetApp’s CIFS ORT results.

NetApp had three new CIFS submissions published this past quarter, all using the FAS3140 system but with varying drive counts/types and Flash Cache installed.  Recall that Flash Cache used to be known as PAM-II and is an onboard system card which holds 256GB of NAND memory used as an extension of system cache.  This differs substantially from using NAND in a SSD as a separate tier of storage as many other vendors currently do.  The newly benchmarked NetApp systems included:

  • FAS3140 (FCAL disks with Flash Cache) – used 56-15Krpm FC disk drives with 512GB of Flash Cache (2-cards)
  • FAS3140 (SATA disks with Flash Cache) – used 96-7.2Krpm SATA disk drives with 512GB of Flash Cache
  • FAS3140 (FCAL disks) – used 242-15Krpm FC disk drives and had no Flash Cache whatsoever

If I had to guess the point of this exercise was to show that one can offset fast performing hard disk drives by using FlashCache and significantly less (~1/4) fast disk drives or by using Flash Cache and somewhat more SATA drives.  In another chart from our newsletter one could see that all three systems resulted in very similar CIFS throughput results (CIFS Ops/Sec.), but in CIFS ORT (see above), the differences between the 3 systems are much more pronounced.

Why does Flash help CIFS ORT?

As one can see, the best CIFS ORT performance of the three came from the FAS3140 with FCAL disks and Flash Cache which managed a response time of ~1.25 msec.  The next best performer was the FAS3140 with SATA disks and Flash Cache with a CIFS ORT of just under ~1.48 msec.  The relatively worst performer of the bunch was the FAS3140 with only FCAL disks which came in at ~1.84 msec. CIFS ORT.  So why the different ORT performance?

Mostly the better performance is due to the increased cache available in the Flash Cache systems.  If one were to look at the SPECsfs 2008 workload one would find that less than 30% is read and write data activity and the rest is what one might call meta-data requests (query path info @21.5%, query file info @12.9%, create = close @9.7%, etc.).  While read data may not be very cache friendly, most of the meta-data and all the write activity are cache friendly.  Meta-data activity is more cache friendly primarily because it’s relatively small in size and any write data goes to cache before being destaged to disk.  As such, this more cache friendly workload generates on average, better response times when one has larger amounts of cache.

For proof one need look no further than the relative ORT performance of the FAS3140 with SATA and Flash vs. the FAS3140 with just FCAL disks.  The Flash Cache/SATA drive system had ~25% better ORT results than the FCAL only system even with significantly slower and much fewer disk drives.

The full SPECsfs 2008 performance report will go up on SCI’s website later this month in our dispatches directory.  However, if you are interested in receiving this report now and future copies when published, just subscribe by email to our free newsletter and we will email the report to you now.