Category: SPECsfs

SPECsfs2008 CIFS vs. NFS results – chart of the month

Posted on by Ray in CIFS/SMB throughput, File Storage, NFS throughput, SPECsfs, SPECsfs2008, Storage, Storage performance
SPECsfs(R) 2008 CIFS vs. NFS 2010Mar17
SPECsfs(R) 2008 CIFS vs. NFS 2010Mar17

We return now to our ongoing quest to understand the difference between CIFS and NFS performance in the typical data center.  As you may recall from past posts and our newsletters on this subject, we had been convinced that in SPECsfs 2008 CIFS had almost 2X the throughput of NFS in SPECsfs 2008 benchmarks.  Well as you can see from this updated chart this is no longer true.

Thanks to EMC for proving me wrong (again).  Their latest NFS and CIFS result utilized a NS-G8 Celerra gateway server in front of V-Max backend using SSDs and FC disks. The NS-G8 was the first enterprise class storage subsystem to release both a CIFS and NFS SPECsfs 2008 benchmark.

As you can see from the lower left quadrant all of the relatively SMB level systems (under 25K NFS throughput ops/sec) showed a consistent pattern of CIFS throughput being ~2X NFS throughput.  But when we added the Celerra V-Max combination to the analysis it brought the regression line down considerably and now the equation is:

CIFS throughput = 0.9952 X NFS throughput + 10565, with a R**2 of 0.96,

what this means is that CIFS and NFS throughput are roughly the same now.

When I first reported the relative advantage of CIFS over NFS throughput in my newsletter I was told that you cannot compare the two results mainly because NFS was “state-less” and CIFS was “state-full” and a number of other reasons (documented in the earlier post and in the newsletter).  Nonetheless, I felt that it was worthwhile to show the comparison because at the end of the day whether some file happens to be serviced by NFS or CIFS may not matter to the application/user, it should matter significantly to the storage administrator/IT staff.  By showing the relative performance of each we were hoping to help IT personnel to decide between using CIFS or NFS storage.

Given the most recent results, it seems that the difference in throughput is not that substantial irregardless of their respective differences.  Of course more data will help. There seems to be a wide gulf between the highest SMB submission and the EMC enterprise class storage that should be filled out.  As Celerra V-Max is the only enterprise NAS to submit both CIFS and NFS benchmarks there could still be many surprises in store. As always, I would encourage storage vendors to submit both NFS and CIFS benchmarks for the same system so that we can see how this pattern evolves over time.

The full SPECsfs 2008 report should have went out to our newsletter subscribers last month but I had a mistake with the link.  The full report will be delivered with this months newsletter along with a new performance report on Exchange Solution Review Program and storage announcement summaries.  In addation, a copy of the SPECsfs report will be up on the dispatches page of our website later next month. However, you can get this information now and subscribe to future newsletters to receive future full reports even earlier, just email us at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter.

As always, we welcome any suggestions on how to improve our analysis of SPECsfs or any of our other storage system performance results.

Latest SPECsfs2008 CIFS performance – chart of the month

Posted on by Ray in CIFS/SMB throughput, File Storage, SPECsfs, SPECsfs2008, Storage, Storage performance

Above we reproduce a chart from our latest newsletter StorInttm Dispatch on SPECsfs(R) 2008 benchmark results.  This chart shows the top 10 CIFS throughput benchmark results as of the end of last year.  As observed in the chart Apple’s Xserve running Snow Leopard took top performance with over 40K CIFS throughput operations per second.  My problem with this chart is that there are no enterprise class systems represented in the top 10 or for that matter (not shown in the above) in any CIFS result.

Now some would say it’s still early yet in the life of the 2008 benchmark but it has been out now for 18 months and still has not a single enterprise class system submission reported.  Possibly, CIFS is not considered an enterprise class protocol but I can’t believe that given the proliferation of Windows.  So what’s the problem?

I have to believe it’s part tradition, part not wanting to look bad, and part just lack of awareness on the part of CIFS users.

  • Traditionally, NFS benchmarks were supplied by SPECsfs and CIFS benchmarks were supplied elsewhere, i.e., NetBenc. However, there never was a central repository for NetBench results so comparing system performance was cumbersome at best.  I believe that’s one reason for SPECsfs’s CIFS benchmark.  Seeing the lack of a central repository for a popular protocol, SPECsfs created their own CIFS benchmark.
  • Performance on system benchmarks are always a mixed bag.  No-one wants to look bad and any top performing result is temporary until the next vendor comes along.  So most vendors won’t release a benchmark result unless it shows well for them.  Not clear if Apple’s 40K CIFS ops is a hard number to beat, but it’s been up there for quite awhile now, and has to tell us something.
  • CIFS users seem to be aware and understand NetBench but don’t have similar awareness on SPECsfs CIFS benchmark yet.  So, given today’s economic climate, any vendor wanting to impress CIFS customers would probably choose to ignore SPECsfs and spend their $s on NetBench.  The fact that comparing results was neigh impossible, could be considered an advantage for many vendors.

So SPECsfs CIFS just keeps going on.  One way to change this dynamic is to raise awareness.  So as more IT staff/consultants/vendors discuss SPECsfs CIFS results, its awareness will increase.  I realize some of  my analysis on CIFS and NFS performance results doesn’t always agree with the SPECsfs party line, but we all agree that this benchmark needs wider adoption.  Anything that can be done to facilitate that deserves my (and their) support.

So for all my storage admins, CIOs and other influencers of NAS system purchases friends out there, you need to start asking to about SPECsfs CIFS benchmark results.  All my peers out their in the consultant community, get on the bandwagon.  As for my friends in the vendor community, SPECsfs CIFS benchmark results should be part of any new product introduction.  Whether you want to release results is and always will be, a marketing question but you all should be willing to spend the time and effort to see how well new systems perform on this and other benchmarks.

Now if I could just get somebody to define an iSCSI benchmark, …

Our full report on the latest SPECsfs 2008 results including both NFS and CIFS performance, will be up on our website later this month.  However, you can get this information now and subscribe to future newsletters to receive the full report even earlier, just email us at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter.

Latest SPECsfs2008 results – chart of the month

Posted on by Ray in File Storage, NFS throughput, SPECsfs, SPECsfs2008, Storage, Storage performance, Storage reliability
Top 10 SPEC(R) sfs2008 NFS throughput results as of 25Sep2009
Top 10 SPEC(R) sfs2008 NFS throughput results as of 25Sep2009

The adjacent chart is from our September newsletter and shows the top 10 NFSv3 throughput results from the latest SPEC(R) sfs2008 benchmark runs published as of 25 September 2009.

There have been a number of recent announcements of newer SPECsfs2008 results in the news of late, namely Symantec’s FileStore and Avere Systems releases but these results are not covered here. In this chart, the winner is the NetApp FAS6080 with FCAL disks behind it, clocking in at 120K NFSv3 operations/second. This was accomplished with 324 disk drives using 2-10Gbe links.

PAM comes out

All that’s interesting of course but what is even more interesting is NetApp’s results with their PAM II (Performance Accelerator Module) cards. The number 3, 4 and 5 results were all with the same system (FAS3160) with different configurations of disks and PAM II cards. Specifically,

  • The #3 result had a FAS3160, running 56 FCAL disks with PAM II cards and DRAM cache of 532GBs. The system attained 60.5K NFSv3 operations per second.
  • The #4 result had a FAS3160, running 224 FCAL disks with no PAM II cards but 20GB of DRAM cache. This system attained 60.4K NFSv3 ops/second.
  • The #5 result had a FAS3160, running 96 SATA disks with PAM II cards and DRAM cache of 532GBs. This system also attained 60.4K NFSv3 ops/second.

Similar results can be seen with the FAS3140 systems at #8, 9 and 10. In this case the FAS3140 systems were using PAM I (non-NAND) cards with 41GB of cache for results #9 and 10, while #8 result had no PAM with only 9GB of Cache. The #8 result used 224 FCAL disks, #9 used 112 FCAL disks, and #10 had 112 SATA disks. They were able to achieve 40.1K, 40.1K and 40.0K NFSv3 ops/second respectively.

Don’t know how much PAM II cards cost versus FCAL or SATA disks but there is an obvious trade off here. You can use less FCAL or cheaper SATA disks but attain the same NFSv3 ops/second performance.

As I understand it, the PAM II cards come in 256GB configurations and you can have 1 or 2 cards in a FAS system configuration. PAM cards act as an extension of FAS system cache and all IO workloads can benefit from their performance.

As with all NAND flash, write access is significantly slower than read and NAND chip reliability has to be actively managed through wear leveling and other mechanisms to create a reliable storage environment. We assume either NetApp has implemented the appropriate logic to support reliable NAND storage or has purchased NAND cache with the logic already onboard. In any case, the reliability of NAND is more concerned with write activity than read and by managing the PAM cache to minimize writes, NAND reliability concerns could easily be avoided.

The full report on the latest SPECsfs2008 results will be up on my website later this week but if you want to get this information earlier and receive your own copy of our newsletter – email me at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter.

Full disclosure: I currently have a contract with NetApp on another facet of their storage but it is not on PAM or NFSv3 performance.

Chart of the Month

Posted on by Ray in CIFS/SMB throughput, NFS throughput, Storage, Storage performance

CIFS vs. NFS Throughput Results from SPECsfs(R) 2008 benchmarks
CIFS vs. NFS Throughput Results from SPECsfs(R) 2008 benchmarks
The chart to the left was sent out in last months SCI newsletter and shows the throughput attained by various storage systems when running the SPECsfs(R) 2008 CIFS and NFS benchmarks. The scatter-plot shows data for NAS systems that have published both NFS and CIFS SPECsfs benchmark results for the same storage system. To date (June 2009), only 5 systems have published results for both benchmarks.

The scatter plot clearly shows with a high regression coefficient (.97) that the same system can typically provide over 2.4X the throughput using CIFS as it can using NFS. My friends at SPECsfs would want me to point out that these two benchmarks are not intended to be comparable and I present this with a few caveats in the newsletter:

  • NFS operations are stateless and CIFS are stateful, the distribution of file sizes are different for the two benchmarks, the relative proportions of the respective IO workloads don’t match up exactly (CIFS has more reads and less writes than NFS), and all remaining (non-read/write) operations are completely different for each workload.
  • Most of these results come from the same vendor (Apple) and it’s implementation of NFS or CIFS target support may skew results
  • Only 5 storage systems have published results for these two benchmarks and probably in all honesty do not represent a statistically valid comparison
  • Usually host implementations of CIFS or NFS impact results such as these but for SPECsfs, the benchmark implements each protocol stack. As such, SPECsfs benchmark’s implementation of CIFS or NFS protocols may also skew results

All that being said, I believe it’s an interesting and current fact that for SPECsfs 2008 benchmarks CIFS has 2.4X the throughput of NFS.

In talking with real world customers and vendors on which is better the story seems much more mixed. I heard where one O/S had a much better implementation of the CIFS protocol and as such, customers moved to use CIFS for those systems. I haven’t seen much discussion about storage systems being better or worse on one protocol over the other but it’s certainly probable.

From my perspective any storage admin looking to configure new NAS storage should try out CIFS first to see if it performs well before trying NFS. Given my inclinations, I would probably try out both, but that’s just me.

If you are interested in seeing the full report on latest SPECsfs 2008 results please subscribe to my newsletter by emailing me at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter.

Otherwise the full report will be on my website sometime next week.