SCI 2010 Mar 31 latest SPECsfs2008 benchmark results analysis including NFS and CIFS performance

In Celerra, CIFS/SMB throughput, Dell EMC, NFS throughput, NS-G8, ORT, SPECsfs, SPECsfs2008, Symmetrix, VMAX by Administrator

We now turn to analysis of the latest SPECsfs® 2008* benchmark results. Fortunately there were three new NFS and CIFS benchmarks over the last quarter, including two for EMC Celerra NS-G8 with a V-Max backend and one for Panasas.  But the most exciting item is that EMC benchmarked their Celerra system in both NFS and CIFS protocols.  Now that we have our 6th combined result we can revisit my contention that CIFS has better throughput than NFSv3.

Latest SPECsfs2008 CIFS results

(SCISFS100317-001) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISFS100317-001) (c) 2010 Silverton Consulting, All Rights Reserved

You may recall the last time we discussed this topic we claimed that CIFS had ~2X the throughput of NFSv3.  That was based on the first 5 results in the bottom left quadrant of this chart (see Figure 1).  With EMC’s latest CIFS result we must change this claim.

As shown above, CIFS and NFSv3 throughput were roughly the same for EMC (110K for NFS vs. 118K for CIFS).  Hence, our regression equation has changed significantly and now shows that CIFS throughput is roughly equal (0.99 multiplier) to NFSv3, with the addition of a 10.5K constant for CIFS.  More results would obviously help, but the results clearly show we were wrong to say

that CIFS had twice the throughput of NFS and now say that CIFS only has a slight advantage when compared to NFSv3 operations.

Of course, everyone we talked with thought we were wrong to compare the two at all.  They all said the two workloads represent completely different protocols, not the least of which that CIFS was state-full and NFSv3 stateless.  As such, they should not be compared.  Nonetheless, I still maintain that these two can be usefully compared and will continue to do so.

(SCISFS100317-002) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISFS100317-002) (c) 2010 Silverton Consulting, All Rights Reserved

We next review the latest CIFS throughput results.  Viewing Figure 2 above, EMC’s Celerra NS-G8/V-Max wins the top spot with no competition whatsoever.  One must realize there have only been somewhat limited CIFS results thus far, and EMC is probably the only tier one system having submitted one to date. Notwithstanding all that, I would say this is a pretty impressive result for EMC.

(SCISFS100317-003) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISFS100317-003) (c) 2010 Silverton Consulting, All Rights Reserved

We have decided to add another chart in our ongoing quixotic comparison of CIFS vs. NFSv3. This time we focus on operational response time or ORT (see Figure 3 above). ORT is the mean response time during the entire duration of the benchmark activity.  As the chart shows, CIFS has a much better response time than NFSv3.  Realize ORT was measured for similar (see discussion above) throughput activity and the R**2 was only ~0.7.

The argument that we are measuring two different protocols probably holds more weight when comparing ORT. State-fullness can only help CIFS in any ORT comparison.  Also, The exact number of non-data transfer operations to data transfer operations may not be that different between the CIFS and NFSv3 workloads but the style and responses to the non-data transfer operations are significantly different between the two.

Next we turn to CIFS absolute ORT result and here one can see the continued dominance of some of the earlier results as EMC Celerra comes in at #5 in the top 10 (see Figure 4 below).

(SCISFS100317-004) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISFS100317-004) (c) 2010 Silverton Consulting, All Rights Reserved

As we have discussed before, ORT results tend to be a transaction-oriented measurement and as such, provides a useful complement to SPECsfs’ other throughput results.

Latest SPECsfs2008 NFS results

There were two new NFS benchmarks, one from Panasas and another EMC Celerra NS-G8/V-Max run.  Both these results broke into the top 10 in throughput but neither altered the Top 10 ORT results for NFSv3.

(SCISFS100317-005) (c) 2010 Silverton Consulting, All Rights Reserved

(SCISFS100317-005) (c) 2010 Silverton Consulting, All Rights Reserved

In Figure 5, one can see the Celerra NS-G8 came in at #7 and the Panasas result came in at #9 in the top ten.  At this point, just about every top tier NAS system vendor has submitted at least one result for SPECsfs 2008 NFSv3.

Significance

Our score card for SPECsfs 2008 submissions now stands at 29 NFSv3 vs. 12 CIFS results.  Given the preponderance of CIFS usage in the field this seems more skewed than necessary.  I would encourage more vendors to submit CIFS results to address this unbalance.  Also, where it makes sense, be sure to include an NFSv3 result as well.  It will definitely help clarify my CIFS vs. NFSv3 comparisons and who knows it may prove me wrong yet again.

Nevertheless, it’s good to see more top end systems submitting SPECsfs 2008 results of any kind.  One can only hope that such submissions encourage more vendors to act.

This performance dispatch was originally sent out to our newsletter subscribers in March of 2010.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of NAS system features and performance please take the time to examine our recently updated (April 2019)  NAS Buying guide available for purchase from our website.

A PDF version of this dispatch can be found at

SCI 2010 March 31 Latest SPECsfs(R) 2008 performance results analysis

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.


* SPECsfs2008 results from http://www.spec.org/sfs2008/results/