Category: SPECsfs

Pure Storage FlashBlade well positioned for next generation storage

Posted on by Ray in Distributed computing, Ethernet, File Storage, NFS throughput, R&D measures, Software Defined Network, SSD storage, Strategic Inflection Points, System effectiveness

IMG_6344Sometimes, long after I listen to a vendor’s discussion, I come away wondering why they do what they do. Oftentimes, it passes but after a recent session with Pure Storage at SFD10, it lingered.

Why engineer storage hardware?

In the last week or so, executives at Hitachi mentioned that they plan to reduce  hardware R&D activities for their high end storage. There was much confusion what it all meant but from what I hear, they are ahead now, and maybe it makes more sense to do less hardware and more software for their next generation high end storage. We have talked about hardware vs. software innovation a lot (see recent post: TPU and hardware vs. software innovation [round 3]).
Continue reading “Pure Storage FlashBlade well positioned for next generation storage” →

SCI SPECsfs2008 NFS throughput per node – Chart of the month

Posted on by Ray in Clustered storage, File Storage, NFS throughput, SPECsfs2008, Storage performance

SCISFS150928-001
As SPECsfs2014 still only has (SPECsfs sourced) reference benchmarks, we have been showing some of our seldom seen SPECsfs2008 charts, in our quarterly SPECsfs performance reviews. The above chart was sent out in last months Storage Intelligence Newsletter and shows the NFS transfer operations per second per node.

In the chart, we only include NFS SPECsfs2008 benchmark results with configurations that have more than 2 nodes and have divided the maximum NFS throughput operations per second achieved by the node counts to compute NFS ops/sec/node.

HDS VSP G1000 with an 8 4100 file modules (nodes) and HDS HUS (VM) with 4 4100 file modules (nodes) came in at #1 and #2 respectively, for ops/sec/node, each attaining ~152K NFS throughput operations/sec. per node. The #3 competitor was Huawei OceanStor N8500 Cluster NAS with 24 nodes, which achieved ~128K NFS throughput operations/sec./node. At 4th and 5th place were EMC  VNX VG8/VNX5700 with 5 X-blades and Dell Compellent FS8600 with 4 appliances, each of which reached ~124K NFS throughput operations/sec. per node. It falls off significantly from there, with two groups at ~83K and ~65K NFS ops/sec./node.

Although not shown above, it’s interesting that there are many well known scale-out NAS solutions in SPECsfs2008 results with over 50 nodes that do much worse than the top 10 above, at <10K NFS throughput ops/sec/node. Fortunately, most scale-out NAS nodes cost quite a bit less than the above.

But for my money, one can be well served with a more sophisticated, enterprise class NAS system which can do >10X the NFS throughput operations per second per node than a scale-out systm. That is, if you don’t have to deploy 10PB or more of NAS storage.

More information on SPECsfs2008/SPECsfs2014 performance results as well as our NFS and CIFS/SMB ChampionsCharts™ for file storage systems can be found in our just updated NAS Buying Guide available for purchase on our web site.

Comments?

~~~~

The complete SPECsfs2008 performance report went out in SCI’s September newsletter.  A copy of the report will be posted on our dispatches page sometime this quarter (if all goes well).  However, you can get the latest storage performance analysis now and subscribe to future free monthly newsletters by just using the signup form above right.

As always, we welcome any suggestions or comments on how to improve our SPECsfs  performance reports or any of our other storage performance analyses.

 

New SPECsfs2008 CIFS/SMB vs. NFS (again) – chart of the month

Posted on by Ray in CIFS/SMB, CIFS/SMB throughput, File Storage, NFS, NFS throughput, SPECsfs, SPECsfs2008, Storage performance
SPECsfs2008 benchmark results for CIFS/SMB vs. NFS protocol performance
SCISFS140326-001 (c) 2014 Silverton Consulting, All Rights Reserved

The above chart represents another in a long line of charts on the relative performance of CIFS[/SMB] versus NFS file interface protocols. The information on the chart are taken from vendor submissions that used the same exact hardware configurations for both NFS and CIFS/SMB protocol SPECsfs2008 benchmark submissions.

There are generally two charts I show in our CIFS/SMB vs. NFS analysis, the one above and another that shows a ops/sec per spindle count analysis for all NFS and CIFS/SMB submissions.  Both have historically indicated that CIFS/SMB had an advantage. The one above shows the total number of NFS or CIFS/SMB operations per second on the two separate axes and provides a linear regression across the data. The above shows that, on average, the CIFS/SMB protocol provides about 40% more (~36.9%) operations per second than NFS protocol does with the same hardware configuration.

However, there are a few caveats about this and my other CIFS/SMB vs. NFS comparison charts:

  • The SPECsfs2008 organization has informed me (and posted on their website) that  CIFS[/SMB] and NFS are not comparable.  CIFS/SMB is a stateful protocol and NFS is stateless and the corresponding commands act accordingly. My response to them and my readers is that they both provide file access, to a comparable set of file data (we assume, see my previous post on What’s wrong with SPECsfs2008) and in many cases today, can provide access to the exact same file, using both protocols on the same storage system.
  • The SPECsfs2008 CIFS/SMB benchmark does slightly more read and slightly less write data operations than their corresponding NFS workloads. Specifically, their CIFS/SMB workload does 20.5% and 8.6% READ_ANDX and WRITE_ANDX respectively CIFS commands vs. 18% and 9% READ and WRITE respectively NFS commands.
  • There are fewer CIFS/SMB benchmark submissions than NFS and even fewer with the same exact hardware (only 13). So the statistics comparing the two in this way must be considered preliminary, even though the above linear regression is very good (R**2 at ~0.98).
  • Many of the submissions on the above chart are for smaller systems. In fact 5 of the 13 submissions were for storage systems that delivered less than 20K NFS ops/sec which may be skewing the results and most of which can be seen above bunched up around the origin of the graph.

And all of this would all be wonderfully consistent if not for a recent benchmark submission by NetApp on their FAS8020 storage subsystem.  For once NetApp submitted the exact same hardware for both a NFS and a CIFS/SMB submission and lo and behold they performed better on NFS (110.3K NFS ops/sec) than they did on CIFS/SMB (105.1K CIFS ops/sec) or just under ~5% better on NFS.

Luckily for the chart above this was a rare event and most others that submitted both did better on CIFS/SMB. But I have been proven wrong before and will no doubt be proven wrong again. So I plan to update this chart whenever we get more submissions for both CIFS/SMB and NFS with the exact same hardware so we can see a truer picture over time.

For those with an eagle eye, you can see NetApp’s FAS8020 submission as the one below the line in the first box above the origin which indicates they did better on NFS than CIFS/SMB.

Comments?

~~~~

The complete SPECsfs2008  performance report went out in SCI’s March 2014 newsletter.  But a copy of the report will be posted on our dispatches page sometime next quarter (if all goes well).  However, you can get the latest storage performance analysis now and subscribe to future free newsletters by just using the signup form above right.

Even more performance information on NFS and CIFS/SMB protocols, including our ChampionCharts™ for file storage can be found in  SCI’s recently (March 2014) updated NAS Buying Guide, on sale from our website.

As always, we welcome any suggestions or comments on how to improve our SPECsfs2008 performance reports or any of our other storage performance analyses.

What’s wrong with SPECsfs2008?

Posted on by Ray in CIFS/SMB throughput, Clustered storage, desktop virtualization, File Storage, NFS throughput, SPECsfs, SPECsfs2008, Storage, Storage performance, Storage virtualization, System effectiveness

I have been analyzing SPECsfs results now for almost 7 years now and I feel that maybe it’s time for me to discuss some of the t problems with SPECsfs2008 today that should be fixed in the next SPECsfs20xx whenever that comes out.

CIFS/SMB

First and foremost, for CIFS SMB 1 is no longer pertinent to today’s data center. The world of Microsoft has moved on to SMB 2 mostly and are currently migrating to SMB 3.  There were plenty of performance fixes in the last years SMB 3.0 release which would be useful to test with current storage systems. But I would be even be somewhat happy with SMB2 if that’s all I can hope for.

My friends at Microsoft would consider me remiss if I didn’t mention that since SMB 2 they no longer call it CIFS and have moved to SMB. SPECsfs should follow this trend. I have tried to use CIFS/SMB in my blog posts/dispatches as a step in this direction mainly because SPEC continues to use CIFS and Microsoft wants me to use SMB.

In my continuing quest to better compare different protocol performance I believe it would be useful to insure that the same file size distributions are used for both CIFS and NFS benchmarks. Although the current Users Guide discusses some file size information for NFS it is silent when it comes to CIFS. I have been assuming that they were the same because of lack of information but this would be worthy to have confirmed in documentation.

Finally for CIFS, it would be very useful if there could be a closer approximation of the same amount of data transfers that are done for NFS.  This is a nit but when I compare CIFS to NFS storage system results there is a slight advantage to NFS because NFS’s workload definition doesn’t do as much reading as CIFS. In contrast, CIFS has slightly less file data write activity than the NFS benchmark workload. Having them be exactly the same would help in any (unsanctioned) comparisons.

NFSv3

As for NFSv3, although NFSv4 has been out for more than 3 years now, it has taken a long time to be widely adopted. However, these days there seems to be more client and storage support coming online every day and maybe this would be a good time to move on to NFSv4.

The current NFS workloads, while great for the normal file server activities, have not kept pace with much of how NFS is used today especially in virtualized environments. As far as I can tell under VMware NFS data stores don’t do a lot of meta-data operations and do an awful lot more data transfers than normal file servers do. Similar concerns apply to NFS used for Oracle or other databases. Unclear how one could incorporate a more data intensive workload mix into the standard SPECsfs NFS benchmark but it’s worthy of some thought. Perhaps we could create a SPECvms20xx benchmark that would test these types of more data intensive workloads.

For both NFSv3 and CIFs benchmarks

Both the NFSv3 and CIFS benchmarks typically report [throughput] ops/sec. These are a mix of all the meta-data activities and the data transfer activities.  However, I think many storage customers and users would like a finer view of system performance. .

I have often been asked just how many files a storage system actually support. This depends of course on the workload and file size distributions but SPECsfs already defines this. As a storage performance expert, I would also like to know how much data transfer can a storage system support in MB/sec read and written.  I believe both of these metrics can be extracted from the current benchmark programs with a little additional effort. Probably another half dozen metrics that would be useful maybe we could sit down and have an open discussion of what these might be.

Also the world has changed significantly over the last 6 years and SSD and flash has become much more prevalent. Some of your standard configuration tables could be better laid out to insure that readers understand just how much DRAM, flash, SSDs and disk drives are in a configuration.

Beyond file NAS

Going beyond SPECsfs there is a whole new class of storage, namely object storage where there are no benchmarks available. I would think now that Amazon S3 and Openstack Cinder are well defined and available that maybe a new set of SPECobj20xx benchmarks would be warranted. I believe with the adoption of software defined data centers, object storage may become the storage of choice over the next decade or so. If that’s the case then having some a benchmark to measure object storage performance would help in its adoption. Much like the original SPECsfs did for NFS.

Then there’s the whole realm of server SAN or (hyper-)converged storage which uses DAS inside a cluster of compute servers to support block and file services. Not sure exactly where this belongs but NFS is typically the first protocol of choice for these systems and having some sort of benchmark configuration that supports converged storage would help adoption of this new type of storage as well.

I think thats about it for now but there’s probably a whole bunch more that I am missing out here.

Comments?

SpecSFS2008 results NFS throughput vs. flash size – Chart of the Month

Posted on by Ray in File Storage, NFS throughput, SPECsfs, SPECsfs2008, SSD storage, Storage performance

Scatter plot with SPECsfs2008 NFS throughput results against flash size, SSD, NFS thoughputThe above chart was sent out in our December newsletter and represents yet another attempt to understand how flash/SSD use is impacting storage system performance. This chart’s interesting twist is to try to categorize the use of flash in hybrid (disk-SSD) systems vs. flash-only/all flash storage systems.

First, we categorize SSD/Flash-only (blue diamonds on the chart) systems as any storage system that has as much or more flash storage capacity than SPECsfs2008 exported file system capacity. While not entirely true, there is one system that has ~99% of their exported capacity in flash, it is a reasonable approximation.  Any other system that has some flash identified in it’s configuration is considered a Hybrid SSD&Disks (red boxes on the chart) system.

Next, we plot the system’s NFS throughput on the vertical axis and the system’s flash capacity (in GB) on the horizontal axis. Then we charted a linear regression for each set of data.

What troubles me with this chart is that hybrid systems are getting much more NFS throughput performance out of their flash capacity than flash-only systems. One would think that flash-only systems would generate more throughput per flash GB than hybrid systems because of the slow access times from disk. But the data shows this is wrong?!

We understand that NFS throughput operations are mostly metadata file calls and not data transfers so one would think that the relatively short random IOPS would favor flash only systems. But that’s not what the data shows.

What the data seems to tell me is that judicious use of flash and disk storage in combination can be better than either alone or at least flash alone.  So maybe those short random IOPS should be served out of SSD and the relatively longer, more sequential like data access (which represents only 28% of the operations that constitute NFS throughput) should be served out of disk.  And as the metadata for file systems is relatively small in capacity, this can be supported with a small amount of SSD, leveraging that minimal flash capacity for the greater good (or more NFS throughput).

I would be remiss if I didn’t mention that there are relatively few (7) flash-only systems in the SPECsfs2008 benchmarks and the regression coefficient is very poor (R**2=~0.14), which means that this could change substantially with more flash-only submissions. However, it’s looking pretty flat from my perspective and it would take an awful lot of flash-only systems showing much higher NFS throughput per flash GB to make a difference in the regression equation

Nonetheless, I am beginning to see a pattern here in that SSD/Flash is good for some things and disk continues to be good for others. And smart storage system developers will do good to realize this fact.  Also, as a side note, I am beginning to see some rational why there aren’t more flash-only SPECsfs2008 results.

Comments?

~~~~

The complete SPECsfs2008 performance report went out in SCI’s December 2013 newsletter.  But a copy of the report will be posted on our dispatches page sometime this quarter (if all goes well).  However, you can get the latest storage performance analysis now and subscribe to future free newsletters by just using the signup form above right.

Even more performance information and ChampionCharts for NFS and CIFS/SMB storage systems are also available in SCI’s NAS Buying Guide, available for purchase from  website.

As always, we welcome any suggestions or comments on how to improve our SPECsfs2008  performance reports or any of our other storage performance analyses.

 

Latest SPECsfs2008 results NFS vs. CIFS – chart-of-the-month

Posted on by Ray in CIFS/SMB throughput, Ethernet, File Storage, Networking, NFS throughput, SPECsfs, Storage, Storage performance
SCISFS121227-010(001) (c) 2013 Silverton Consulting, Inc. All Rights Reserved
SCISFS121227-010(001) (c) 2013 Silverton Consulting, Inc. All Rights Reserved

We return to our perennial quest to understand file storage system performance and our views on NFS vs. CIFS performance.  As you may recall, SPECsfs2008 believes that there is no way to compare the two protocols because

  • CIFS/SMB is “statefull” and NFS is “state-less”
  • The two protocols are issuing different requests.

Nonetheless, I feel it’s important to go beyond these concerns and see if there is any way to assess the relative performance of the two protocols.  But first a couple of caveats on the above chart:

  • There are 25 CIFS/SMB submissions and most of them are for SMB environments vs. 64 NFS submissions which are all over the map
  • There are about 12 systems that have submitted exact same configurations for CIFS?SMB and NFS SPECsfs2008 benchmarks.
  • This chart does not include any SSD or FlashCache systems, just disk drive only file storage.

All that being said, let us now see what the plot has to tell us. First the regression line is computed by Excel and is a linear regression.  The regression coefficient for CIFS/SMB is much better at 0.98 vs NFS 0.80. But this just means that their is a better correlation between CIFS/SMB throughput operations per second to the number of disk drives in the benchmark submission than seen in NFS.

Second, the equation and slope for the two lines is a clear indicator that CIFS/SMB provides more throughput operations per second per disk than NFS. What this tells me is that given the same hardware, all things being equal the CIFS/SMB protocol should perform better than NFS protocol for file storage access.

Just for the record the CIFS/SMB version used by SPECsfs2008 is currently SMB2 and the NFS version is NFSv3.  SMB3 was just released last year by Microsoft and there aren’t that many vendors (other than Windows Server 2012) that support it in the field yet and SPECsfs2008 has yet to adopt it as well.   NFSv4 has been out now since 2000 but SPECsfs2008 and most vendors never adopted it.  NFSv4.1 came out in 2010 and still has little new adoption.

So these results are based on older, but current versions of both protocols available in the market today.

So, given all that, if I had an option I would run CIFS/SMB protocol for my file storage.

Comments?

More information on SPECsfs2008 performance results as well as our NFS and CIFS/SMB ChampionsCharts™ for file storage systems can be found in our NAS Buying Guide available for purchase on our web site.

~~~~

The complete SPECsfs2008 performance report went out in SCI’s December newsletter.  But a copy of the report will be posted on our dispatches page sometime this month (if all goes well).  However, you can get the latest storage performance analysis now and subscribe to future free newsletters by just using the signup form above right.

As always, we welcome any suggestions or comments on how to improve our SPECsfs2008  performance reports or any of our other storage performance analyses.

 

NFS ChampionsChart™ – chart-of-the-month

Posted on by Ray in NFS throughput, ORT, SPECsfs, SPECsfs2008, Storage performance
SCISFS120926-001, Q4-2012 NFS ChampionsChart(tm) (c) 2012 Silverton Consulting, Inc., All Rights Reserved
SCISFS120926-001, Q4-2012 NFS ChampionsChart(tm) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

We had no new performance data to report on in our September StorInt™ newsletter so we decided to publish our NAS Buying Guide ChampionsCharts™.   The chart above is our Q4-2102 NFS ChampionsChart which shows the top performing NFS storage systems from published SPECsfs2008 benchmark results available at the time.

We split up all of our NAS and SAN ChampionsCharts into four quadrants: Champions, Sprinters, Marathoners and Slowpokes.  We feel that storage Champions represent the best overall performers,  Sprinters have great response time but lack in transaction throughput as compared to storage Champions, Marathoners have good transaction throughput but are defficient in responsiveness and Slowpokes need to go back to the drawing board because they suffer both poor transaction throughput and responsiveness.

You may notice that there are two categories of systems identified in the NFS Champions Quadrant above.  These represent the more “normal” NAS systems (numbered 1-7) such as integrated systems and NAS gateways with SAN storage behind them vs. the more caching oriented, NAS systems (denoted with letters A-E) which have standalone NAS systems behind them.

In our Dispatch we discuss the top 3 NAS Champions in the integrated and gateway category which include:

  1. NetApp FAS6080 – although a couple of generations back, the FAS6080 did remarkably well for its complement of hardware.
  2. Huawei Symantec Oceanspace N8500 Clustered NAS – this product did especially well for its assortment of hardware in response time and not necessarily that great in NFS throughput but still respectable.
  3. EMC Celerra VG8, 2 DM and VMAX hardware – similar to number one above, a relatively modest amount of cache and disks but seemed to perform relatively well.

One negative to all our ChampionsCharts is that they depend on audited, published performance data which typically lag behind recent product introductions.  As evidence of this the FAS6080 and Celerra VG8 listed above are at least a generation or two behind current selling systems.  I am not as familiar with the Huawei systems but it may very well be a generation or two behind current selling products.

As for our rankings, this is purely subjective but our feeling is that transaction performance comes first with responsiveness a close second. For example in the above ranking Huawei’s system had the best overall responsiveness but relatively poorer transaction performance than any of the other Champions.  Nonetheless as the best in responsiveness, we felt it deserved a number two in our Champions list.

The full Champions quadrants for the NFS and CIFS/SMB ChampionsCharts are detailed in our NAS Buying Guide available for purchase on our website (please see NAS buying guide page).  The dispatch that went out with our September newsletter also detailed the top 3 CIFS/SMB Champions.

~~~~

The complete SPECsfs2008 performance report with both NFS and CIFS/SMB ChampionsCharts went out in SCI’s September newsletter.  But a copy of the report will be posted on our dispatches page sometime this month (if all goes well).  However, you can get the latest storage performance analysis now and subscribe to future free newsletters by just using the signup form above right.

As always, we welcome any suggestions or comments on how to improve our SPECsfs2008 performance analysis or any of our other storage performance analyses.


SPECsfs2008 NFS SSD/NAND performance, take two – chart-of-the-month

Posted on by Ray in File Storage, NFS throughput, SPECsfs, SPECsfs2008, SSD storage, Storage performance

SCISFS120623-010(002) (c) 2012 Silverton Consulting, Inc. All Rights Reserved

For some time now I have been experimenting with different approaches to normalize IO activity (in the chart above its NFS throughput operations per second) for systems that use SSDs or Flash Cache.  My previous attempt  (see prior SPECsfs2008 chart of the month post) normalized base on GB of NAND capacity used in a submission.

I found the previous chart to be somewhat lacking so this quarter I decided to use SSD device and/or Flash Cache card count instead.  This approach is shown in the above chart. Funny thing, although the rankings were exactly the same between the two charts one can see significant changes in the magnitudes achieved, especially in the relative values, between the top 2 rankings.

For example, in the prior chart Avere FXT 3500 result still came in at number one but whereas here they achieved ~390K NFS ops/sec/SSD on the prior chart they obtained ~2000 NFS ops/sec/NAND-GB. But more interesting was the number two result. Here the NetApp FAS6240 with 1TB Flash Cache Card achieved ~190K NFS ops/sec/FC-card but on the prior chart they only hit ~185 NFS ops/sec/NAND-GB.

That means on this version of the normalization the Avere is about 2X more effective than the NetApp FAS6240 with 1TB FlashCache card but in the prior chart they were 10X more effective in ops/sec/NAND-GB. I feel this is getting closer to the truth but not quite there yet.

We still have the problem that all the SPECsfs2008 submissions that use SSDs or FlashCache also have disk drives as well as (sometimes significant) DRAM cache in them.  So doing a pure SSD normalization may never suffice for these systems.

On the other hand, I have taken a shot at normalizing SPECsfs2008 performance for SSDs-NAND, disk devices and DRAM caching as one dimension in a ChampionsChart™ I use for a NAS Buying Guide, for sale on my website.  If your interested in seeing it, drop me a line, or better yet purchase the guide.

~~~~

The complete SPECsfs2008 performance report went out in SCI’s June newsletter.  But a copy of the report will be posted on our dispatches page sometime next month (if all goes well).  However, you can get the SPECsfs2008 performance analysis now and subscribe to future free newsletters by just using the signup form above right.

For a more extensive discussion of current NAS or file system storage performance covering SPECsfs2008 (Top 20) results and our new ChampionsChart™ for NFS and CIFS storage systems, please see SCI’s NAS Buying Guide available from our website.

As always, we welcome any suggestions or comments on how to improve our analysis of SPECsfs2008 results or any of our other storage performance analyses.