A tale of two AFAs: EMC DSSD D5 & Pure Storage FlashBlade

There’s been an ongoing debate in the analyst community about the advantages of software only innovation vs. hardware-software innovation (see Commodity hardware loses again and Commodity hardware always loses posts). Here is another example where two separate companies have turned to hardware innovation to take storage innovation to the next level.

DSSD D5 and FlashBlade

DSSD-d5Within the last couple of weeks, two radically different AFAs were introduced. One by perennial heavyweight EMC with their new DSSD D5 rack scale flash system and the other by relatively new comer Pure Storage with their new FlashBlade storage system.FB

These two arrays seem to be going after opposite ends of the storage market: the 5U DSSD D5 is going after both structured and unstructured data that needs ultra high speed IO access (<100µsec) times and the 4U FlashBlade going after more general purpose unstructured data. And yet the two have have many similarities at least superficially.
Continue reading “A tale of two AFAs: EMC DSSD D5 & Pure Storage FlashBlade”

SCI SPECsfs2008 NFS throughput per node – Chart of the month

As SPECsfs2014 still only has (SPECsfs sourced) reference benchmarks, we have been showing some of our seldom seen SPECsfs2008 charts, in our quarterly SPECsfs performance reviews. The above chart was sent out in last months Storage Intelligence Newsletter and shows the NFS transfer operations per second per node.

In the chart, we only include NFS SPECsfs2008 benchmark results with configurations that have more than 2 nodes and have divided the maximum NFS throughput operations per second achieved by the node counts to compute NFS ops/sec/node.

HDS VSP G1000 with an 8 4100 file modules (nodes) and HDS HUS (VM) with 4 4100 file modules (nodes) came in at #1 and #2 respectively, for ops/sec/node, each attaining ~152K NFS throughput operations/sec. per node. The #3 competitor was Huawei OceanStor N8500 Cluster NAS with 24 nodes, which achieved ~128K NFS throughput operations/sec./node. At 4th and 5th place were EMC  VNX VG8/VNX5700 with 5 X-blades and Dell Compellent FS8600 with 4 appliances, each of which reached ~124K NFS throughput operations/sec. per node. It falls off significantly from there, with two groups at ~83K and ~65K NFS ops/sec./node.

Although not shown above, it’s interesting that there are many well known scale-out NAS solutions in SPECsfs2008 results with over 50 nodes that do much worse than the top 10 above, at <10K NFS throughput ops/sec/node. Fortunately, most scale-out NAS nodes cost quite a bit less than the above.

But for my money, one can be well served with a more sophisticated, enterprise class NAS system which can do >10X the NFS throughput operations per second per node than a scale-out systm. That is, if you don’t have to deploy 10PB or more of NAS storage.

More information on SPECsfs2008/SPECsfs2014 performance results as well as our NFS and CIFS/SMB ChampionsCharts™ for file storage systems can be found in our just updated NAS Buying Guide available for purchase on our web site.



The complete SPECsfs2008 performance report went out in SCI’s September newsletter.  A copy of the report will be posted on our dispatches page sometime this quarter (if all goes well).  However, you can get the latest storage performance analysis now and subscribe to future free monthly newsletters by just using the signup form above right.

As always, we welcome any suggestions or comments on how to improve our SPECsfs  performance reports or any of our other storage performance analyses.


Latest SPECsfs2008 results – chart of the month

(SCISFS110318-003) (c) 2011 Silverton Consulting, Inc., All Rights Reserved
(SCISFS110318-003) (c) 2011 Silverton Consulting, Inc., All Rights Reserved

The above chart comes from our last month’s newsletter on the lastest SPECsfs2008 file system performance benchmark results and depicts a scatter plot of system NFS throughput operations per second versus the number of disk drives in the system being tested.  We eliminate from this chart any system that makes use of Flash Cache/SSDS or any other performance use of NAND (See below on why SONAS was still included).

One constant complaint of benchmarks is that system vendors can just throw hardware at the problem to attain better results.   The scatter plot above is one attempt to get to the truth in that complaint.

The regression equation shows that NFS throughput operations per second = 193.68*(number of disk drives) + 23834. The regression coefficient (R**2) is 0.87 which is pretty good but not exactly perfect. So given these results, one would have to conclude there is some truth in the complaint but it doesn’t tell the whole story. (Regardless of how much it pains me to admit it).

A couple of other interesting things about the chart:

  • IBM released a new SONAS benchmark with 1975 disks, with 16 interface and 10 storage nodes to attain its 403K NFS ops/second. Now the SONAS had 512GB of NV Flash, which I assume is being used for redundancy purposes on writes and not as a speedup for read activity. Also the SONAS system complex had over 2.4TB of cache (includes the NV Flash).  So there was a lot of cache to throw at the problem.
  • HP BL860c results were from a system with 1480 drives, 4 nodes (blades) and ~800GB of cache to attain its 333KNFS ops/second.

(aside) Probably need to do a chart like this with amount of cache as the x variable (/aside)

In the same report we talked about the new #1 performing EMC VNX Gateway  that used 75TB of SAS-SSDs and 4 VNX5700’s as its backend. It was able to reach 497K NFS ops/sec.   It doesn’t show up on this chart because of its extensive use of SSDs.  But according to the equation above one would need to use ~2500 disk drives to attain similar performance without SSDS and I believe, a whole lot of cache.


The full performance dispatch will be up on our website after the middle of next month (I promise) but if one is interested in seeing it sooner sign up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send the current issue along with download instructions for this and other reports.  If you need an even more in-depth analysis of NAS system performance please consider purchasing SCI’s NAS Buying Guide also available from our website.

As always, we welcome any constructive suggestions on how to improve any of our storage performance analysis.



IBM Scale out NAS (SONAS) v1.1.1

IBM SONAS from IBM's Flickr stream (c) IBM
IBM SONAS from IBM's Flickr stream (c) IBM

We have discussed other scale out NAS products on the market such as Symantec’s FileStoreIBRIX reborn as HP networked storage, and why SO/CFS, why now (scale out/cluster file systems) in previous posts but haven’t talked about IBM’s highend scale out NAS (SONAS) product before. There was an announcement yesterday of a new SONAS version so thought it an appropriate time to cover it.

As you may know SONAS packages up IBM’s well known GPFS system services and surrounds it with pre-packaged hardware and clustering software that supports a high availability cluster of nodes serving native CIFS and NFS clients.

One can see SONAS is not much to look at from the outside but internally it comes with three different server components:

  • Interface nodes – which provide native CIFS, NFS and now with v1.1.1 HTTP interface protocols to the file store.
  • Storage nodes – which supply backend storage device services.
  • Management nodes – which provide for administration of the SONAS storage system.

The standard SONAS system starts with a fully integrated hardware package within one rack with 2-management nodes, 2- to 6-interface nodes, 2-storage pods (one storage pod consists of of 2-storage nodes and 60 to 240 attached disk drives).  The starter system can then be expanded with either a single interface rack with up to 30 interface nodes and/or multiple storage racks with 2 storage pods in each rack.

With v1.1.1, a new hardware option has been provided, specifically the new IBM SONAS gateway for IBM’s XIV storage.  With this new capability SONAS storage nodes can now be connected to an IBM XIV storage subsystem using 8GFC interfaces through a SAN switch.

Some other new functionality released in SONAS V1.1.1 include:

  • New policy engine – used for internal storage tiering and for external/hierarchical storage through IBM’s Tivoli Storage Managere (TSM) product. Recall that SONAS supports both SAS and SATA disk drives and now one can use policy management to migrate files between internal storage tiers.  Also, with the new TSM interface, data can now be migrated out of SONAS and onto tape or any of the other over 600 storage devices supported by TSM’s Hierarchical Storage Management (HSM) product.
  • Asynch replication – used for disaster recovery/business continuance.  SONAS uses standard Linux based RSYNC capabilities to replicate file systems from one SONAS cluster to another cluster.  SONAS replication only copies changed portions of files within file systems being replicated and uses SSH data transfer to encrypt data-in-flight between the two SONAS systems.

There were some other minor enhancements for this announcement namely, higher capacity SAS drive support (now 600GB), using NIS authentication, increased cache per interface node (now up to 128GB), and the already mentioned new HTTP support.

In addition, IBM stated that a single interface node can pump out 900MB/sec (out of cache) and 6 interface nodes can sustain over 5GB/sec (presumably also from cache).  SONAS can currently scale up to 30 interface nodes but this doesn’t appear to be an architectural limitation but rather just what has been validated by IBM.

Can’t wait to see this product show up in SPECsfs 2008 performance benchmarks to see how it compares to other SO and non-SA file system products.

More cloud storage gateways come out

Strange Clouds by michaelroper (cc) (from Flickr)
Strange Clouds by michaelroper (cc) (from Flickr)

Multiple cloud storage gateways either have been announced or are coming out in the next quarter or so. We have talked before about Nasuni’s file cloud storage gateway appliance, but now that more are out one can have a better appreciation of the cloud gateway space.


Last week I was talking with StorSimple that just introduced their cloud storage gateway which provides a iSCSI block protocol interface to cloud storage with an onsite data caching.  Their appliance offers a cloud storage cache residing on disk and/or optional flash storage (SSDs) and provides iSCSI storage speeds for highly active working set data residing on the cache or cloud storage speeds for non-working set data.

Data is deduplicated to minimize storage space requirements.  In addition data sent to the cloud is compressed and encrypted. Both deduplication and compression can reduce WAN bandwidth requirements considerably.    Their appliance also offers snapshots and “cloud clones”.  Cloud clones are complete offsite (cloud) copies of a LUN which can then be maintained in synch with the gateway LUNs by copying daily change logs and applying the logs.

StorSimple works with Microsoft’s Azure, AT&T, EMC Atmos, Iron Mountan and Amazon’s S3 cloud storage providers.   A single appliance can support multiple cloud storage providers segregated on a LUN basis.  Although how cross-LUN deduplication works across multiple cloud storage providers was not discussed.

Their product can be purchased as a hardware appliance with a few 100GB of NAND/Flash storage up to a 150TB of SATA storage.  It also can be purchased as a virtual appliance at lower cost but also much lower performance.


In addition to StorSimple, I have talked with Cirtas which has yet to completely emerge from stealth but what’s apparent from their website is that the Cirtas appliance provides “storage protocols” to server systems, and can store data directly on storage subsystems or on cloud storage.

Storage protocols could mean any block storage protocol which could be FC and/or iSCSI but alternatively, it might mean file protocols I can’t be certain.  Having access to independent, standalone storage arrays may mean that  clients can use their own storage as a ‘cloud data cache’.  Unclear how Cirtas talks to their onsite backend storage but presumably this is FC and/or iSCSI as well.  And somehow some of this data is stored out on the cloud.

So from our perspective it looks somewhat similar to StorSimple with the exception that it uses external storage subsystems for its cloud data cache for Cirtas vs. internal storage for StorSimple.  Few other details were publicly available as this post went out.


Although I have not talked directly with Panzura they seem to offer a unique form of cloud storage gateway, one that is specific to some applications.  For example, the Panzura SharePoint appliance actually “runs” part of the SharePoint application (according to their website) and as such, can better ascertain which data should be local versus stored in the cloud.  It seems to have  both access to cloud storage as well as local independent storage appliances.

In addition to a SharePoint appliance they offer a “”backup/DR” target that apparently supports NDMP, VTL, iSCSI, and NFS/CIFS protocols to store (backup) data on the cloud. In this version they show no local storage behind their appliance by which I assume that backup data is only stored in the cloud.

Finally, they offer a “file sharing” appliance used to share files across multiple sites where files reside both locally and in the cloud.  It appears that cloud copies of shared files are locked/WORM like but I can’t be certain.  Having not talked to Panzura before, much of their product is unclear.

In summary

We now have both a file access and at least one iSCSI block protocol cloud storage gateway, currently available, publicly announced, i.e., Nasuni and StorSimple.  Cirtas, which is in the process of coming out, will support a “storage protocol” access to cloud storage and Panzura offers it all (SharePoint direct, iSCSI, CIFS, NFS, VTL & NDMP cloud storage access protocols).  There are other gateways just focused on backup data, but I reserve the term cloud storage gateways for those that provide some sort of general purpose storage or file protocol access.

However, Since last weeks discussion of eventual consistency, I am becoming a bit more concerned about cloud storage gateways and their capabilities.  This deserves some serious discussion at the cloud storage provider level and but most assuredly, at the gateway level.  We need some sort of generic statement that says they guarantee immediate consistency for data at the gateway level even though most cloud storage providers only support “eventual consistency”.  Barring that, using cloud storage for anything that is updated frequently would be considered unwise.

If anyone knows of another cloud storage gateway I would appreciate a heads up.  In any case, the technology is still young yet and I would say that this isn’t the last gateway to come out but it feels like these provide coverage for just about any file or block protocol one might use to access cloud storage.

SPECsfs2008 CIFS vs. NFS results – chart of the month

SPECsfs(R) 2008 CIFS vs. NFS 2010Mar17
SPECsfs(R) 2008 CIFS vs. NFS 2010Mar17

We return now to our ongoing quest to understand the difference between CIFS and NFS performance in the typical data center.  As you may recall from past posts and our newsletters on this subject, we had been convinced that in SPECsfs 2008 CIFS had almost 2X the throughput of NFS in SPECsfs 2008 benchmarks.  Well as you can see from this updated chart this is no longer true.

Thanks to EMC for proving me wrong (again).  Their latest NFS and CIFS result utilized a NS-G8 Celerra gateway server in front of V-Max backend using SSDs and FC disks. The NS-G8 was the first enterprise class storage subsystem to release both a CIFS and NFS SPECsfs 2008 benchmark.

As you can see from the lower left quadrant all of the relatively SMB level systems (under 25K NFS throughput ops/sec) showed a consistent pattern of CIFS throughput being ~2X NFS throughput.  But when we added the Celerra V-Max combination to the analysis it brought the regression line down considerably and now the equation is:

CIFS throughput = 0.9952 X NFS throughput + 10565, with a R**2 of 0.96,

what this means is that CIFS and NFS throughput are roughly the same now.

When I first reported the relative advantage of CIFS over NFS throughput in my newsletter I was told that you cannot compare the two results mainly because NFS was “state-less” and CIFS was “state-full” and a number of other reasons (documented in the earlier post and in the newsletter).  Nonetheless, I felt that it was worthwhile to show the comparison because at the end of the day whether some file happens to be serviced by NFS or CIFS may not matter to the application/user, it should matter significantly to the storage administrator/IT staff.  By showing the relative performance of each we were hoping to help IT personnel to decide between using CIFS or NFS storage.

Given the most recent results, it seems that the difference in throughput is not that substantial irregardless of their respective differences.  Of course more data will help. There seems to be a wide gulf between the highest SMB submission and the EMC enterprise class storage that should be filled out.  As Celerra V-Max is the only enterprise NAS to submit both CIFS and NFS benchmarks there could still be many surprises in store. As always, I would encourage storage vendors to submit both NFS and CIFS benchmarks for the same system so that we can see how this pattern evolves over time.

The full SPECsfs 2008 report should have went out to our newsletter subscribers last month but I had a mistake with the link.  The full report will be delivered with this months newsletter along with a new performance report on Exchange Solution Review Program and storage announcement summaries.  In addation, a copy of the SPECsfs report will be up on the dispatches page of our website later next month. However, you can get this information now and subscribe to future newsletters to receive future full reports even earlier, just email us at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter.

As always, we welcome any suggestions on how to improve our analysis of SPECsfs or any of our other storage system performance results.

Ibrix reborn as HP X9000 Network Storage

HP X9000 appliances pictures from HP(c) presentation
HP X9000 appliances pictures from HP(c) presentation

On Wednesday 4 November, HP announced a new network storage system based on the Ibrix Fusion file system called the X9000. Three versions were announced:

  • X9300 gateway appliance which can be attached to SAN storage (HP EVA, MSA, P4000, or 3rd party SAN storage) and provides scale out file system services
  • X9320 performance storage appliance which includes a fixed server gateway and storage configuration in one appliance targeted at high performance application environments
  • X9720 extreme storage appliance using blade servers for file servers and separate storage in one appliance but can be scaled up (with additional servers and storage) as well as out (by adding more X9720 appliances) to target more differentiated application environments

The new X9000 appliances support a global name space of 16PB by adding additional X9000 network storage appliances to a cluster. The X9000 supports a distributed metadata architecture which allows the system to scale performance by adding more storage appliances.

X9000 Network Storage appliances

With the X9300 gateway appliance, storage can be increased by adding more SAN arrays. Presumably, multiple gateways can be configured to share the same SAN storage creating a highly available file server node. The gateway can be configured to support the following Gige, 10Gbe, and/or QDR (40gb/s) Infiniband interfaces for added throughput.

The Extreme appliance (X9720) comes with 82 TB in the starting configuration and storage can be increased by in 82TB raw capacity block increments (7u-1/2rack wide/35*2 drive enclosures + 1-12 drive tray for each capacity block) up to a maximum of 656TB in two rack (42U) configuration. Capacity blocks are connected to the file servers via 3gb SAS, and the X9720 includes a SAS switch as well as two ProCurve 10Gbe ethernet switches. Also, file system performance can be scaled by independently adding performance blocks, essentially C-class HP blade servers. The starter configuration includes 3 performance blocks (blades) but up to 8 can be added to one X9720 appliance.

For the X9320 scale out appliance, performance and capacity are fixed in a 12U rack mountable appliance that includes 2-X9300 gateways and 21.7TB SAS or 48TB SATA raw storage per appliance. The X9320 comes with either GigE or 10Gbe attachments for added performance. The 10Gbe version supports up to 700MB/s raw potential throughput per gateway (node).

X9000 capabilities

All these systems have separate, distinct internal-like storage devoted to O/S, file server software and presumably metadata services. In the X9300 and X9320 storage, this internal storage is packaged in the X9300 gateway server itself. In the X9720, presumably this internal storage is configured via storage blades in the blade server cabinet which would need to be added with each performance block.

All X9000 storage is now based on the Fusion file system technology acquired by HP from Ibrix, an acquisition which closed this summer. Ibrix’s Fusion file system provided a software only implementation of a distributed (or segmented) metadata serviced file system which allowed the product to scale out performance and/or capacity, independently by adding appropriate hardware.

HP’s X9000 supports both NFS and CIFS interfaces. Moreover, a\Advanced storage features such as continuous remote file replication, snapshot, high availability (with two or more gateways/performance blocks), and automated policy driven data tiering also come with the X9000 Network Storage system. In additition, file data is automatically re-distributed across all nodes in X9000 appliance to ballance storage performance across nodes. Every X9000 Network Storage system requires a separate management server to manage the X9000 Network Storage nodes but one server can support the whole 16PB name space.

I like the X9720 and look forward to seeing some performance benchmarks on what it can do. In the past Ibrix never released a SPECsfs(tm) benchmark, presumably because they were a software only solution. But now that HP has instantiated it with top-end hardware there seems to be no excuse to providing benchmark comparisons.

Full disclosure: I have an current contract with another group within HP StorageWorks, not associated with HP X9000 storage.

Symantec's FileStore

Picture of old filing shelves to hold spare parts
Data Storage Device by BinaryApe (cc) (from flickr)
Earlier this week Symantec GA’ed their Veritas FileStore software. This software was an outgrowth of earlier Symantec Veritas Cluster File System and Storage Foundation software which were combined with new frontend software to create scaleable NAS storage.

FileStore is another scale-out, cluster file system (SO/CFS) implemented as NAS head via software. The software runs on a hardened Linux OS and can run on any commodity x86 hardware. It can be configured with up to 16 nodes. Also, it currently supports any storage supported by Veritas Storage Foundation which includes FC, iSCSI, and JBODs. Symantec claims FileStoreo has the broadest storage hardware compatibility list in the industry for a NAS head.

As a NAS head FileStore supports NFS, CIFS, HTTP, and FTP file services and can be configured to support anywhere from under a TB to over 2PB of file storage. Currently FileStore can support up to 200M files per file system, up to 100K file systems, and over 2PB of file storage.

FileStore nodes work in an Active-Active configuration. This means any node can fail and the other, active nodes will take over providing the failed node’s file services. Theoretically this means that in a 16 node system, 15 nodes could fail and the lone remaining node could continue to service file requests (of course performance would suffer considerably).

As part of cluser file system, FileStore support quick failover of active nodes. This can be accomplished in under 20 seconds. In addition, FileStore supports asynchronous replication to other FileStore clusters to support DR and BC in the event of a data center outage.

One of the things that FileStore brings to the table is that as it’s running standard Linux O/S services. This means other Symantec functionality can also be hosted on FileStore nodes. The first Symantec service to be co-hosted with FileStore functionality is NetBackup Advanced Client services. Such a service can have the FileStore node act as a media server for it’s own backup cutting network traffic required to do a backup considerably.

FileStore also supports storage tiering whereby files can be demoted and promoted between storage tiers in the multi-volume file system. Also, Symantec EndPoint Protection can be hosted on a FileStore node provided anti-virus protection completely onboard. Other Symantec capabilities will soon follow to add to the capabilities already available.

FileStore’s NFS performance

Regarding performance, Symantec has submitted a 12 node FileStore system for SPECsfs2008 NFS performance benchmark. I looked today to see if it was published yet and it’s not available but they claim to currently be the top performer for SPECsfs2008 NFS operations. I asked about CIFS and they said they had yet to submit one. Also they didn’t mention what the backend storage looked like for the benchmark, but one can assume it had lots of drives (look to the SPECsfs2008 report whenever it’s published to find out).

In their presentation they showed a chart depicting FileStore performance scaleability. According to this chart, at 16 nodes, the actual NFS Ops performance was 93% of theoretical NFS Ops performance. In my view, scaleability is great but often as you approach some marginal utility as the number of nodes increases, the net performance improvement decreases. The fact that they were able to hit 93% with 16 nodes of what a linear extrapolation of NFS ops performance was from 2 to 8 nodes is pretty impressive. (I asked to show the chart but hadn’t heard back by post time

Pricing and market space

At the lowend, FileStore is meant to compete with Windows Storage Server and would seem to provide better performance and availability versus Windows. At the high end, I am not sure but the competition would be with HP/PolyServe and standalone NAS heads from EMC and NetApp/IBM and others. List pricing is about US$7K/node and that top performing SPECsfs2008 12-node system would set you back about $84K for the software alone (please note that list pricing <> street pricing). You would need to add node hardware and the storage hardware to provide a true apples-to-apples pricing comparison with other NAS storage.

As far as current customers they range from large from the high end (>1PB) E-retailers to SAAS providers (Symantec SAAS offering), and at the low end (<10TB) universities and hospitals. FileStore with it’s inherent scaleability and ability to host storage applications from Symantec on the storage nodes can offer a viable solution to many hard file system problems.

We have discussed scale-out and cluster file systems (SO/CFS) in a prior post (Why SO/CFS, Why Now) so I won’t elaborate on why they are so popular today. But, suffice it to say Cloud and SAAS will need SO/CFS to be viable solutions and everybody is responding to supply that market as it emerges.

Full disclosure: I currently have no active or pending contracts with Symantec.