Symantec's FileStore

Picture of old filing shelves to hold spare parts
Data Storage Device by BinaryApe (cc) (from flickr)
Earlier this week Symantec GA’ed their Veritas FileStore software. This software was an outgrowth of earlier Symantec Veritas Cluster File System and Storage Foundation software which were combined with new frontend software to create scaleable NAS storage.

FileStore is another scale-out, cluster file system (SO/CFS) implemented as NAS head via software. The software runs on a hardened Linux OS and can run on any commodity x86 hardware. It can be configured with up to 16 nodes. Also, it currently supports any storage supported by Veritas Storage Foundation which includes FC, iSCSI, and JBODs. Symantec claims FileStoreo has the broadest storage hardware compatibility list in the industry for a NAS head.

As a NAS head FileStore supports NFS, CIFS, HTTP, and FTP file services and can be configured to support anywhere from under a TB to over 2PB of file storage. Currently FileStore can support up to 200M files per file system, up to 100K file systems, and over 2PB of file storage.

FileStore nodes work in an Active-Active configuration. This means any node can fail and the other, active nodes will take over providing the failed node’s file services. Theoretically this means that in a 16 node system, 15 nodes could fail and the lone remaining node could continue to service file requests (of course performance would suffer considerably).

As part of cluser file system, FileStore support quick failover of active nodes. This can be accomplished in under 20 seconds. In addition, FileStore supports asynchronous replication to other FileStore clusters to support DR and BC in the event of a data center outage.

One of the things that FileStore brings to the table is that as it’s running standard Linux O/S services. This means other Symantec functionality can also be hosted on FileStore nodes. The first Symantec service to be co-hosted with FileStore functionality is NetBackup Advanced Client services. Such a service can have the FileStore node act as a media server for it’s own backup cutting network traffic required to do a backup considerably.

FileStore also supports storage tiering whereby files can be demoted and promoted between storage tiers in the multi-volume file system. Also, Symantec EndPoint Protection can be hosted on a FileStore node provided anti-virus protection completely onboard. Other Symantec capabilities will soon follow to add to the capabilities already available.

FileStore’s NFS performance

Regarding performance, Symantec has submitted a 12 node FileStore system for SPECsfs2008 NFS performance benchmark. I looked today to see if it was published yet and it’s not available but they claim to currently be the top performer for SPECsfs2008 NFS operations. I asked about CIFS and they said they had yet to submit one. Also they didn’t mention what the backend storage looked like for the benchmark, but one can assume it had lots of drives (look to the SPECsfs2008 report whenever it’s published to find out).

In their presentation they showed a chart depicting FileStore performance scaleability. According to this chart, at 16 nodes, the actual NFS Ops performance was 93% of theoretical NFS Ops performance. In my view, scaleability is great but often as you approach some marginal utility as the number of nodes increases, the net performance improvement decreases. The fact that they were able to hit 93% with 16 nodes of what a linear extrapolation of NFS ops performance was from 2 to 8 nodes is pretty impressive. (I asked to show the chart but hadn’t heard back by post time

Pricing and market space

At the lowend, FileStore is meant to compete with Windows Storage Server and would seem to provide better performance and availability versus Windows. At the high end, I am not sure but the competition would be with HP/PolyServe and standalone NAS heads from EMC and NetApp/IBM and others. List pricing is about US$7K/node and that top performing SPECsfs2008 12-node system would set you back about $84K for the software alone (please note that list pricing <> street pricing). You would need to add node hardware and the storage hardware to provide a true apples-to-apples pricing comparison with other NAS storage.

As far as current customers they range from large from the high end (>1PB) E-retailers to SAAS providers (Symantec SAAS offering), and at the low end (<10TB) universities and hospitals. FileStore with it’s inherent scaleability and ability to host storage applications from Symantec on the storage nodes can offer a viable solution to many hard file system problems.

We have discussed scale-out and cluster file systems (SO/CFS) in a prior post (Why SO/CFS, Why Now) so I won’t elaborate on why they are so popular today. But, suffice it to say Cloud and SAAS will need SO/CFS to be viable solutions and everybody is responding to supply that market as it emerges.

Full disclosure: I currently have no active or pending contracts with Symantec.

Chart of the Month

CIFS vs. NFS Throughput Results from SPECsfs(R) 2008 benchmarks
CIFS vs. NFS Throughput Results from SPECsfs(R) 2008 benchmarks
The chart to the left was sent out in last months SCI newsletter and shows the throughput attained by various storage systems when running the SPECsfs(R) 2008 CIFS and NFS benchmarks. The scatter-plot shows data for NAS systems that have published both NFS and CIFS SPECsfs benchmark results for the same storage system. To date (June 2009), only 5 systems have published results for both benchmarks.

The scatter plot clearly shows with a high regression coefficient (.97) that the same system can typically provide over 2.4X the throughput using CIFS as it can using NFS. My friends at SPECsfs would want me to point out that these two benchmarks are not intended to be comparable and I present this with a few caveats in the newsletter:

  • NFS operations are stateless and CIFS are stateful, the distribution of file sizes are different for the two benchmarks, the relative proportions of the respective IO workloads don’t match up exactly (CIFS has more reads and less writes than NFS), and all remaining (non-read/write) operations are completely different for each workload.
  • Most of these results come from the same vendor (Apple) and it’s implementation of NFS or CIFS target support may skew results
  • Only 5 storage systems have published results for these two benchmarks and probably in all honesty do not represent a statistically valid comparison
  • Usually host implementations of CIFS or NFS impact results such as these but for SPECsfs, the benchmark implements each protocol stack. As such, SPECsfs benchmark’s implementation of CIFS or NFS protocols may also skew results

All that being said, I believe it’s an interesting and current fact that for SPECsfs 2008 benchmarks CIFS has 2.4X the throughput of NFS.

In talking with real world customers and vendors on which is better the story seems much more mixed. I heard where one O/S had a much better implementation of the CIFS protocol and as such, customers moved to use CIFS for those systems. I haven’t seen much discussion about storage systems being better or worse on one protocol over the other but it’s certainly probable.

From my perspective any storage admin looking to configure new NAS storage should try out CIFS first to see if it performs well before trying NFS. Given my inclinations, I would probably try out both, but that’s just me.

If you are interested in seeing the full report on latest SPECsfs 2008 results please subscribe to my newsletter by emailing me at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_Newsletter.

Otherwise the full report will be on my website sometime next week.