SCI 2011Dec21 Latest SPECsfs2008* NFS systems results analysis

Dec 212011

We return now to file system performance and analyze the latest SPECsfs® 2008* benchmark results. There have been a number of NFS submissions from NetApp using their Data ONTAP 8.1 with various configurations of clustered FAS6240 (4-, 8-, 12-, 16-, 20- & 24-node) and Avere FXT 3500 in a 44-node configuration. There were no new CIFS results.

Latest SPECsfs2008 results

Column chart showing the top 10 NFS ops/sec subsystem results

Here one can see almost all of the new results starting with the 44-node Avere FXT3500 at #1 and the NetApp 24-node FAS6240 at #2 all the way down to the NetApp 8-node FAS 6240 at #8. This is the first time we have seen NetApp’s clustered operations (C-mode) benchmarks since Spinnaker days on SpecSFS97. It appears to perform much better than before and shows up well against the competition from EMC Isilon and Avere.

Figure 1 Top 10 NFS throughput operations per second

Clustered NAS systems dominate NFS throughput op/sec. In fact there are no, non-clustered systems on this chart. The node counts on this chart range from 140-node EMC Isilon system (#4) down to a 4-X blade (with 1 standby) EMC VNX VG8/VNX5700 (#9).

The other factor not readily apparent in the above is the amount of SSD, Flash Cache or DRAM cache used by these systems. For example, the 44-node Avere FXT 3500 had 6.8TB of DRAM cache (plus 800GB of SSD boot volumes), the 24-node NetApp FAS6240 had 13.5TB of Flash Cache (12TB) and DRAM (1.5TB) cache and the 140-node EMC Isilon system had 6.8TB of DRAM cache and 25TB of backend SSDs. Almost as much DRAM/Flash cache/SSD as capacity supplied by SMB storage systems.

Column chart showing top 10 NFS ORT (response time) results

Figure 2 Top 10 NFS ORT results

Lower is better on the ORT chart. Recall that Overall Response Time (ORT) is an average of response times at a set series of throughput levels during the benchmark run. Similar to throughput results discussed previously, clustered systems also seem to be providing top response time results. Only two systems on this chart, the #1 Alcratech ANX 1500-20 and the #8 Apple system are monolithic systems, with all the rest being scale-out, clustered NAS devices.

Why the 16-node NetApp FAS6240 system placed better than any of its multi-node brethren is a mystery but it could be noise. The “slowest” NetApp C-mode run (20-node system) had a 1.56 msec. ORT whereas the #10 result here had a 1.48 msec. ORT. A difference of only 80 µsec. is almost too small to measure accurately.

Column chart showing top 10 NFS throughput op/sec per disk spindle

Figure 3 Top 10 NFS throughput operations per second per disk drive

Another way to view NFS throughput operations is to normalize it over the number of disk drives. Note, for this chart we exclude any system that uses Flash Cache or data SSDs for data/meta-data storage (Avere systems use SSDs as boot volumes). The latest Avere 44-node FXT3500 came in as #1 performer in NFS throughput ops/disk followed by prior Avere runs and BlueArc/HDS Mercury system runs. As discussed previously, although the Avere #1 system had no data SSDs, it did have almost 7TB of DRAM.

Significance

It’s great to see NetApp’s Data ONTAP 8.1 C-mode benchmarks. Also, the 24-node FAS6240 system result provides a significant proof point for NetApp’s clustering.

The fact that clustered, scale-out NAS systems have come to dominate throughput results probably indicates a need to rethink our throughput charts. We probably need to break out scale-out, clustered systems from monolithic systems. Also the immense DRAM memory, Flash cache, and/or SSD storage present in these systems indicates that we should somehow incorporate memory/Flash cache/SSD size as an additional normalizer for throughput activity.

Also, our throughput per disk drive count chart needs some changes. Although we have always excluded Flash cache and data SSD systems from this chart, the all DRAM systems seem to hold an unfair advantage here. We may need to exclude some level of DRAM caching as well as exclude data SSD and Flash cache use from this chart.

As always we welcome any recommendations for improvement of our SPECsfs2008 analysis. For the discriminating storage admin or for anyone who wants to learn more, we now include a additional results and all our other SPECsfs2008 charts in our recently update NAS Buying Guide available for purchase on our website.

[This performance dispatch was originally sent out to our newsletter subscribers in December of 2011. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more, in-depth analysis of NAS storage system features and performance please see our NAS Buying Guide available on our website.]

~~~~

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* SPECsfs2008 results from http://www.spec.org/sfs2008/results/

EMC 2011 Apr 11 Isilon hardware and software announcement

Apr 112011

EMC releases new Isilon hardware and software

EMC® recently announced new Isilon software and two new versions of Isilon hardware the S200 and X200.

Isilon and big data

EMC Isilon is a scale out NAS solution, which provides up to 14 PB of file storage within one file system, using a cluster of storage nodes, and maintaining a globally coherent cache. OneFS is their file operating system and it supports both file and block storage access. Currently, OneFS can support a cluster of up to 144 nodes that interoperate to create a single, multi-tier file system.

The Isilon team discussed their perspective on “big data”. It’s more than pure data analytics and encompasses 4 key activities: aggregating data from many sources; storing that data effectively; leveraging that data; and creating new value from that data.

Media and entertainment companies can collect content from film, animation, or special effects houses into one big file system where content can be edited, rendered, and derivative works generated ultimately creating new media products.
Life sciences companies can aggregate data from sequencers or research, storing potentially PBs of data that can be searched and analyzed for genomic mutations or new interactions, which is then used to focus more research and/or create new drugs.

New Isilon hardware

EMC’s has introduced new hardware for their high performance and high capacity Isilon systems. Namely, the new high-performance, S200 system and the new high capacity, X200 system. Isilon also has a nearline (archive) hardware (NL series) appliance but this was not upgraded at this time

Specifically the new hardware platforms includes the:

S200 platform - optimized for IOPS performance and as such, offers up to 10K-IOPS per node or 600MB/sec in a 2U sized package, ~2X more IOPS and ~1.9X more throughput than the previous generation S node. The S200-SAS uses 2.5” SAS drives with a small amount of SSDs for meta-data. The S200-SSD uses SSD only drives for the highest performance. The S200 supports 2-Intel Westmere processors, 96GB of DRAM, and has a maximum of 14TB of capacity per node at roughly the same price as the old S-series node.
X200 platform - optimized for capacity with concurrent streaming performance and as such, offers ~2K-IOPS per node or 250MB/sec in a 2U sized package, ~7% more IOPS and ~19% more throughput than the previous generation X node. The X200 uses 3.5” SATA-2 drives for capacity and supports 1-Intel Nehalem processor, 48GB of DRAM, and has a maximum of 24TB of capacity per node at roughly the same price as the old X-series node.

With the addition of the all SSD S200 node, Isilon now supports four storage tiers, i.e., tier 0 to tier 3 (nearline) storage. OneFS®, Isilon’s file operating system supports automated, transparent, policy-driven data migration across all tiers of the storage cluster.

New Isilon software

EMC also released new versions of Isilon’s OneFS, SyncIQ™ and InsightIQ software products:

OneFS 6.5 now supports native CIFS (both SMB1 and SMB2), NFS 4.0, Kerberizied NFSv3 support and enhanced enterprise authentication. Native CIFS is probably the most interesting here as now users can have their Windows user home directories on Isilon as well as their big data workloads, all under the same single file system.
SyncIQ 3.0 now supports full integration with SnapshotIQ™, supports smaller data mirroring, multi-thread processing, and multi-node replication increasing data mirror performance by 2 to 3X.
InsightIQ 1.5 now provides even better reporting on capacity as well as more scaleable performance reporting.

These software products operate in conjunction with SmartPools, SmartConnect, SmartQuotas and SnapshotIQ to provide automated policy based storage tiering and data protection for customer unstructured (file) data.

Announcement significance

EMC continues to upgrade their storage platforms with the latest Intel and drive hardware. With each new generation providing significant increases in performance, throughput while lowering cost in the process. The addition of CIFS and SSD only storage seem to attempts to become a one-stop, all-encompassing NAS solution for customer data. How this fits with the newly introduced VNX unified storage family is another question.

In addition, the need for big data solutions seems to be getting hotter as institutions look to gain increasing value from the mass of data they generate every business day. As such, EMC believes Isilon is unique in that it provides both enterprise class capabilities and a scale out NAS storage platform in the same system. NetApp, HP, IBM and others may wish to argue this advantage, but it’s hard to argue with Isilon’s success, as they currently have over 1700 customers today.

—-

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

SCI 2010 Dec 14 latest SPECsfs2008 benchmark results

Dec 142010

We once again return to analyze the latest Standard Performance Evaluation Corporation Network (System) File System 2008* (SPECsfs® 2008) benchmark results. There were three new NFS benchmarks, two from NetApp (FAS6240 with 1TB Flash Cache and FAS3270) and one from LSI [ONStor] (Cougar 6720). In addition, there were two new CIFS benchmarks one from EMC (Celerra VG8 with VMAX) and one from NetApp (FAS3210 with 512GB Flash Cache).
Latest SPECsfs2008 NFS results

SCISFS101214-001 (c)2011 Silverton Consulting, All Rights Reserved

Figure 1 SPECsfs2008* NFS throughput vs. memory size
We introduce a new chart for our SPECsfs2008 analysis a throughput vs. memory size scatter plot. For some reason I was very intrigued by this chart when I first created it. Originally all the data was together in one data series with only one linear trend line. I decided to break the data out into two groups, those systems using DRAM caching plus SSDs or NAND caching (6 systems) and those systems using only DRAM caching (33 systems).

After splitting the two types of systems out, I understood better what was shown here. There appears to be a distinct difference in the throughput gained from DRAM caching systems versus SSD use or NAND caching systems. Of course what’s missing from these charts is any comparison of pricing (because it’s not supplied in SPECsfs reports). Also the sample size is very small for the SSD/NAND caching systems and this may skew results.

SCISFS101214-002 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 2 SPECsfs2008* NFS throughput operations per second

The other issue with this chart is that SSD use is not accounted for in the memory quantity (just like SPECsfs2008). Adding SSD usable capacity to the memory in a system changed this chart significantly as the SSD system used ~19TB of SSD. Nonetheless, from this chart it seems clear that DRAM caching offers better throughput performance for the same amount of memory. Given today’s limited sample size we cannot discern any statistical difference between NAND cache system versus SSD. This analysis must until more SSD and NAND caching systems report in.

NetApp’s FAS6240 showed up as the new #2 in our top 10 NFS throughput operations per second, the only new submission on this chart. The other thing about the FAS6240 was it use of SAS disks. There haven’t been a lot of NFS results using SAS disk devices so the results here are encouraging.

SCISFS101214-003 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 3 SPECsfs2008* NFS top ORT results
Here we show the top 10 NFS ORT (overall response time results). Recall that ORT is a median-like response time over the whole benchmark duration. One can see all of the new NFS benchmarks in this chart. The NetApp 6240 showed up as the #1 with an ORT of 1.17 msec., the FAS 3270 came in at #7 with an ORT of 1.66 msec, and the LSI (ONStor) Cougar 6720 came in at #9 with an ORT of 1.67 msec.

I would have expected the NAND Cached FAS6240 to do well with ORT but the Avere Systems continue to amaze. However I must say that the Avere systems at #2, 3 and 5 may enjoy some advantage due to their relatively large DRAM caches (98, 163, and 424 GB respectively).

CIFS analysis

Below we report on top CIFS throughput results which include the latest submissions from EMC and NetApp.

SCISFS101214-004 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 4 SPECsfs2008* CIFS top throughput results

EMC’s Celerra VG8 came in as the new #1 with ~143K CIFS throughput operations per second and the NetApp FAS3210 came in at #3 with ~65K CIFS throughput operations per second. The strange thing about the top three results is that the #2 and #3 results used SSDs and NAND cache respectively but the #1 result just had a lot of disks (312). As we have said in the past, SSDs or NAND cache can substitute and perform just as well as more spindles if you don’t need the capacity.

SCISFS101214-005 (c) 2011 Silverton Consulting, All Rights Reserved

Figure 5 Top CIFS ORT results
Here we show the top 10 CIFS ORT (overall response time) results. Similar to NFS ORT results, CIFS ORT is a median-like response time over the whole benchmark duration. Both new submissions showed up in the top 10, NetApp FAS3210 at #4 and EMC Celerra VG8 at #9. Not surprising is the fact that 3 out of the top 4 ORT systems used FlashCache. Apple still holds the coveted #1 spot with an ORT of 1.22msec., but others are starting to encroach.

Significance

I struggled to understand the NFS memory size vs. throughput scatter plot chart. It should have been plain to see but it was unclear until I split out the data into two series. The fact that DRAM provides better throughput than NAND or SSDs is pretty noticeable, but lacking cost information, it’s impossible to compare cost effectiveness.

As always we welcome any recommendations for improvement of our SPECsfs2008 analysis. We also now include a top 30 version of these charts and other charts plus further analysis in our NAS briefing which is available for purchase from our website.

This performance dispatch was originally sent out to our newsletter subscribers in December of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of NAS system features and performance please take the time to examine our NAS Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 Dec 14 latest SPECsfs2008 benchmark results analysis for NFS and CIFS (PDF 721.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.spec.org as of 14 Dec 2010

NetApp 2010 Nov 09 New storage systems, Data ONTAP, and other announcements

Nov 092010

NetApp recently announced a new generations of their high-end and mid-range storage systems, denser and faster drive solutions, new Data ONTAP licensing and enhanced functionality, new rebranded/bundled management solutions, new unified connect adapters and new FlexPod for VMware pre-configured data center building blocks.

FAS/V6280, FAS/V6240, and FAS/V6210 enterprise storage

NetApp has updated their enterprise class storage systems by leveraging the latest multi-core processor hardware to double current performance, adding more capacity (up to 2.9PB on FAS/V6280 & FAS/V6240, 2.4PB on the FAS/V6210), more Flash Cache and the latest interface technology, 10GbE and 8GFC with faster SAS drive connections.

NetApp was the first enterprise storage with NAND caching and the FAS/V6280 now sports up to 8TB of NAND data cache. The FAS/V6240 and FAS/V6210 support 6 TB and 3 TB of Flash Cache, respectively. With the new larger NAND cache and latest processing hardware, IO performance should improve substantially. We look forward to seeing these systems showing up on SPC/SPECsfs2008/ESRP reports in the near future.

SAS seems to be displacing FC as a backend interface for enterprise storage. The fact that SAS hardware can also support SATA disks just makes this transition even better.

FAS/V3270, FAS/V3240, and FAS/V3210 mid-range storage

Along with the enterprise class storage refresh, NetApp also upgraded their mid-range storage line as well. Their new FAS/V3200 product line now supports up to 1.9PB of storage and up to 2TB of Flash Cache (both on the FAS/V3270). The FAS/V3240 and FAS/V3210 storage systems support 1.2PB capacity with 1TB Flash cache and 480TB capacity with 512GB Flash Cache, respectively. Recall that this series of subsystems is primarily intended for business and virtualization applications and mid-size enterprises.

New Denser/Faster (SSD) storage

Also, NetApp rolled out their new DS2246 disk shelf, which holds 24-2.5” disk drives in 2U of space. This increases disk density by 60%, and reduces power and cooling with equivalent reliability of their current DS4243 3.5” disk shelf.

Furthermore, NetApp now supports SSDs. The DS4243 disk shelf can be configured with 24-100GB SSDs to provide ultra-quick data access. NetApp did not mention their SSD supplier but with this low capacity drive we would assume they have SLC NAND under the covers. To correctly position use of Flash Cache versus SSDs, NetApp suggested that customers:

Use Flash Cache when their IO workload is dynamic and hot-data cannot be determined ahead of time, and
Use SSDs when their IO workload is deterministic and all reads must be faster.

New Data ONTAP 8.0.1 licensing and enhancements

Currently, Data ONTAP 8 has over 1000 customers and its licensing just got simpler. The Data ONTAP base license now includes business continuity, secure multi-tenancy, automated management, and a single storage protocol at no additional charge (CIFS may be a popular choice as that’s their most expensive protocol). Also the myriad of Data ONTAP features have now been combined into 7 key products: SnapRestore®, SnapMirror®, FlexClone®, SnapVault®, and SnapManager® Suite, optional storage protocols, and a complete offering bundle.

In addition, the new Data ONTAP 8.0.1 supports larger aggregates, data compression, transparent data mobility and unified connect.

Larger aggregates – Data ONTAP 8.0.1 now supports aggregates of 100TB more than tripling previous maximums. Larger aggregate size has been the number one customer request.
Data compression – Data ONTAP 8.0.1 now offers an option to compress data as well as their current deduplication capabilities. Compression should help reduce storage footprint for more environments.
Transparent data mobility – Data ONTAP 8.0.1 now supplies automated, non-disruptive data migration between aggregates within the same storage system. Customers can use this to non-disruptively move data between aggregates based on different storage types like between FC and SATA

New Unified Connect

With Data ONTAP 8.0.1, a new CNA and new interface hardware, NetApp now supports a unified connect interface whereby servers can talk iSCSI, FCoE, CIFS, and NFS all over the same wire, from the same server port into the same storage port. Unified connect should reduce cabling and management requirements substantially while improving port and bandwidth utilization and interface expansion slot efficiency. While at SNW last month, we talked with one chip vendor that supported all these protocols from the same unified controller chip.

OnCommand Management Suite

NetApp has taken the opportunity to rebrand and bundle all their management solutions under one name – OnCommand Management Suite. This is their new branding for their current System Manager, Ops Manager, My AutoSupport™, Provisioning Manager, Protection Manager, SnapManager®, SnapDrive®, and SANscreen® software products. With the new umbrella suite of products look for more common APIs, look and feel and future product offerings.

FlexPod data center building blocks

Similar to EMC/Cisco/VMware Acadia Vblocks, NetApp now offers FlexPods which incorporate Cisco® UCS® blade systems, Cisco Nexus switching, VMware® vSphere™ and vCenter™, and NetApp FAS storage into one pre-tested, pre-configured, easy to install, offering. FlexPod is targeted at data centers with 1000 to 2000 users and wanting a quick and painless way to offer virtualized data processing services.

Announcement significance

Within almost the last 18 months EMC, HDS and now NetApp have rolled out new enterprise class storage. Within the last month IBM and NetApp have rolled out new mid-range storage as well. It’s good to see new hardware on the market and perhaps NetApp was due for an upgrade and it’s a great time to be in the storage business.

NetApp is not moving that aggressively into SSD deployment as some of their competitors and currently have no plans to offer sub-LUN migration for SSDs like IBM, HDS, and EMC. Perhaps, NetApp sees Flash Cache as a better solution to dynamic IO workloads and feel sub-LUN migration doesn’t provide that much additional benefit to warrant implementation. On the other hand, what with WAFL, aggregates and other internal capabilities, it wouldn’t seem to take much to offer sub-LUN migration if it were deemed a requirement. Also offering 8TB of Flash Cache seems to be a wager that NetApp has the right solution here.

Data ONTAP licensing was due for simplification. The over 30 product features was becoming a bit to complex to understand and order. The rebundled/re-branded OnCommand management suite provides a nice touch but most tangible benefits are still a ways off.

FlexPods probably indicate that Acadia is starting to succeed in the market and that NetApp wants to share in that success. All this probably makes Cisco and VMware very happy. Why EMC felt the need to spin it out, as a separate JV vs. NETAPP’s technology partnership isn’t as clear. How support is coordinated may be the key customer differential from a JV vs. a technical partnership. NetApp states that there will be collaborative support but how this plays out in real support calls is TBD.

Probably what’s missing from all this is a more scale-able platform something like VMAX, HP’s 3PAR or to a lesser extent Hitachi’s VSP. Yes, a single architecture spans their complete product line but it’s still different hardware for the most part and can’t just simply be incrementally scaled up from one to the next higher/lower system. We suppose this is the ultimate promise in Data ONTAP 8 mode, where by multiple FAS/V storage systems can be combined to offer one seamless, clustered storage environment. We eagerly anticipate the unification of Data ONTAP 7 and Data ONTAP 8 mode into one Data ONTAP clustered storage sometime in the near future.

A PDF version of this can be found at

NetaApp 2010 Nov 09 New NetApp storage systems, Data ONTAP, and other announcements (PDF 374.7 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

HP 2010 Jun 22 StorageWorks StoreOnce, P4800 Storage Blade SAN and other announcements

Jun 222010

New HP StorageWorks StoreOnce, P4800 Storage Blade SAN, EVA FCoE&iSCSI, and SVSP-EVA bundle

HP StorageWorks made some new announcements coincident with the start of Technology Forum this week. HP announced new software functionality called StoreOnce that provides deduplication services for multiple product incarnations. Also, HP promoting their converged infrastructure approach has released a P4800 storage controller blade for their blade enclosure that provides iSCSI storage. Finally, EVA through the use of a router now supports FCOE and iSCSI interfaces and HP announced a new EVA Cluster which bundles their SVSP (HP SAN Virtualization Services Platform) with EVA storage subsystems

HP’s StoreOnce deduplication software

HP has introduced a new software deduplication service intended to be used across their product line. Using this software, deduplicated data should no longer need to be re-constituted as it hops from appliance to network back to storage subsystem. The software will eventually be used in backup clients, virtual machines, inline appliances, and scale-out storage systems and once implemented should allow for a relatively low-bandwidth data replication between diverse HP StorOnce systems and software.

The first instantiation of HP’s StoreOnce appears in their new D2D4312 Backup System as well as other current D2D products. The D2D4312 system supports up to 48TB of raw storage and HP says it has 20% better performance and 2X price-performance than the leading competitor. Also, they have now added NFS, CIFS, and OST to their current interfaces of FC or iSCSI VTL for the D2D4312 appliance.

P4800 BladeSystem SAN

HP StorageWorks also announced a new packaging of their P4000 (formerly LeftHand) storage controllers that now comes in a blade configuration. The P4800 blade system can support up to 4-storage blades, SAS attached to 140 disk drives in a high density MDS600 cabinet for a total of 63TBs of iSCSI storage.

The P4800 makes use of HP’s Virtual Connect Flex-10 Gbe networking technology providing high performance, eliminating switches, cabling and reducing administrator burden. HP has demonstrated the P4800 using VDI services in a reference architecture, which supports up to 1600 users at ½ the cost and 60% less space than competitive offerings. In any case, as standard iSCSI storage the P4800 can support any application’s storage needs. HP’s current BladeSystem has slots for up to 16 blades but can be extended by adding another blade enclosure for an additional 16 blade slots.

Other Enhancements

In addition, HP announced that their MPX200 router now supports FCoE and iSCSI attachments for EVA and XP storage. MPX200′s FCoE interface protocol support is provided free of charge, via a firmware download.

Also HP is now offering a system bundle that includes SVSP and EVA 6K and/or 8K storage subsystems. Previously, all of this could have been ordered separately but now can be ordered direct from the factory, in a pre-configured, factory-tested configuration. EVA Cluster configurations scale from 2 to 6 EVA subsystems and includes two Virtualization Services Manager servers as standard. The SVSP can also provide storage virtualization services for other, heterogeneous storage subsystems and as such, the bundled SVSP-EVA configuration can attach other storage subsystems behind the SVSP nodes.

Announcement significance

HP’s rolling out of StoreOnce is a good idea but one implementation does not suffice to show any true advantage other than performance gains. On that note, D2D4312′s performance gains have been independently verified. However, given the lack of standard and independent performance benchmarks for deduplication systems, any realistic performance comparisons are very difficult to validate.

The new P4800 Blade System SAN shows HP’s converged infrastructure strategy is starting to bite. It’s entirely understandable that LeftHand storage would be first as they already support a virtual appliance and prior to HP’s acquisition ran on standard x86 servers. Using this to implement a super VDI-in-a-box infrastructure is a great application

The other two announcements are much less exciting. Bundling SVSP and EVA may be good channel strategy if demonstrable need exists. SVSP, in its prior incarnation may be a bit more complex than most channel players can deal with but bundling like this makes it easier to sell by isolating most configuration complexity at the factory and not at the channel.

As for the MPX200, HP seems to be causing some turmoil in FCoE standards bodies with HP and Cisco trading claims of standard breaking but we must leave that for another time. However, anything that adds new storage protocols to the EVA seems a smart play from our perspective. That the MDX200 is a bump in the cable is ok for now but in our opinion this functionality or something like it needs to be natively integrated into base EVA storage.

A PDF version of this can be found at

HP 2010 June 22 StoreOnce dedupe and new P4800 BladeSystem SAN announcement summary (PDF 276.9 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

SCI 2009 Dec 31 Latest SPECsfs2008 results analysis

Dec 312009

We now turn to analysis of the latest SPECsfs® 2008* benchmark results. Fortunately there were a number of new NFS and CIFS benchmarks over the last quarter, including eight new NFS results in the top 10 and a couple of new CIFS results in the top 10.

Latest SPECsfs2008 NFS results

(SCISFS091230-001) 2009 (c) Silverton Consulting, All Rights Reserved

Huawei Semantec’s N8500 NAS storage system came in with a new number one result at over 176K NFS throughput operations per second. This was accomplished on Huawei server and storage hardware running Semantec’s FileStore Clustered NAS appliance software. Rounding out the rest of the top 3 were BlueArc’s new “midrange” Mercury 100 cluster, and HP BL860C. Both the HP and Huawei Semantec systems supported multi-node clusters, 12 nodes and 4 nodes respectively and both were running Semantec’s VxFS software. The BlueArc system has hardware acceleration and running on a two-node cluster.

Avere is a new NAS storage system and also supports a multi-node cluster. There were three Avere systems benchmarked in this last quarter with six nodes, two nodes, and one node. Their six-node system attained #4 of the top 10, running at over 130K NFS throughput operations with minimal disk drives (using only

79 drives). Most of the other systems in the top 10 had many more drives with the exception of NetApp FAS3160 with PAM acceleration that ran with 56 drives.

(SCISFS091230-002) 2009 (c) Silverton Consulting, All Rights Reserved

For operational response time (ORT) results, we can see more of Avere systems in the top results attaining three out of the top four ORT results at 1.3, 1.33, and 1.38 msec. respectively. It’s interesting to see that the other top 10 NFS throughput results from Huawei Semantec and the NetApp systems also achieved top 10 ORT results as well.

Latest SPECsfs2008 CIFS results

(SCISFS091230-003) 2009 (c) Silverton Consulting, All Rights Reserved

There were two new CIFS benchmarks from Fujitsu submitted this last quarter, one for their Primergy TX200 S5 and one for their Primergy BX920 S1. Both results managed to attain a position in a top 10 CIFS result, one in throughput and the other in ORT.

It doesn’t appear that the SPECsfs CIFS benchmark is gaining much mindshare. As of this dispatch there are only 11 total submissions. Nonetheless, the latest Fujitsu Primergy TX200 reached the top 10 (out of 11 – sigh). The dominant result remains with Apple’s Xserve running Snow Leopard running over 44K CIFS throughput operations per second.

(SCISFS091230-004) 2009 (c) Silverton Consulting, All Rights Reserved

In the CIFS operational response time results, the new Fujitsu Primergy BX920 S1 attained #5 with a 2.9 msec. ORT. Once again an older Apple storage system holds the top ORT result at 1.2 msec.

Significance

Nice to see some activity at the top end in NFS results. The Huawei Semantec system has laid down the gauntlet with some pretty impressive numbers but we have yet to hear from some other vendors with enterprise class systems. NetApp, as always, was early to submit benchmark results for their current systems and their next generation Data ONTAP 7G will also support a clustered file system. Can’t wait to see how well that performs.

We are a bit disappointed in the paucity of CIFS results and yet continue to report them in the hope that more will be released. But as the benchmark has been out for over 18 months now it is not gaining many adherents. Also it would be wonderful to see more submissions for both benchmarks using the same hardware/software so that end-users could see what they are getting when they use CIFS or NFS.

This performance dispatch was originally sent out to our newsletter subscribers in December of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of NAS system features and performance please take the time to examine our NAS Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 December 31 Update to SPECsfs® 2008 performance results (PDF 578.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* SPECsfs2008 results from http://www.spec.org/sfs2008/results/

SCI 2009 Sep 25 latest SPECsfs2008 results analysis

Sep 252009

We now turn to analysis of the new SPECsfs®2008* benchmark results. Unfortunately there were no new CIFS reports but there were more than a couple of new NFS benchmarks, mostly from NetApp but we also have a first time Isilon benchmark as well.

Latest SPECsfs2008 NFS results

(SCISFS090925-001) 2009 (c) Silverton Consulting, All Rights Reserved

NetApp’s FAS6080 now takes top honors with over 120K SpecSFS2008 ops, and aside from ExaStore 8-node cluster NetApp’s FAS 3160 rounds out the rest of the top 5. The FAS6080 was using 2-10Gbe links. Isilon IQ5400s came in 6^th with a 10 storage node system with 20-Gige links. Recall that ExaStore had an 8 node cluster and was using 48-Gige links.

NetApp’s FAS 3160’s illustrates how their performance accelerator module (PAM) can improve performance. For the

#3 3160 had 56 FC disk drives with PAM,
#4 3160 had 224 FC disks with no PAM had roughly equal throughput performance
#5 3160 had 96 SATA disks with PAM and had roughly equal throughput performance.

(SCISFS090925-002) 2009 (c) Silverton Consulting, All Rights Reserved

NetApp’s PAM module seem to also positively impact response time results. In the top 10 (13 actually) FAS with PAM took numbers 2, 3 and was tied for 10th at 1.6, 1.7, and 2.2msec., respectively. Having PAM onboard a FAS system seems to shave 25% off of –system response time vis a vis an equivalent FAS3160. It would have been interesting to see what PAM could have done for the FAS6080 but for that, we will need to wait until another time.

Isilon came in at number 5 with a respectable 1.9msec for 10 storage node system. I would have thought all that intercluster overhead would have adversely impacted latency, guess not.

Latest SPECsfs2008 CIFS results

There were no new CIFS benchmarks submitted during the last quarter and so, for that data please look to our prior SPECsfs2008 StorInt dispatches.[1].

Significance

Finally some mainstream, enterprise class NAS systems are showing up in SPECsfs2008 benchmark results. I am very happy to see NetApp’s top end FAS6080 and their mid-range FAS3160 system benchmarks released under SPECsfs2008. Now that NetApp has put a stake in the ground perhaps the rest of the high-end NAS system vendors will follow in their footsteps.

It’s also interesting to see the performance benefits of NetApp’s PAM. It’s somewhat surprising to see it roughly make up for 4x the spindles with no degradation in throughput performance and significantly better latency. I believe these are generation 2 Flash based PAM cards, and added about 512GB of cache to their NAS systems. NetApp’s approach to using NAND flash is to place it in the cache where it benefits all workloads. As far as I can see (from the SPECsfs2008) data, it seems to be working.

A PDF version of this can be found at

SCI 2009 September 25 Update to SPECsfs® 2008 performance results analysis (PDF 433.7 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

* SPECsfs2008 results from http://www.spec.org/sfs2008/results/

[1] See http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html for prior performance StorInt’s

SCI 2009 Jun 25 latest SPECsfs2008 results analysis

Jun 252009

We now turn to analysis of the new SPECsfs®2008* benchmark results. Unfortunately there were not a lot of highend SPECsfs2008 results, most notably ONStor and Apple for NFS, and Apple and Fujitsu Siemens for CIFS.

Latest SPECsfs2008 results

(SCISFS090625-001) 2009 (c) Silverton Consulting, All Rights Reserved

ONStor Cougar now takes second spot in the top 10 throughput results. The Cougar system had ~½ the disks of the ExaStore box and ~7 times less memory (cache). Given all that, its results standup pretty well. The two new Apple NFS benchmark results (Snow Leopard and Leopard server) round out the rest of the new members to the top 10 list at numbers 7 and 10 respectively.

Recall from our last report# that some NetApp results utilized their PAM card. Also, the SGI product result used Infiniband, both ExaStore benchmarks used10GbE and all the rest use GigE. In all fairness the networking connection may not be a limiting factor in SPECsfs2008 results.

(SCISFS090625-002) 2009 (c) Silverton Consulting, All Rights Reserved

As discussed last time for NFS ORT results, one can clearly see the advantage of NetApp’s PAM with FC disks and yet, the new ONStor Cougar benchmark shows up at number 3, only ~60 microsec behind the NetApp/PAM result. The only other new showing was Apple’s Snow Leopard server coming in at number 9.

Next we turn to CIFS results, the five new results have more than doubled SPECsfs2008 CIFS benchmarks. Recall the SGI is using Infiniband while all the others use GigE interfaces.

(SCISFS090625-003) (c) 2009 Silverton Consulting, All Rights Reserved

We suppose it’s not surprising to see Apple’s Snow Leopard leading the pack, coming in at the new #1 in CIFS throughput considering its market place but one would think others could do better. More impressive is that the Snow Leopard result used only 65 disks whereas the SGI result sported 242 disks (~4X). It’s unclear to us whether this is the new Apple OEM of Sun ZFS file system at work here, but clearly Apple CIFS performance has improved significantly.

(SCISFS090625-004) 2009 (c) Silverton Consulting, All Rights Reserved

Once again, Apple shows up well in CIFS ORT results. Although, as best we can determine this #1 result was an early Leopard version (Mac OSX10.5.1) whereas the #3 result (using Mac OSX10.5.7) had a 2.93Ghz Nehalem processor. The other major difference was a dual port GigE card for the #1 result vs. a 6-port GigE card in the slower version.

(SCISFS090625-005) 2009 (c) Silverton Consulting, All Rights Reserved

Figure 5 SPECsfs2008* CIFS vs. NFS throughput correlation

We have discussed this in earlier reports but once again the results would support our contention that the CIFS protocol results in better throughput than NFSv3. As pointed out to me, a couple of provisos are warranted here, namely:

NFS workloads are not readily comparable to CIFS in a number of dimensions not the least of which is that NFS is stateless and CIFS is state-full. Also, the relative proportions of the actual workloads don’t exactly matchup, e.g. percentages for NFS read and write operations versus CIFS read_andx and write_andx operations are slightly different (NFS read@18% vs. CIFS read_andx@20.5% and NFS write@10% vs. CIFS write_andx@8.6%), file sizes are different, and all the remaining operations, which, to be fair, represent a significant majority of their respective workloads, are by definition, nigh impossible to compare. SPECsfs benchmarks for the two are implemented to reflect all of these differences.
A majority of these results (3 of 5) come from the same vendor (Apple) and their great CIFS and/or poor NFS implementations may be skewing results.
Only five subsystems have recorded results for both interfaces but the correlation looks pretty good for now.
Normally, host operating system affects could skew these results but the SPECsfs2008 benchmarks emulate their own client side stacks for both protocols, thus negating any operating system affects.

Nonetheless, once again, considering that at the user level all specific protocol details result in emulating comparable end-user workloads, the results do show a significant advantage for CIFS (~2.4X) throughput over NFS.

Significance

Our earlier discussion on CIFS vs. NFSv3 throughput differences resulted in quite a lot of discussion. It was early then, and still is now, but we continue to stand by our claim, given benchmark results, CIFS seems to perform better than NFSv3. More dual protocol results should help clarify this relationship.

Slowly, more SPECsfs2008 results are being released. But, where are the major NAS systems. It’s been 10 months since the old SPECsfs benchmark was retired and we still lack benchmark results for all the major NAS systems. In the mean time, smaller players continue to release results; just happy to get any visibility, validity and traction they can muster.

This performance dispatch was originally sent out to our newsletter subscribers in June of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of NAS system features and performance please take the time to examine our NAS Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 June25 Update to SPECsfs® 2008 performance results (PDF 675.4 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* SPECsfs2008 results from http://www.spec.org/sfs2008/results/

# Available at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2009 Mar 30 latest SPECsfs2008 results analysis

Mar 302009

We now turn to analysis of the new SPECsfs®2008* benchmark results. Unfortunately there were not a lot of new SPECsfs2008 results, most notably NetApp released results with their Performance Acceleration Module (PAM) for both FC and SATA disks and one new CIFS result from Fujitsu Siemens Computers.

Latest SPECsfs2008 results

(SCISFS090330-001) 2009 (c) Silverton Consulting, All Rights Reserved

NetApp’s use of their PAM to speed up SATA disk displayed some interesting results, namely that SATA disks can perform similar to FCAL disks when used with PAM. Two items of note about the PAM results:

The FCAL result had half the number of spindles as the non-PAM result (112 disks with PAM vs. 224 without PAM) but attained similar SPEFsfs2008 throughput.
The SATA results also had the same number of disks as the FCAL-PAM benchmark or 112 disks. The only ignificant difference between the two PAM results was in the ORT results (see below).

Recall that NetApp’s PAM is an onboard cache extension using DRAM devices and comes in 16GB cards with special FlexScale software to control PAM use. These FAS3140 results incorporated 2-PAM DRAM modules adding 32GB on top of the 9GB of cache used in the base FAS3140 benchmark, for a total of 41GB of cache.

Also, as mentioned last time, the SGI product result used Infiniband, both ExaStore benchmarks used10GbE and the rest used gigabit Ethernet. In all fairness the networking connection may not be a limiting factor in SPECsfs2008 results.

(SCISFS090330-002) 2009 (c) Silverton Consulting, All Rights Reserved

Turning to ORT results, here one can clearly see the advantage of PAM for FC disks. The PAM-FCAL benchmark reported a 1.7 msec. ORT. Without PAM, the FAS3140 system with twice the disk drives, only delivered a 2.6 msec. ORT. It is also interesting that the PAM-SATA benchmark came in at an ORT of 2.8 msec., just an ~8% degradation from the FAS3140 with FCAL disks. Unclear what the cost difference would be between a FAS3140 with 224 FCAL disks vs. a FAS3140 with 32GB PAM and 112 SATA disks but they look about the same from a SPECsfs2008 performance perspective.

One rule of thumb often used in the mainframe space is that doubling cache size alone should reduce response time by 10%. NetApp’s benchmarked systems increased cache size by 4X (9GB to 41GB) leading us to expect a decrease in response time by ~20%. NetApp managed to exceed this with a reported ~35% decrease in ORT while at the same time halving the number of spindles.

Next we turn to CIFS results. So far only four results have been released but what reports exist are shown below. Recall the SGI is using Infiniband while the others all use GigE hardware interfaces.

(SCISFS090330-003) 2009 (c) Silverton Consulting, All Rights Reserved

Figure 3 SPECsfs2008* CIFS results

Significance

Our last SPECsfs2008[1] analysis showed some preliminary correlations showing that CIFS has about 2X the throughput of NFSv3 for similar systems. Some interesting discussions resulted from this claim but we still stand by what we said there. More combined CIFS-NFSv3 results for the same systems would certainly help clarify this claim.

Presently, SPECsfs2008 remains more of a curiosity from an end-user perspective given the limited number of vendor submissions. However, it clearly has some uses considering the recent NetApp results. Hopefully, more vendors will see the benefit in releasing results that should transform this curiosity into viable tool for end-users to better understand NAS performance.

This performance dispatch was originally sent out to our newsletter subscribers in March of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of NAS system features and performance please take the time to examine our NAS Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 March 30 Update to the SPECsfs® 2008 performance results (PDF 453.7 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* SPECsfs2008 results from http://www.spec.org/sfs2008/results/

[1] Available at http://www.silvertonconsulting.com/page2/page2d/page2d2/StorInt_Dispatch_2008.html

SCI 2008 Dec 21 first SPECsfs2008 results analysis

Dec 212008

This marks our first analysis of the new SPECsfs®2008* benchmark results. One will no doubt recall that SPEC® SFS97 was closed out earlier this year and a new SPECsfs2008 benchmark was released. Unfortunately there are not a lot of SPECsfs2008 results posted at this time but what exists is reported here.

Latest SPECsfs2008 results

(SCISFS081221-001) (c) 2008 Silverton Consulting, All Rights Reserved

One advantage of the new SPECsfs2008 benchmark is the support for both CIFS and NFS performance data. Although as stated on the SPECsfs2008 website, “SPECsfs2008 results may only be compared to other SPECsfs2008 results for the same protocol. SPECsfs2008_cifs and SPECsfs2008_nfs.v3 are not comparable because they are generated using completely different workloads.” # SCI has provided below a hypothetical comparison of the two protocol benchmark results. My friends at Microsoft will be pleased to see that when comparing the same subsystems running CIFS and NFSv3 for SPECsfs2008 workloads CIFS throughput is on average twice the NFS throughput. In figure 1 we have a scatter plot of the three subsystems that reported both NFS and CIFS results.

As can be seen by the chart the linear regression is quite good (R**2 at 0.93) and the regression equation is

CIFS_throughput = NFS_throughput * 2.1 – 679
A couple of provisos to watch for here:

Only three subsystems recorded results for both interfaces but the data looks pretty consistent at the moment. As more data is reported for both CIFS and NFSv3 we will update the regression but for now it looks pretty solid.
NFS workloads are not readily comparable to CIFS in a number of dimensions not the least of which that NFS is stateless and CIFS is state-full. SPECsfs benchmarks for the two are implemented to reflect those differences. Also, the relative proportions of the actual workload counterparts don’t exactly matchup, e.g. workload percentages for NFS read and write operations versus CIFS read_andx and write_andx operations are slightly different (NFS read@18% vs. CIFS read_andx@20.5% and NFS write@10% vs. CIFS write_andx@8.6%)$, file sizes used are different, and all the remaining operations, which in all fairness represent a significant majority of their respective workloads, are by definition relatively impossible to compare.

Nonetheless, taking a significant leap of faith that at the user level all of these protocol details result in emulating comparable end-user workloads, the results do show a significant difference in throughput results between the two protocols.

Both SPEC SFS97 and SPECsfs2008 support any networking hardware supplying TCP or UDP but SPECsfs2008 is the first one with published results for Infiniband, GigE, and 10GbE as system interfaces. One would think that Infiniband would have the edge in any comparison among these three but as figure 2 shows it’s not apparent that networking hardware has any advantage. The SGI product is using Infiniband and both ExaStore benchmarks use 10GbE for these results. In all fairness the networking connection may not be a limiting factor in SPECsfs2008 results.

(SCISFS081221-002) (c) 2008 Silverton Consulting, All Rights Reserved

The SPECsfs2008 User’s Guide also states the workloads have changed between the old SPEC SFS97 and the latest benchmark. Some of these changes include:

Operation percentages have been adjusted to reflect recent usage
Maximum file sizes have been increased
Total file set size for a given workload has been increased
Percentage of files accessed during a run has been increased
Logical file transfer size has been increased
NFS block (physical) transfer size is now negotiated with the server

All of this would make comparing SPEC SFS97 against SPECsfs2008 workloads a difficult endeavor. Also, no vendor has currently released a SPECsfs2008 and a SPECsfs97 benchmark for the same system. However, if more than a few vendors did this someday we might be able to calibrate results between the two. By not showing how results between the old and new benchmarks can differ may hold vendors back from releasing new SPECsfs2008 results. However, if one were able to compare results under both the old and new benchmarks, one could more easily show that current results were due to the changes in the benchmark and not an artifact of the system being benchmarked.

(SCISFS081221-003) (c) 2008 Silverton Consulting, All Rights Reserved

Another nail in the Infiniband coffin is delivered by ORT results. Again one result does not make a statistical relationship and physical network connections may not be a limiting factor in ORT measurements. However, as you may recall this is the average overall response time delivered by a storage system and SGI is the only Infiniband user present. Of course this doesn’t say much about 10GbE either as ExaStore was the only vendor reporting 10GbE results. Also of interest is the great showing of the Apple Xserve device (with 49 disk drives), not bad for this class system. If only they could grow the top end they could have something here.
Next we turn to CIFS results. So far only three results have been released but what is shown below. Recall the SGI is using Infiniband while the other two are using GigE hardware interfaces.

(SCISFS081221-004) (c) 2008 Silverton Consulting, All Rights Reserved

Significance

SPEC SFS97 has served the IT industry well but its time has come. SPECsfs2008 has the potential to be a worthy substitute but only wider adoption and more result submissions can prove this out.

Notwithstanding all the provisos against comparing NFS and CIFS SPECsfs2008 results, we especially like that we can at least try to compare CIFS and NFS results albeit with “… completely different workloads”. Whether these comparisons are real or not only time will tell. More data here will help refine the apparent advantage CIFS enjoys. Perhaps this was obvious to most vendors inasmuch as NFS was a stateless protocol and CIFS was state-full protocol, but it had never been shown quite so publicly before.

Intermixing Infiniband, 10GbE and GigE with the same results should be informative. Most HPC users would swear by the advantage inherent in using Infiniband for I/O operations. However, current results show that this doesn’t seem to offer much of an advantage in SPECsfs2008 results. Results are skimpy as of yet and we may live to see these words thrown back at us but for now they stand as is. It would certainly be better if the same system released three results with GigE, 10GbE and Infiniband hardware but its unclear why any vendor would see an advantage doing this.

In retrospect, it’s probably the right time to move onto a benchmark that can show CIFS and NFSv3 results and doesn’t care what hardware is used. Hopefully, the end-user community will require their storage vendors to publish more SPECsfs2008 results and thereby help to make this a worthy successor to SPEC SFS97.

This performance dispatch was originally sent out to our newsletter subscribers in December of 2008. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of NAS system features and performance please take the time to examine our NAS Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2008 December 21 SPECsfs® 2008 performance Report (PDF 1.0 MiB)

* SPECsfs2008 results from http://www.spec.org/sfs2008/results/

# SPECsfs2008 statements from http://www.spec.org/sfs2008/results/ as of 1/22/09

$ SPECsfs2008 User’s Guide, available from http://www.spec.org/sfs2008/ as of 1/22/09

Older Entries

Latest SPECsfs2008 results

Significance

Isilon and big data

New Isilon hardware

New Isilon software

Announcement significance

CIFS analysis

Significance

FAS/V6280, FAS/V6240, and FAS/V6210 enterprise storage

FAS/V3270, FAS/V3240, and FAS/V3210 mid-range storage

New Denser/Faster (SSD) storage

New Data ONTAP 8.0.1 licensing and enhancements

New Unified Connect

OnCommand Management Suite

FlexPod data center building blocks

Announcement significance

HP’s StoreOnce deduplication software

P4800 BladeSystem SAN

Other Enhancements

Announcement significance

Latest SPECsfs2008 NFS results

Latest SPECsfs2008 CIFS results

Significance

Latest SPECsfs2008 NFS results

Latest SPECsfs2008 CIFS results

Significance

Latest SPECsfs2008 results

Latest SPECsfs2008 results

Significance

Latest SPECsfs2008 results

Significance

NEWS and EVENTS

TOP DOWNLOADS

TWITTER FEED...

POPULAR POSTS

NEWSLETTER

TRANSLATE THIS PAGE

Dispatch Tag Cloud

Dispatch categories

Dispatch archives