This is SCI’s third report on Microsoft Exchange Solution Review Program (ESRP) performance results. This report focuses on 0 to 1000 mailboxes category and our prior reports discussed over 5000 and 1000 to 5000 mailboxes result categories. See our prior ESRP/Jetstress dispatches for a better description of ESRP benchmarks and our reported metrics.
Latest ESRP V2.0 results
In order to better compare Jetstress results we report on both normalized and un-normalized results. For normalized results in this bottom-tier category we use operations per mailbox (mbx). As an example, Compellent reports results for 600 mailboxes at ~331, but normalized to per mbx their results are 0.5 database transfers/sec/mbx (see Figure 1).
We have added a new chart for this dispatch ranking the log file playback.. This metric depicts the average time a 1MB log file takes to be played against the database and is not always specified for ESRP results but all of these ESRP results in this bottom-tier include this metric.
The Top 3 normalized Jetstress results for the 0 to 1000 mailbox category were IBM DS3300 in Clustered Continuous Replication (CCR), Dell PowerEdge 2900 CCR and Sun Fire X4150 (see Figure 1). A few considerations on normalized results
- Most of these results were for iSCSI subsystems, Dell and Sun results were SAS attached storage and the Compellent result was for FC attached storage.
- Normalized results don’t always scale well. Although 5 of these results were for 1000 mailboxes (IBM, Dell, SUN, EMC NS20, Compellent, and one of the EMC CX3-20) the rest had an average of ~730 mailboxes. For example, although the AMI StorTrends 3200i may do well at 600 mbx it may or may not scale much beyond that.
- Another surprise is that the normalized results span such a large range. The top 10 normalized results range from over 1.8 down to around .5 database operations per second per mbx over a factor of 3X, for relatively similar numbers of mailboxes. The IBM DS3300 is the clear winner here.
The top 3 subsystems in this category are IBM DS3300 CCR, Dell and the Sun subsystems. In contrast to the normalized results above all of the unnormalized top 10 reported results are for 1000 mailboxes. There was one less iSCSI result here in comparison to the normalized results as the other Dell PowerEdge 3000 result was also SAS.
There should be some correlation between the other results and the database backup results but apparently there is none. The top 3 results were all SAS subsystems and the next two were iSCSI and FC respectively. This implies that SAS has an advantage when it comes to raw data reading over iSCSI. Not sure why the FC product didn’t do better but that’s subject for another discussion. Again the storage group results bear little relation to the aggregate database backup results but the number of storage groups is one of those vendor-optimized variables that confound ESRP/Jetstress result comparisons.
Our first report on Log playout results does not appear that interesting. However, the Log results don’t correlate that well with any of the previous results. As the ESRP defines this metric, this sort of activity would typically occur during a crash recovery. Given these results all the subsystems are pretty comparable. But if crash recovery time is important perhaps the Dell PowerEdge 2900 or EMC CX3-20 may be worth a look.
From our view on ESRP results in this bottom-tier competition is abating. The only new benchmark during the past quarter for this tier was the AMI result at 600 mailboxes. In fact, if one closely examined this and our prior ESRP result dispatches they would notice that some results showed up twice. All results for 1000 mailboxes were reported on in both this tier and the next higher tier (1K to 5K mailboxes). We did this because there are so few results <1000 mailboxes.
iSCSI storage represents the majority of reported results in this category although the few SAS subsystems do almost as well. SAS does seem to have an advantage in backup speeds but other than that iSCSI handles the other workloads better. FC is just not well represented in this tier probably due to cost.
This is our third ESRP report and we have nov analyzed top 10 results for every category. We continue to welcome any feedback on how to do better. Jetstress results are inherently un-comparable. Nonetheless we believe Exchange results deserve some comparison and result rankings so that the public can be properly informed and as such make better storage purchases. Our next ESRP/Jetstress report will return to the >5K mailbox tier.
This performance dispatch was sent out to our newsletter subscribers in October of 2008. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.
A PDF version of this can be found atSCI 2008 October 21 Update to ESRP/Jetstress benchmark performance results
Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community
 ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 21 October 2008