SCI 2008 Jul 22 latest ESRP v2 (Exchange 2007) performance results analysis – 1K to 5K mailboxes

In Celerra, Dell EMC, DS3300, ESRP, ESRP v2/Exchange 2007, HPE, IBM, NS40 by Administrator

This is SCI’s second report on Microsoft Exchange Solution Review Program (ESRP)[1] performance results.  This report focuses on 1000 to 5000 mailboxes category and our previous report discussed the over 5000 mailbox category.

Latest ESRP V2.0 results

(SCI080722-001) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080722-001) (c) 2008 Silverton Consulting, All Rights Reserved

ESRP uses a Jetstress benchmark together with a formalized reporting framework to publish results.  Jetstress benchmark variables are many, involving not only mail store database and log layouts, but also the number of Exchange servers and numerous other storage, host and software configuration options.

In order to better compare Jetstress results we report on both normalized and un-normalized results.  For normalized results in this mid-tier category we use operations per 1000 mailboxes (1Kmbx).  As an example, HP reports MSA70 results for 4000 mailboxes at ~6350 (see Figure 2), but normalized to 1Kmbx their results are ~1590 database transfers/sec/1Kmbx (see Figure 1).

Both the normalized and un-normalized results are for database transfer activity only and do not include log file transfers or other reported characteristics.   Also ‘DB wt/sec/1Kmbx’ is database write transfers per second per 1000 mailboxes and similarly, ‘DB rd/sec/1Kmbx’ is database read transfers per second per 1000 mailboxes.

For this dispatch we are adding a section on database backup results.  ESRP results also report on aggregate and storage group database backups and in this dispatch we publish our first analysis of these results.

We considered reporting on Log write/second but there is a very high correlation between log writes and database transactions per second so find they add little discrimination to performance comparisons.  There are other performance results that are reported in ESRP results and perhaps we will discuss them in a future dispatch on ESRP performance.

The Top 3 normalized Jetstress results for the 1000 to 5000 mailbox category were IBM DS3300 in Clustered Continuous Replication(CCR), HP MSA70 with the P800 controller in CCR and the HPMSA60 with the E500 controller in CCR (see Figure 1).  A few considerations on normalized results:

  • Normalized results can’t always scale well.  For example, although the IBM DS3300 may do well at 1000 mbx it may or may not scale much beyond that.
  • Surprisingly, most subsystems in the top 10 use SAS connections between the Exchange server and storage.  However, the top and bottom results (IBM DS3300 and EMC NS40 respectively) use iSCSI.  There is no FC storage in the top 10 and the top FC result would show up as number 13 in this chart.
  • Another surprise is that the normalized results span a such a large range.  The top 10 normalized results range from over 1800 down to around 570 per 1Kmbx.  This is much larger than the top category of 5000 mailboxes and over.  Not sure why this is more prominent in this category, probably worth some more study but here we just report results.
(SCI080722-002) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080722-002) (c) 2008 Silverton Consulting, All Rights Reserved

We were afraid this chart would show a direct correlation to the number of mailboxes under test. However this is not the case, as top result and the number 4 result (HPMSA70 and Sun StorageTek 2530) report on 4000 mailboxes, while both the Dell and NetApp results in the top 5 were for 5000 mailboxes.  For this category there does not seem to be a direct correlation between mailbox count and database transfers.

The top 3 subsystems in this category are HP MSA70 with P800 controller CCR, HP MSA60 with E500 controller CCR and Dell PowerVault MD3000i with 4000, 2500, and 5000 mailboxes respectively.  A few considerations about un-normalized results:

  • One can easily see significant performance advantages here where the top result is over 2X the nearest competition.  Furthermore any of the other results in the top 5 were over 2X the bottom of the top 10.
  • Here there are a few more iSCSI results three out of the top 10 and two of which were in the top 5.
  • The database transfers to number of mailboxes doesn’t correlate well in this category at all.
(SCI080722-003) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080722-003) (c) 2008 Silverton Consulting, All Rights Reserved

Once again HP seems to dominate this category at the number one and number two slots.  IBM DS3200 with Local Continuous Replication (LCR) comes in third using 2000 mailboxes.  The storage group level backup results have little correlation with the aggregate (total) database backup results.  This may be hinting at one potential advantage to having many storage groups in these benchmarks.

Conclusions

From our view on ESRP results we would conclude that competition is heating up in this category.  There have been a number of recent benchmark results reported and HP seems to have dialed up the performance of their products, over 2X above the nearest competition.  The only one to come close was IBM DS3300 and only in normalized performance.

SAS storage seems to do well in this category for ESRP and seems to be the interface of choice for both un-normalized and normalized performance results.  iSCSI shows up infrequently at best and FC not at all.

This is our second ESRP report we welcome any feedback on how to do better.  Jetstress results are inherently uncomparable.  Hopefully our approach will prove to have merit and if so, look to future ESRP SCI StorInt™ Performance Result Dispatches to follow this lead.

This performance dispatch was sent out to our newsletter subscribers in July of 2008.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at:

SCI 2008 July 22 ESRP performance report update

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.


[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 17 July 2008