SCI 2008 Apr 23 first ESRP v2 (Exchange 2007) performance results analysis dispatch – over-5K mailboxes

In Dell EMC, ESRP, ESRP v2/Exchange 2007, HPE, IBM, Symmetrix by Administrator

This is SCI’s first report on Microsoft Exchange Solution Review Program (ESRP)[1] performance results.  SCI found ESRP reports as a group inconsistent and somewhat difficult to compare.   However, upon further examination, we came to a clear understanding of the results and discuss our comparisons below.

Latest ESRP V2.0 results

(SCI080423-001) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080423-001) (c) 2008 Silverton Consulting, All Rights Reserved

ESRP uses a Jetstress benchmark together with a formalized reporting framework to publish results.  Jetstress benchmark variables are many, involving not only mail store database and log layouts, but also the number of Exchange servers and numerous other storage, host and software configuration options.  We have chosen to ignore most of these variables in our analysis and only use those few we deem important for comparing subsystems.

As a starting place this dispatch focuses on results for the 5000 mailboxes and over ESRP category.  This is the top and most challenging category in ESRP reports.  Future SCI performance reports will discuss the 1001 to 5000 mailbox and the 1000 and under mailbox result categories.

Moreover, Jetstress comparisons are confounded by the high correlation between database operations and the number of mailboxes tested.  In order to explore more fully this correlation effect, SCI reports on both normalized and un-normalized results.  For normalized results we use operations per 5000 mailboxes (5Kmbx) in this dispatch.  As an example, EMC Symmetrix originally reported results on 84,000 mailboxes (see Figure 2), but normalized they perform around 2800-database operations/second/5Kmbx. (see Figure 1).

Both the normalized and un-normalized results are for database transfer activity only and do not include log file transfers or other characteristics.   Also ‘DB wt/sec/5Kmbx’ is database write transfers per second per 5000 mailboxes and similarly, ‘DB rd/sec/5Kmbx’ is database read transfers per second per 5000 mailboxes.

The Top 3 normalized Jetstress results were Sun StorageTek 6540, IBM DS4800 and HDS AMS1000 storage subsystems reporting on 20-, 8-, and 25-Kmbx respectively (see Figure 1).  A few considerations on normalized results:

  • Normalized results can’t always scale all the way up to 50Kmbx.  For example while the HP MSA60 may do well at 6Kmbx it is unlikely to scale much beyond that.  On the other hand, the NetApp FAS3070 FC should scale well up to 26Kmbx.
  • Most subsystems in the top 10 use FC interfaces, the lone exception being number 4, HP MSA60 that is SAS attached.  The closest iSCSI attached subsystem came in at number 11, the DELL PowerVault MD3000i.
  • Overall the top 10 performance results span a small range, about 600-database transfers/second/5Kmbx.  The span for the top 10 un-normalized results is much larger.
(SCI080423-002) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080423-002) (c) 2008 Silverton Consulting, All Rights Reserved

As shown in the above chart un-normalized results suffer somewhat from mailbox correlation – the more mailboxes under test the higher the database operations achieved.  Although not all rankings are a function of mailbox counts, certainly the top 3 ranked results show the impact of more mailboxes.

The top 3 subsystems in this category are EMC Symmetrix, DELL Equal Logic, and Lefthand SANIQ® at 84-, 65- and 50-Kmbx respectively.  A few considerations about un-normalized results:

  • There seems to be a performance clustering around 20Kmbx as shown in the last 7 of the top 10.  The performance range for this group of results narrows, only around 6000-database transfer difference from highest to lowest.  Also, the mailbox-database transfer correlation breaks down for this group.
  • Here iSCSI results compare well against FC subsystems (e.g., see both FAS3070 results above).
  • Equal Logic and Lefthand results are also iSCSI so for the top 10 un-normalized results, 7 are FC and the rest, iSCSI.
  • We might add that these iSCSI results were with Gige interfaces.  The closest 10Gbe iSCSI result came in at number 14 for the Intransa IP SAN subsystem.

Conclusions

It seemed like normalized results was the only valid comparison to use but total database results were also of interest, especially where the number of mailboxes are similar.  Also, ESRP reports on other activity, namely log writes, database backups, latencies, and log replays.  SCI chose to focus on database activity because it seemed most valid for comparing normal Exchange mailbox activity.  Perhaps future versions of this dispatch will report on some of these other reported results as well.

As this is our first ESRP report we welcome any feedback on how to do better.  Jetstress results are inherently difficult to compare.  Hopefully our approach will prove to have merit and if so, look to future ESRP SCI StorInt™ Performance Result Dispatches to follow this lead.

This performance dispatch was sent out to our newsletter subscribers in April of 2008.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.A PDF version of this can be found at

SCI 2008 April 23 First performance results dispatch on ESRP results

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community


[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 23 April 2008