SCI 2009 Jul 28 latest ESRP results analysis – 1K-to-5K mailboxes category

In DS3300, ESRP, ESRP v2/Exchange 2007, HPE, IBM, MSA2000 by Administrator

This is SCI’s third report in our new series on Microsoft Exchange Solution Review Program (ESRP)[1] performance results and reports on the 1000 and under mailbox results category.  Prior reports discussed the 1001 to 5000 mailboxes and the over 5000 mailboxes result categories[2].  For this class of results we no longer provide normalized results.

ESRP was never intended to compare subsystem performance but rather as a proof of concept for Microsoft and storage vendors to depict a configuration supporting a given workload.  Hence, any comparisons necessarily come with some caveats and may not be real.  Nonetheless, SCI feels comparisons can well serve both the vendor and end-user storage community and thus, worth noting.

Latest ESRP V2.0 results

We now use a ranking for this analysis depicting the average database backup throughput across all storage groups, which roughly corresponds to an IT or datacenter view of Exchange performance.  This value correlates moderately to aggregate database transfers per second.  (See figure 1).

(SCIESRP090728-001) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090728-001) (c) 2009 Silverton Consulting, All Rights Reserved

We double checked our data and have determined that HP StorageWorks MSA 2000fc G2 is tied for first place with Dell PowerEdge 2900 III and the Dell PowerEdge 3000 comes in at third place.  All these results supported 1000 mailboxes except the EMC NS20, which only had 700 mailboxes.  Also the MSA result used FC while the others in the top four used SAS and fifth place used iSCSI.  A couple of caveats worth noting here database backup performance can be impacted by

  • Number of disk drives in a configuration
  • How message store databases are split across those spindles
  • Subsystem RAID level
(SCIESRP090728-002) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090728-002) (c) 2009 Silverton Consulting, All Rights Reserved

Another new chart we added shows the ESRP reported database latencies for read, write and log write operations.  SCI and others feel the read latency metric may best show end-user experience attainable from a storage subsystem configuration.  The list is sorted by read latency.  A couple of considerations to note:

  • While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated.
  • There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage.  As of V2.1, all ESRP results now publish the number of simulated IOPS/mailbox used during the Jetstress run and as more V2.1 results become available we may be better able to detect gaming of latency factors, if present.  At the moment, ESRP V2.1 reports are scarce and only one of the top 10 have reported on this value.
  • For an ESRP benchmark to be accepted, read latency must be under 20 msecs.  Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware.  Such vendors may not show up well on a top 10-latency ranking.
(SCIESRP090728-003) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090728-003) (c) 2009 Silverton Consulting, All Rights Reserved

We have also started to examine the log playback ranking.  This is a rather complex workload that encompasses log reading, database reading and updating (or writing).  The timings are reported as the average time in seconds it takes to playback or process a 1 MB log.  The top 3 systems are Huawei-Symantec’s OceanStor S5500, Dell’s PowerEdge 2900 III, and EMC CX-3-20.  The range for the top ten subsystems is not large, ~0.39 to 0.55 seconds from lowest to highest.  It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations.  Similar to the backup discussions above, some caveats would include:

  • The number of disk drives can significantly impact playback performance.
  • How message store databases are split across those spindles also can impact this
  • Subsystem RAID level may also impact playback performance
  • Replication type may also impact log playback performance
(SCIESRP090728-004) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090728-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to overall database transfer results and the Top 3 un-normalized ESRP/Jetstress results belong to HP MSA 2000fc G2, IBM DS3300, and Dell PowerEdge 2900 III. A few considerations are warranted on un-normalized results:

  • While all these results were for 1000 mailboxes, it’s unlikely that these systems will scale much above this.
  • One surprise here is the close running of everyone behind the top three results and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox.  We saw a similar narrow result range for our last report on the over 5Kmbx submissions as well.

Conclusions

From our perspective, ESRP results in this 0 to 1K-mailbox tier are becoming less interesting.  There were only a few new ESRP results in this category over the last 9 months, and only one over the last quarter.  Most new ESRP submissions have been in the over 5Kmbx category and will be discussed in next quarters ESRP review.

However, it is interesting to see FC, SAS, and iSCSI subsystem results in this category all fighting for prominence.  One would think that FC would have a natural advantage across the board here but iSCSI and SAS also place very well at this level.  Such competitiveness probably says real world workloads (like Exchange) are less dependent on interface protocols and more dependent on subsystem architecture or other factors.

ESRP/Jetstress results are inherently difficult to compare.  Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases.  Our next ESRP/Jetstress report will return to over 5K-mailbox tier.  We continue to welcome any feedback on how to do better.

This performance dispatch was sent out to our newsletter subscribers in July of 2009.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 July 28 Update to ESRP performance results

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community


[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 28 July 2009

 

[2] All prior SCI ESRP Dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html