Feb 272013
 

This Storage Intelligence (StorInt™) dispatch covers Microsoft Exchange 2010 Solution Review Program (ESRP) v3.0 results.[1]  Fortunately, over the last quarter the there have been at least 14 new submissions in this over 5,000 mailbox category and 23 in all categories but I have lost count how many new submissions since we last discussed this category in January of 2012. As such, just about every chart we report on has some changes.

Latest ESRP V3.0 results

We start our discussion with our normalized database transfers per second per 1000 mailboxes.

Stacked bar chart showing normalized database transfers per second for IBM DS8700, IBM XIV, HDS AMS2100, HP P9500 and others

As can be seen above there are four new submissions that ranked in the top ten since last time.  Specifically, the IBM XIV 2810 (#2), the HDS HUS 110 Dynamic Provisioning (#6), the HDS AMS2300 Dynamic Provisioning (#7), and the HDS USP-V Dynamic Provisioning (#8).

Figure 1 ESRP Top 10 Normalized DB transfers per second 

Recall that this is a normalized metric (using 1000 mailboxes) for the denominator. As such, it tries to account for the fact that larger mailbox counts often generate more database transfers/second (all things being equal).

The only other caveat for our normalized DB/xfers/sec is that Jetstress 2010 parameterizes its IO activity, which in this case ranges anywhere from 0.80 down to 0.18 IO/sec/mbx.  Only the Exchange gods can fathem why this rate should vary so much for this category but it has a direct bearing on rankings for this and other charts.  For instance both of the top two systems used a rate of 0.8 and the next highest rate was a 0.6 used for #6.  One would almost have to normalize again for projected IO rate to better portray system performance.

Next we turn to Exchange Log playback.

Bar chart showing top 10 log playback time for ESRP, with HP4400 EVA, HP P9500, IBM DS8700, IBM SVC5.1, and HDSAMS2100 along with other block storage systems

Figure 2 ESRP Log playback for over 5000 mailboxes

In Figure 2, the only new submission is the Infortrend EonStor at #9. All the others were on the previous version of this chart. In the past Infortrend storage has been OEMed from (NetApp) Engenio.  I am not sure whether this subsystem is another OEM but it could very well be.

We have always liked log playback because it doesn’t correlate well with Jetstreess’s IO rate (see above).  However, it may have some relationship with number of disk drives in the system. For instance, drive counts for this top ten vary from a low of 16 (#9, Infortrend) to a high of 512 (#8, HDS USP-V) and, #1 used 96 drives while #2 used 336 drives.  So, maybe the correlation between log playback time and drive count is not as strong as I suspect.

Not sure how to characterize log playback activity. It is processing database log records and applying them to the various databases so it’s a combination of sequential activity (for the log) and random updates for the databases.  Previously I thought log playback performance might be correlated to the structure of the Exchange databases and their spread across Exchange servers.  And this may indeed be the case because the #1 HP EVA submission had 4 (2 active and 2 passive) Exchange servers for 6000 mailboxes, which seems a bit much.

Next we review total database backup throughput.

Bar chart, top 10 ESRP total database backup throughput storage systems with IBM XIV (twice), HDS USP-V (twice) and EMC CLARiiON among other SAN storage systemsFigure 3 ESRP Total database backup (MB/sec) for over 5K mailboxes

Figure 3 shows another computed metric that represents the total Exchange database backup throughput, across all servers and all databases for a storage subsystem. ESRP reports database backup throughput on a per database or per Exchange server level. But I find from a pure storage perspective it’s better to understand backup throughput for the whole storage system.

In Figure 3 above there are five new submissions, four from IBM, XIV (#1&2), DS5300 (#9) and DCS3700 (#10). I believe the DS5300 and DCS3700 are OEMed solutions.  The only other new submission here is the HDS USP-V (#4) using 32Kmbx.

I view database backup as essentially a highly intensive sequential throughput activity. The fact that XIV Gen3 demonstrated such high throughput here (~12.7 GB/sec) is impressive. Their #2 submission had less disk drives but added SSDs to compensate, but in this case didn’t necessarily perform as well.

Next we analyze database transfers per second on a spindle basis.

Figure 4 ESRP Scatter plot DB/xfers/second against spindle countScatter plot of ESRP results with horizontal as # of disk drives and vertical total Database (read + write) operations per second, Correlation formula y=39.85x+320.17, R**2=0.51962

Finally, in Figure 4 we show the correlation between spindle count and un-normalized database transfer per second.  The regression coefficient is not that good at ~0.5 and there are some significant outliers, especially as they get over ~250 drive spindles.  One caveat here is that we eliminate any systems that use NAND as cache or SSD from this chart. As such, this chart shows disk-only storage system submissions.

We often hear arguments that benchmarks are just a game, where he who throws the most hardware at it wins. The correlation above refutes this notion.  As long as one assumes that the number of drives is a reasonable surrogate for the amount of hardware used in a submission.  Yes it’s missing SSD or FlashCache systems and yes it’s still impacted by Jetstress IO rates, but assuming the former is minimal and the later is set to show the system at its best, then we have a reasonable data set to understand if more hardware wins.  The answer, it appears, is not always.

[Unclear why the formula (probably unreadable here) was different than what was in the original report.  In the original report y=39.852x+471.66 with R**2=0.50328, this version has y=39.85x+320.17 with R**2=0.51962. The only thing I can conclude is that more ESRP reports were added to the spreadsheet after publication. Sorry for any confusion.]

Conclusions

I thought we had seen the last upsurge in ESRP submissions but this past 3 months saw an avalanche. I now believe Microsoft’s upcoming rollout of a new Exchange (sometime in 2013) is forcing all old submissions to be published.

Reiterating sentiment from previous reports, there are more mysteries in ESRP reports than in any other performance results we examine. Also, ESRP/Jetstress storage system performance seems designed to be difficult to compare but in our view, merit the effort.  Indeed, Microsoft would state that ESRP is not a benchmark at all but we feel it’s as true a view of storage performance as any we examine.

Please, feel free to contact us with any constructive ideas on how to improve our ESRP analysis.  We are always open to change our views if that will lead to a clearer picture of system performance.  In that regard, our contact information can be found in the footer below or on our website at SilvertonConsulting.com.

For a more complete explanation of ESRP and for that matter SPC-1 and SPC-2 block storage performance see our recently (August 2013) updated SAN Storage Buying Guide available for purchase on our website.

If you received this report from someone else, please signup for our free monthly newsletter using the QR code below right to receive your own copy. We analyze storage system performance results/benchmarks each month and will analyze Microsoft ESRP results again in three months.

~~~~

[This performance dispatch was originally sent out to our newsletter subscribers in February of 2013.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Dispatches are posted to our website at least a quarter or more after they are sent to our subscribers, so if you are interested in current results you will need to signup for our newsletter.]  

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

 

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 27 February 2013