This Storage Intelligence (StorIntTM) dispatch covers recent Microsoft Exchange 2010 Solution Review Program (ESRP)¹ v3.0 results. Since we last discussed this over-5K mailbox category nine months ago, there have been at least thirteenn new ESRP submissions. Future dispatches will report on the 1K-to-5K and 1K-and-under mailbox categories, but all previous ESRP V2 and ESRP V3 performance dispatches are ultimately available on SCI’s website².
Latest ESRP V3.0 results
We start our ESRP analysis with Exchange database backup throughput.
Figure 1 Top ESRP database backup MB/sec.
For Figure 1, every result is new to this analysis. Recall that ESRP backup throughput is not directly reported in ESRP but we calculate this by taking the backup throughput/DB times the number of DB in a configuration. We prefer this metric that provides an overall database backup for a storage subsystem to the ESRP reported values. However, the number and speed of disks will impact this metric. For example, the EMC VMAX (#7) used 7200rpm disks while their CLARiiON (#2) used 10Krpm disks.
But I am more perplexed with the HDS USP-V results (#1,3&8) that used 512-2TB 7200rpm, 512-600GB 15Krpm and 512-2TB 7200rpm disks, respectively. We would have thought the two results with 7200rpm disks would have been closer and should have been worse than the 15Krpm disk result given that they all used the same disk count. However, the number of databases was significantly different, the two 7200rpm (#1&8) had ~192 databases each while the 15Krpm system had 64 databases. It seems that the database count multiplier effect has overcome the disadvantage in disk speed. Not sure I like this, faster disks should provide better backup throughput. I may have to reconsider this metric.
We now turn to ESRP database transfers per second.
Figure 2 Top ESRP Database transfers per second
In Figure 2 above, only the IBM DS8700 and XIV were in the prior version of this analysis, all the rest are new results. Note that these results are un-normalized and as such, may be unduly influenced by mailbox count. Nonetheless, it’s intriguing that the top 3 systems, i.e., the HP 3PAR T800, Dell EqualLogic PS series and the IBM DS8700 supported 140K, 75K and 20K mailboxes respectively. Maybe mailbox counts are not the only factor in database transfers/second results.
The other interesting aspect on this chart is the great performance by HP’s 3PAR T800 storage system (#1) whose database read transfers/second alone is better than every other subsystem’s combined read and write transfers/second. It probably says that wide stripping provides a big advantage for random database transaction workloads like Exchange. [Full disclosure: recently we were under contract to HP to discuss their 3PAR T800 results in a separate white paper.]
Another way to look at database transfers activity is to do a scatter plot of total database transfers/second against spindle count.
Figure 3 Scatter plot: total database transfers/sec vs. spindle count
Figure 3 above excludes any system that uses SSDs or FlashCache in this category. We have created similar charts for our other performance analyses but this is a first for ESRP. Many analysts believe that great benchmark results are achieved by throwing more hardware at them. But the R**2 of 0.4835 on the linear regression line proves them wrong, at least for ESRP results. The low regression coefficient implies that system sophistication has a significant effect on ESRP performance. Similarly, all the outliers (above the line) around 20K and the lone wolf around 60K database transfers/second, almost the entire top 10 DB transfers/second seems to reiterate that sentiment, i.e. system sophistication matters in Exchange 2010 database transfer performance.
Next we turn to another of our perennial favorites, Log Playback Time.
Figure 4 Top 10 log playback times
In Figure 4 lower is better. The HP 4400EVA, HP P9500 G2 and the HDS USP-V (#1, 2 & 8) are new results to this analysis. How the 4400EVA managed a log playback around 110 msecs. is beyond me. However, we should say all but #2 and 9 used 15Krpm disks, and the top 2 results both had the smallest capacity disks (300GB).
One would think that number of disks (or SSD/FlashCache use) might be a factor here but disk counts run from 24 (#6 HP P411) to 512 (#8 HDS USP-V) and the only use of NAND was #9 (NetApp FAS3140). So once again we are puzzled by the results seen here.
Finally, we discuss Database and Log Access Latencies.
Figure 5 Top database access latencies
The only new submission in Figure 5 above is HDS USP-V at #2. Recall that this chart is sorted by read latency and lower is better. We believe database latency correlates well to Exchange end user experience. Yes there is an awful lot of buffering that goes on in Exchange Servers, Outlook clients, etc., but we firmly believe that database latency does impact how quickly end users can process emails, especially for heavily loaded systems.
The IBM DS8700 shines here with the #1 read latency, #3 write latency and #4 log latency but the #2 HDS USP-V didn’t do as well in write latency (#8) and log latency (#7).
There are more mysteries in ESRP results than in any other performance benchmark we analyze. Our metric on total database backup throughput is not as well behaved, as we would like. Similarly, we have no great explanation for database transfers per second per spindle results. The fact that there are some open questions on ESRP performance results just makes them more interesting, at least to us.
Why the over 5K mailbox category has been so busy is yet another riddle. It seems just about every major storage vendor has released ESRP results for their high-end storage over the past 9 months. Of course when the ESRP activity is completed and when it’s published on the ESRP website are two different events.
Finally, as discussed in prior dispatches, ESRP/Jetstress results seem destined to be difficult to compare but in our view, merit the effort. As such, feel free to contact us with any constructive ideas on how to improve. In that regard, our contact information can be found below or on our website at SilvertonConsulting.com.
[This performance dispatch was originally sent out to our newsletter subscribers in January of 2012. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more, in-depth analysis of SAN storage system features and performance please see our SAN Storage Buying Guide available on our website.]
Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community
1 ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as Silverton of 25 January 2012
2 All prior SCI ESRP Dispatches can be found at StorIntTM dispatch web page http://silvertonconsulting.com/cms1/news-4/