SCI 2012Jul24 Microsoft ESRP performance results analysis

In Dell EMC, ESRP, ESRP v3/Exchange 2010, FAS2040, FAS2220, NetApp, VNX, VNXe, VNXe 3100, VNXe 3300by AdministratorLeave a Comment

This Storage Intelligence (StorInt™) dispatch covers recent Microsoft Exchange 2010 Solution Review Program (ESRP)[1] v3.0 results. Unfortunately, since we last discussed this 1K-and under mailbox category nine months ago, there has been only one new submission from NetApp on their FAS2220.  But for some reason all of Xiotech’s submissions have been removed from the ESRP website so they have been deleted from our charts. As their 500 mailbox submission was in all prior top ten charts in this category, all chart rankings have changed.  Future dispatches will report on the over-5K and 1K-to-5K mailbox categories, but all previous ESRP V2 and ESRP V3 performance dispatches are available on SCI’s website[2].

Latest ESRP V3.0 results

We start our ESRP analysis with Exchange database transfers/second/spindle.

(SCIESRP120724-001) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

(SCIESRP120724-001) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

Figure 1 Top ESRP database transfers per second per disk spindle

As discussed above, Xiotech’s submission has been dropped from our charts, it had been #1 here, so all others have moved up.  In addition, previously we had mislabeled EMC’s VNXe3300 as VNXe3100, which has been fixed in these charts. The new NetApp FAS2220 submission came in at #6 here.

We now turn to average Log Playback Time results.

(SCIESRP120724-002) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

(SCIESRP120724-002) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

Figure 2 Top ESRP Log playback time

In Figure 2 lower is better.  The new NetApp FAS2220 came in at #9 on this measure. We have always been big fans of log playback time because ESRP performance is so configuration driven. Log playback is just 1MB of log data that is played back against the database as fast as a storage system can process it.  ESRP/Jetstress parameters don’t seem to have much impact on this performance metric.

One thing I find interesting here is that the NetApp FAS2040 (previous generation storage) was running slower disks (1TB/7200rpm) with 8 spindles while the NetApp FAS2220 used faster drives (600GB/10Krpm) with 7 spindles.  In the rankings above the FAS2040 came in at #1 while the FAS2220 came in at #9.  This probably says something about the random nature of log playback and that having more spindles can sometimes trump faster disks.  Also, I must add that the FAS2220 storage actually is designated below the FAS2040 in NetApp’s FAS product family so it’s probably all right for it to rank slower here.  The next generation version of the FAS2040 is the new FAS2240 that we have yet to see in ESRP results.

We now show (un-normalized) database transfers per second.

(SCIESRP120724-003) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

(SCIESRP120724-003) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

Figure 3 Top ESRP database transfers per second (un-normalized)

Figure 3 shows the Top 10 database transfer performance results.  The new NetApp FAS2220 came in at #6 and the FAS2040 didn’t even place in the top 10 on this metric.

A few caveats are needed to understand this measure.  We normally prefer normalized numbers to this un-normalized metric as with all ESRP/Jetstress runs, mailbox counts can significantly affect database transaction performance.

One Jetstress parameter of specific interest here is IO rate. Five of these top ten used an IO rate of 0.24 IO/sec/mailbox, three more used 0.18 and the NetApp FAS2220 used the least of all, at 0.12 IO/sec/mailbox.  Nonetheless, it’s unclear how much the ESRP/Jetstress IO rate governs database transfer performance as most submissions in this category exceed their targeted IO rate by 50% or more.

Next, we return to another of our perennial favorites, database access latencies.

(SCIESRP120724-004) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

(SCIESRP120724-004) (c) 2012 Silverton Consulting, Inc., All Rights Reserved

Figure 4 Top 10 Database read access latency

In Figure 4 lower is better. This chart is sorted by read access latency but we show write and log latency as well.    The NetApp FAS2220 and a few others systems showed significant variance, over 8X longer read as compared to write latencies.  We attribute this difference to the use of cache for write (using fast writes) and log write activity.  This is great as long as your Exchange email working set (read and write data) can fit inside cache.

But the other thing this points out is potentially better caching.  We would need to do a comparison of ESRP/Jetstress data throughput normalized by the amount of cache memory present in a submission in order to understand this better.

Conclusions

For some reason this category doesn’t seem to generate as much interest as the higher categories (1K-to-5K and over-5K).  However, with both EMC and NetApp now active in this low-end segment of the market, it seems like this category should be getting hotter.  Of course, HP, IBM, Dell and others are also engaged here as well.  In any case, there were at least 8 new over-5Kmbx submissions this past quarter so next quarter’s ESRP analysis should show a bit more change.

Not sure why Xiotech’s submissions were recently dropped from ESRP. They used to have 3 submissions one with 4Kmbx, one with 3Kmbx and the one in this category at 0.5Kmbx.  I can only surmise this is per X-IO’s request and is in anticipation of newer submissions coming out shortly.

I must apologize for any confusion from my mislabeling of the EMC VNXe3300 submission.  Not sure how that happened but it’s fixed here and will be correct from this point forward.

In addition, our recently revised (May 2019)  SAN Storage Buying Guide provides even more in depth analysis of top 20 ESRP and SPC results.  We now include our SAN Storage ChampionsCharts™ that ranks system performance across a set of metrics exclusive to SCI  that we track.  The buying guide can be purchased from our web site.

Reiterating sentiment from previous reports, there are more mysteries in ESRP results than in any other performance benchmark we look at. Also, ESRP/Jetstress results seem designed to be difficult to compare but in our view, merit the effort.  As such, feel free to contact us with any constructive ideas on how to improve.  In that regard, our contact information can be found below or on our website at SilvertonConsulting.com.  

~~~~

[This performance dispatch was originally sent out to our newsletter subscribers in July of 2012.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports.  

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.


[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 24 July 2012

[2] Prior SCI ESRP Dispatches available at http://silvertonconsulting.com/cms1/news-4/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.