Latest ESRP results for 1K and under mailboxes – chart of the month

SCIESRP120724(004) (c) 2012 Silverton Consulting, All Rights Reserved

The above chart was from our July newsletter Exchange Solution Reviewed Program (ESRP) performance analysis for 1000 and under mailbox submissions. I have always liked response times as they seem to be mostly the result of tight engineering, coding and/or system architecture.  Exchange response times represent a composite of how long it takes to do a database transaction (whether read, write or log write).  Latencies are measured at the application (Jetstress) level.

On the chart we show the top 10 data base read response times for this class of storage.  We assume that DB reads are a bit more important than writes or log activity but they are all probably important.  As such,  we also show the response times for DB writes and log writes but the ranking is based on DB reads alone.

In the chart above, I am struck by the variability in write and log write performance.  Writes range anywhere from ~8.6 down to almost 1 msec. The extreme variability here begs a bunch of questions.  My guess is the wide variability probably signals something about caching, whether it’s cache size, cache sophistication or drive destage effectiveness is hard to say.

Why EMC seems to dominate DB read latency in this class of storage is also interesting. EMC’s Celerra NX4, VNXe3100, CLARiiON CX4-120, CLARiiON AX4-5i, Iomega ix12-300 and VNXe3300 placed in the top 6 slots, respectively.  They all had a handful of disks (4 to 8), mostly 600GB or larger and used iSCSI to access the storage.  It’s possible that EMC has a great iSCSI stack, better NICs or just better IO scheduling. In any case, they have done well here at least with read database latencies.  However, their write and log latency was not nearly as good.

We like ESRP because it simulates a real application that’s pervasive in the enterprise today, i.e., email.  As such, it’s less subject to gaming, and typically shows a truer picture of multi-faceted storage performance.

~~~~

The complete ESRP performance report with more top 10 charts went out in SCI’s July newsletter.  But a copy of the report will be posted on our dispatches page sometime next month (if all goes well).  However, you can get the ESRP performance analysis now and subscribe to future free newsletters by just using the signup form above right.

For a more extensive discussion of current SAN block system storage performance covering SPC (Top 30) results as well as ESRP results with our new ChampionsChart™ for SAN storage systems, please see SCI’s SAN Storage Buying Guide available from our website.

As always, we welcome any suggestions or comments on how to improve our analysis of ESRP results or any of our other storage performance analyses.


ESRP v3 (Exchange 2010) log playback results, 1Kmbox&under – chart-of-the-month

(SCIESRP111029-003) (c) 2011 Silverton Consulting, All Rights Reserved
(SCIESRP111029-003) (c) 2011 Silverton Consulting, All Rights Reserved

The above chart is from our last Exchange [2010] Solution Review Program (ESRP) performance dispatch released in our October newsletter (sign-up upper right).  The 1K mailbox and under category for ESRP represents Exchange storage solutions for SMB data centers.

As one can see from the above the NetApp FAS2040 has done well but an almost matching result came in from the HP P2000 G3 MSA system.  What’s not obvious here is that the FAS2040 had 8 disks and the P2000 had 78 so there was quite a difference in the spindle counts. The #3&4 runs from EMC VNXe3100 also posted respectable results (within 1sec of top performer) and only had 5 and 7 disks respectively, so they were much more inline with the FAS2040 run.  The median number of drives for this category is 8 drives which probably makes sense for SMB storage solutions.

Why log playback

I have come to prefer a few metrics in the Exchange 2010 arena that seem to me to capture a larger part of the information available from an ESRP report.  The Log Playback metric is one of them that seems to me to fit the bill nicely.  Specifically:

  • It doesn’t depend on the Jetstress IO/rate parameter that impacts the database transfers per second rate.  The log playback is just the average time it takes to playback a 1MB log file against a database.
  • It is probably a clear indicator of how well a storage system (configured matching the ESRP) can support DAG log processing.

In addition, I believe Log Playback is a great stand-in for any randomized database transaction processing. Now I know that Exchange is not necessarily a pure relational database but it does have a significant component of indexes, tables, and sequentiality to it.

My problem is that there doesn’t seem to be any other real database performance benchmark out there for storage.  I know that TPC has a number of benchmarks tailored to database transaction activity but these seem to be more a measure of the database server than the storage.  SPC-2 has some database oriented queries but it’s generally focused on through put and doesn’t really represent randomized database activity and for other reasons it’s not as highly used as SPC-1 or ESRP so there is not as much data to report on.

That leaves ESRP.  For whatever reason (probably the popularity of Exchange), almost everyone submits for ESRP. Which makes it ripe for product comparisons.

Also, there are a number of other good metrics in ESRP results that I feel have general applicability outside Exchange as well.  I will be reporting on them in future posts.

~~~~

Comments?

Sorry, I haven’t been keeping up with our chart-of-the-month posts, but I promise to do better in the future.  I plan to be back in synch with our newsletter dispatches before month end.

The full ESRP performance report for the 1K and under mailbox category went out to our newsletter subscriber’s last October.  A copy of the full report will be up on the dispatches page of our website sometime this month (if all goes well). However, you can get performance information now and subscribe to future newsletters to receive these reports even earlier by just sending us an email or using the signup form above right.

For a more extensive discussion of block storage performance in ESRP (top 20) and SPC-1&-2 (top 30) results please consider purchasing our recently updated SAN Storage Buying Guide available on our website.

As always, we welcome any suggestions on how to improve our analysis of ESRP results or any of our other storage system performance discussions.

 

Recent ESRP v3.0 (Exchange 2010) performance results – chart of the month

SCIESRP110127-003 (c) 2011 Silverton Consulting, Inc., All Rights Reserved
SCIESRP110127-003 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

We return to our monthly examination of storage performance, and this month’s topic is Exchange 2010 performance comparing results from the latest ESRP v3.0 (Exchange Solution Review Program).  This latest batch is for the 1K-and-under mailbox category and the log playback chart above represents the time it takes to process a 1MB log file and apply this to the mailbox database(s).  Data for this report is taken from Microsoft’s ESRP v3.0 website published results and this chart is from our latest storage intelligence performance dispatch sent out in our free monthly newsletter.

Smaller is better on the log playback chart.  As one can see it takes just under 2.4 seconds for the EMC Celerra NX4 to process a 1MB log file whereas it takes over 7.5 seconds on an EMC Iomega IX12-300r storage subsystem.  To provide some perspective in the next larger category, for storage supporting from 1K-to-5K mailboxes,  the top 10 log playback times range from ~0.3 to ~4.5 seconds and as such, the Celerra NX4 system and the other top four subsystems here would be in the top 10 log playback times for that category as well.

Why log playback

I have come to believe that log playback is an important metric in Exchange performance, for mainly one reason, it’s harder to game using Jetstress paramaterization.   For instance, with Jetstress one must specify how much IO activity is generated on a per mailbox basis, thus generating more or less requests for email database storage. Such specifications will easily confound storage performance metrics such as database accesses/second when comparing storage. But with log playback, that parameter is immaterial and every system has the same 1MB sized log file to process as fast as passible, i.e., it has to be read and applied to the configured Exchange database(s).

One can certainly still use a higher performing storage system, and/or throw SSD, more drives or more cache at the problem to gain better storage performance but that also works for any other ESRP performance metric.  But with log playback, Jetstress parameters are significantly less of a determinant of storage performance.

In the past I have favored database access latency charts for posts on Microsoft Exchange performance but there appears to be much disagreement as to the efficacy of that metric in comparing storage performance (e.g., see the 30+ comments on one previous ESRP post).  I still feel that latency is an important metric and one that doesn’t highly depend on Jetstress IO/sec/mailbox parameter but log playback is even more immune to that parm and so, should be less disputable.

Where are all the other subsystems?

You may notice that there are less than 10 subsystems on the chart. These six are the only subsystems that have published results in this 1K-and-under mailbox category.  One hopes that the next time we review this category there will be more subsystem submissions available to discuss here.  Please understand, ESRP v3.0 is only a little over a year old when our briefing came out.

—-

The full performance dispatch will be up on our website after month end but if one needs to see it sooner, please sign up for our free monthly newsletter (see subscription widget, above right) or subscribe by email and we’ll send you the current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of block storage performance please consider purchasing SCI’s SAN StorInt Briefing also available from our website.

As always, we welcome any constructive suggestions on how to improve any of our storage performance analyses.

Comments?