SCI 2011 Apr 28 Latest Microsoft ESRP v3.0 (Exchange 2010) results

Apr 282011

This Storage Intelligence (StorInt™) dispatch covers recent Microsoft Exchange Solution Review Program (ESRP)[1] v3.0 results for Exchange 2010. Since the last time we discussed this over-5K mailbox category nine months ago, there have been more than a dozen new submissions. Future dispatches will report on the under-1K and 1K-to-5K mailbox categories, but all prior ESRP V2 and ESRP V3 performance dispatches are available on SCI’s website[2].

Latest ESRP V3.0 results

We start our ESRP analysis with Exchange database and log access latency results. Recall that this chart is sorted by database read latencies.

SCIESRP110428-001 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 1 Top ESRP database-read latencies

For Figure 1, the new IBM DS8700 takes top honors with HDS AMS systems taking then next 4 slots followed by HP. The IBM DS8700 is the latest hardware from IBM for open and mainframe enterprise class storage and used 128-450GB, 15Krpm FC attached drives with 114.6GB of cache. In contrast, the (#6) HP and (#10) Compellent both used SAS 450GB 15Krpm drives with 24 and 66 drives respectively.

There has been quite a lot of commentary on my blog[3] about ESRP’s latency numbers. What’s clear is that Jetstress has a cut off at 20 msec. and as long as an ESRP result fits within that latency it’s considered acceptable by Microsoft. I argued that even though that’s the case superior latency is still something to be strived for regardless of the maximal cutoff. Other parties disagreed and said something to the effect that anything less than 20msec is a passing grade. Nevertheless, that the DS8700 had almost ½ the read latency of the Compellent system is significant and worth noting.

We now turn to absolute ESRP database transfers per second.

SCIESRP110428-002 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 2 Top 10 Database transfers per second

There are three new submissions on this chart namely, the (#1) IBM DS8700, their Storwize v7000 subsystem (#7) with 60K mailboxes and the HDS AMS 2500 with 20.9K mailboxes (#10). We should note that the Storwize system had (24) SSDs in their configuration while no one else in this top 10 used FlashCache or SSDs.

Normally, at this mailbox level we don’t show absolute transferst because the spread is so wide. For example, we have from 20K (#1 IBM DS8700) to 100.8K mailboxes (#3 HDS AMS 2500) on display here. But there is continued interest in absolute numbers and as such, report them occasionally.

On the other hand, we always report on the normalized ESRP database transfer counts per 1K mailboxes (see Figure 3 below).

SCIESRP110428-003 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 3 Top normalized DB (per 1K mailboxes) transfers per second

IBM’s DS8700 system showed up well here but used a 0.80 IOs/mailbox Jetstress driver parameter. In contrast, the #2, HDS AMS 2100 system used 0.18 IOs/mailbox that just happened to be the median for these top 10 systems.

Unclear what advantage having more or less IOs/mailbox would be but if one’s subsystem can handle more workload, probably raising the IO rate or reducing the mailbox count would help this ranking. Naturally, having both helps even more.

Next we examine log playback performance.

SCIESRP110428-004 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 4 Top log playback times

Over time I have come to better appreciate a select set of ESRP metrics and the Log playback times is one of them. Log playback is the average time it takes a storage system to process a number of 1MB log files applied to a single database.

We assume that all 1MB log files contain the same number of transactions. Possibly the number of mailboxes per database might impact this processing (for these top 10, the numbers range from 250 to over 1400 mailboxes per database). Certainly, how a single database is spread across drives (and/or SSDs) and system cachinge size/effectiveness should impact this value.

In any event, it simulates a pretty complex, database workload, driven as fast as the storage can handle it. Whereas, many of the other ESRP workloads are all driven by the IO/mbx/sec, number of mailboxes, and/or other Jetstress parameters.

Such results as shown above (see Figure 4) probably says that the IBM DS8700 is a pretty good subsystems to process Exchange-like, database transactions.

Another of my favorite ESRP reported metrics is DB backup activity. There are two measurements reported in ESRP for database backup activity 1) Database data backed up in MB/sec/database and 2) Database data backed up in MB/sec/server. For reasons of brevity, we shall only discuss the later and save the former for another time.

SCIESRP110428-005 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 5 Top 10 database backup activity in MB/second/server

Figure 5, shows somebody other than IBM DS8700 in top place, the Dell PowerVault MD1200 with 24-2TB SAS connected, 7200RPM drives. One surprising result here is that Dell (now that it owns Compellent) has five of the top seven results on this metric. The other surprise is that seven of the top ten on this chart used SAS connected disks. It’s apparent that the drive count does not impact this particular metric.

As stated in prior Dispatches using Exchange 2010’s DAGs and lagged databases has reduced the need for backups. But, we firmly believe there are many valid reasons for database backups and as such, an ongoing need for mailbox backup performance will remain. Thus, we continue to report on this valuable metric.

SCIESRP110428-006 (c) 2011 Silverton Consulting, Inc., All Rights Reserved

Figure 6 Top 10 DB xfers/sec/spindle

Finally, our last chart shows the top ten per drive performance in total database transfers/sec/spindle. We exclude any subsystem with FlashCache or SSDs from this analysis to get a pure per drive view of performance. We see that Dell PowerVault MD1200 and IBM’s DS8700 has done well here again.

Conclusions

Well this is our second ESRP analysis on the over-5K mailbox category and we have quite a few new entries for this report. Nonetheless, the obvious ESRP winner in this mailbox level was the IBM DS8700. This system is the latest generation of enterprise storage hardware that IBM has been delivering since the ESS (AKA, the “shark”) was retired and seems to be well represented.

Finally, as discussed in prior dispatches, ESRP/Jetstress results seem destined to be difficult to compare but in our view, merit the effort. Thus, we strive to refine our analysis with each report. As always, feel free to contact us with any ideas on how to improve. In that regard, our contact information can be found below or on our website at SilvertonConsulting.com.

[This performance dispatch was originally sent out to our newsletter subscribers in April of 2011. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.]

—–

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 28 April 2011

[2] All prior SCI ESRP Dispatches can be found at http://silvertonconsulting.com/cms1/dispatches/

[3] http://www.RayOnStorage.com

SCI 2010 Oct 25 latest Microsoft ESRP v3 (Exchange 2010) results – 1K to 5K mboxes

Oct 252010

This Storage Intelligence (StorInt™) dispatch covers recent Microsoft Exchange Solution Review Program (ESRP)[1] V3 results for Exchange 2010. There have been a large number of submissions this past quarter with at least five new ones in the 1K-to-5K mailbox category discussed below. Future dispatches will report on the 1K-and-under and the over-5K mailbox categories while previous ESRP V2 and ESRP V3 analysis are available on SCI’s website[2].

Latest ESRP V3.0 results

We start our ESRP analysis with Exchange database and log access latency results. Recall that this chart is sorted by the database read latencies.

SCIESRP101025-001 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 1 Top 10 ESRP database-read latencies

For Figure 1, the Xiotech Emprise 5000 and HP StorageWorks 2000sa take top honors with the latest StorageWorks P2000 SAS G3 subsystem coming in a close 3^rd. Note that the #5 result for Xiotech used 500GB 10Krpm drives while their #1 result used 350GB 15Krpm drives. As discussed previously, the other anomaly with Xiotech’s #1 result was that it provided no database redundancy, probably a rare configuration for most users.

Other odd results on this chart were for the HP Smart Array, and both Dell PowerEdge R510 subsystems (#7, 6 and 10 respectively). They all had great database and log write latencies but poor read latencies. It may have something to do with the size of their drives 932GB (for HP) and 2TB (for Dell subsystems), drive speed -7200 RPM and the few number of disk spindles (24, 6 and 12 respectively) used in the testing (being able to cache database and log writes but unable to service random database reads). However, the #1 Xiotech subsystem also had only 20 drives but used 15Krpm drives.

Next we turn to ESRP database transfer counts for the 1K-to-5K mailbox category which span such a wide spectrum that we have normalized results to accesses per 1,000 mailboxes (1Kmbx).

SCIESRP101025-002 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 2 Top 10 normalized DB transfers per second

Surprisingly, even normalized results look pretty skewed. For Figure 2, the #1 result, by a wide margin, was the HP StorageWorks P2000 SAS G3 subsystem. Unclear just what HP did right here but using SAS probably helped. Their #3 result was a similar HP P2000 G3 subsystem but was attached to Exchange servers via FC. Both subsystems used the same 300GB 15Krpm drives, a similar number of drives (90 and 94, for the #1 and #3) respectively), and similar IO intensity (0.18 IO/sec) but attained vastly different results. The only other significant change (outside of interface used and mailbox counts) was the size of the mailboxes (2.5GB and 1GB respectively). Why the smaller mailbox size would generate such poor results in comparison is perplexing to say the least.

A couple of caveats for normalized results:

Normalized results may or may not scale much beyond the reported mailbox counts. For example, the #1 result only supported 2K mailboxes and may not handle much more than that.
Normalized results can be impacted by over provisioning. For example, the #1 result only used ~53% of its storage for Exchange services allowing it to use more spindles to spread the workload.

SCIESRP101025-003 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 3 Top 10 data base transfers per spindle

Speaking of over provisioning, another way to look at Exchange storage performance is to normalize it over the number of disk drives or spindles used for the solution. Figure 3 shows the total number of (read and write) database operations per second per drive done by each subsystem. It’s surprising to see a dead heat here for the top spot. Realize that the HP subsystem (#2) used SAS and the Xiotech used FC attached storage and both subsystems sported 15Krpm drives.

Some caveats for database transfers per spindle results:

Drive speed can affect this metric, i.e. faster drives perform better and four of the top six performers used 15Krpm drives.
Drive over provisioning usually harms one’s performance on this metric. However this did not seem to hurt the #1 Xiotech and #2 HP results which used 48% and ~53% of storage for database space. In contrast, the #3 Xiotech result used ~87% of its disk storage for databases.

SCIESRP101025-004 (c) 2010 Silverton Consulting, Inc., All Rights Reserved

Figure 4 Top 10 database backup activity in MB/second/database

Finally, Figure 4 shows Exchange 2010 backup throughput rates in MB/sec/database. Please note that the top four solutions here all were SAS connected storage. Of the remaining subsystems, there were two more SAS and the remainder, were all FC attached storage.

With Exchange 2010’s mailbox resiliency through DAGs, database backup activity no longer seems important. Indeed, there was at least one submission that didn’t even report on this metric. However, there are many valid reasons for database backups and we continue to believe that a need for mailbox backups will persist. As such, we will continue to report on this metric wherever we deem it valuable to our analysis.

Conclusions

There have been a limited number of ESRP V3 submissions to date – it’s still too early. Nonetheless, to date there have been just 11 submissions in this 1K-to-5K mailbox category and just over 37 total reports, with more coming out each quarter. For all categories there were 15 new submissions since our last report. Unfortunately, most of these were in the over 5K mailbox category to be covered in a future dispatch.

Finally, ESRP/Jetstress results seem destined and designed to be difficult to compare but merit the effort. Thus, we strive to refine our analysis with each report. As always, feel free to contact us with any ideas on how to improve. In that regard, our contact information can be found below or on our website at SilvertonConsulting.com.

This performance dispatch was sent out to our newsletter subscribers in October of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 Oct 25 Latest Microsoft ESRP v3 (Exchange 2010) results (PDF 1.6 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 25 October 2010

[2] All prior SCI ESRP Dispatches can be found at http://silvertonconsulting.com/cms1/dispatches/

SCI 2010 Jul 28 latest ESRP results analysis – over-5K mailboxes

Jul 282010

This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] V3.0 for Exchange 2010. There have been a large number of submissions this past quarter with at least ten new ones in the over 5K mailbox category discussed here. Future reports will cover the 1001 to 5K and the 1K and under mailbox categories. Previous ESRP V2 and ESRP V3.0 analysis reports are available on our website[2].

Latest ESRP V3.0 results

We start our analysis with Exchange database access latency results. Recall that this chart lists the top 10 database-read latencies reported by ESRP for the over-5K mailbox category.

(SCIESRP100728-001) (c) 2010 Silverton Consulting, All Rights Reserved

For database reads the HDS AMS 2100 and 2300 take the top two spots with HP’s EVA coming in a close third. However, I find the HP Smart Array results (#4) very interesting.

HP’s Smart Array consists of just a bunch of SAS interfaced disks, connected to Exchange Servers. Its log-write latency is almost immeasurable (~0.1msec). As log writes are primarily sequential writes and we would expect a JBOD to do well here but this seems too good. That this (sub-)system had excellent database read and log-write latencies implies that both disks and controllers were well tuned for random and sequential I/O – in my experience, a hard thing to do without cache. HP’s Smart Array used a mailbox database size = disk drive size, which may have resulted in good access times, but it’s unclear why. As a counter example, Dell’s submission (#7) also used direct connected SAS drives but had a database size ~2X the size of their disks.

The other odd result in Figure 1 is the variability in Exchange 2010 database write access times. One would think that caching subsystems would accommodate most database writes at high speed. But, we believe this write variability is an affect of Exchange 2010’s larger database blocksizes forcing more destage activity and simulated DAG I/O activity to replicate database data for each write operation.

DAG I/O results from all the over 5K mailbox results supporting Exchange 2010 mailbox (database) resiliency. As such, any Exchange database write must be replicated to alternate (usually 2 or 3) copies. It’s unclear how Jetstress measures simulated replication I/O vis-a-vis database write latencies but we assume that in real Exchange 2010 environments, any database write could not complete until all DAG copies were updated.

Next we turn to database transfer counts. As ESRP mailbox counts for the over 5K mailbox category span such a wide spectrum we have elected to normalize database transfer counts to accesses per 1,000 mailboxes (1Kmbx).

(SCIESRP100728-002) (c) 2010 Silverton Consulting, All Rights Reserved

The #1 result was the same HP Smart Array discussed above an HP Prolient server SAS connected to an D2600 disk array. In addition, there were two IBM XIV submissions (#2 and #6) at the same mailbox count (40K) with substantive differences between the two being the mailbox size (3- vs. 1-GB for slower), drive capacity (1- vs. 2-TB for slower), and storage used (40% vs. 88% for slower). As discussed in prior reports, the read database transfer counts are significantly higher than the write transfers, in some cases almost 2X the rate.

A couple of caveats for normalized results:

Normalized results may or may not scale much beyond the reported mailbox counts. For example, the #1 result supported 6K mailboxes and may not support much more than that.
Normalized results can be impacted by over provisioning. For example, the #2 result only used 40% of its storage for Exchange services allowing it to use more spindles than necessary for the workload.

(SCIESRP100728-003) (c) 2010 Silverton Consulting, All Rights Reserved

Speaking of over provisioning, another way to look at Exchange storage performance is to normalize it over the number of disk drives or spindles used for the configuration. Figure 3 above shows the total number of (read and write) database operations per second per drive done by each subsystem. Here one can see the same two IBM XIV submissions in the same order discussed for Figure 2.

Some caveats for database transfers per spindle results:

Drive speed can help one do well on this metric, i.e. 15Krpm drives can perform better than 7.2Krpm drives and four of the top six performers used 15Krpm drives (both XIV results used 7.2Krpm drives).
Drive over provisioning usually reduces one’s performance on this metric. However this was not evident with the XIV placements (the over provisioned XIV did better than the normally provisioned one).

(SCIESRP100728-004) (c) 2010 Silverton Consulting, All Rights Reserved

Next, we show the top 10 Exchange backup throughput rates in MB/sec/database. The #1 and #2 positions went to IBM with SVC and XIV. Tied for #2 spot was Dell PowerVault, the other SAS connected disk system.

With Exchange 2010 and mailbox resiliency through DAGs, database backup activity no longer seems as important. In fact, there was at least one submission that didn’t even report on this metric. However, there are many valid reasons for database backups and we continue to believe that there will be an ongoing need for mailbox backups. As such, reporting on backup speed need to be re-instated and preserved.

Conclusions

ESRP results for the over 5K mailbox category are always difficult to analyze due to the wide span of mailbox counts (from 6K to almost ~69K for current submissions). That said, with the limited submissions to date, it appears for smaller mailbox counts, a properly configured SAS direct connected storage system may perform well enough. Above that is subject to some debate but more results should help clarify this.

Nonetheless, it’s still early in ESRP v3 history. To date there have been only 12 submissions in this category and just over 20 overall (with reports available). Even so, we were surprised to see this many, since our last report only showed 4 results for all categories.

Finally, ESRP/Jetstress results seem designed to be difficult to compare but merit the effort. Thus, we strive to improve our analysis with each report. As always, feel free to contact us with any ideas on how to improve. In that regard, our contact information can be found below or on our website at SilvertonConsulting.com.

This performance dispatch was sent out to our newsletter subscribers in July of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 July 28 Latest ESRP V3.0 for Exchange 2010 results (PDF 744.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 27 Julyl 2010

[2] All prior SCI ESRP Dispatches can be found at http://silvertonconsulting.com/cms1/dispatches/

SCI 2010 Apr 29 First ESRP v3 (Exchange 2010) results analysis – all mailbox counts

Apr 292010

This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] V3.0 for Exchange 2010. There have been only four submissions for ESRP V3.0 and they are the HP 8400 EVA, EMC AX4-5i, IBM SVC 5.1 with DS3400s, and IBM XIV 2810 storage subsystems ranging from 500 to 40,000 mailboxes. Given the limited submissions we have chosen to report on all results in this dispatch. Previous reports for ESRP V2.1 and Exchange 2007 are available here[2].

Latest ESRP V3.0 results

As the mailbox counts span such a wide spectrum we have elected to normalize the database access counts to accesses per 1,000 mailboxes (1Kmbx). Realize that the EMC AX4 supported only 500 mailboxes and may not be able to sustain this workload for the 1000 mailbox we show below.

(SCIESRP100429-001) (c) 2010 Silverton Consulting, All Rights Reserved

In every case the read database transfer counts are significantly more than the write transfers, in some cases almost 2X the rate. Unsure why this is and this was not evident in ESRP v2.2. Both IBM’s XIV and HP’s EVA seem to do considerably better than the other two products in this metric.

(SCIESRP100429-002) (c) 2010 Silverton Consulting, All Rights Reserved

Figure 2 shows the total number of (read and write) database operations per second per spindle done by each subsystem. The IBM XIV once again dominates this category. It’s worth noting that

Both the XIV and EMC’s AX4 used 7200RPM 1TB SATA drives whereas the other two systems used 15Krpm 450GB drives
The AX4 only had 8 disks
The AX4 was attached to the Exchange servers via iSCSI while all the others used FC, and
The XIV used only about 40% of its storage for Exchange data whereas all the other systems used 61 to 86% of their storage for data.

One would think short stroking drives like XIV did would show worse performance on a per spindle basis. However, apparently XIV is able to spread the workload over its higher spindle count to actually attain a better per spindle performance. This is certainly worth more investigation from our perspective.

(SCIESRP100429-003) (c) 2010 Silverton Consulting, All Rights Reserved

Figure 3 depicts the average latency for database operations by subsystem. Here the HP EVA shows up very well. Unclear why XIV had such a poor showing here but, slower drives probably played some role. It’s also interesting to see that IBM’s SVC very good write latency. We would attribute this to advanced write caching and more cache available to absorb write data but in any case it does well with database random writes and very well with log I/O sequential writes.

(SCIESRP100429-004) (c) 2010 Silverton Consulting, All Rights Reserved

Next, we show the Exchange backup MB/sec per database for the four submissions. Again the three top systems used FC while the EMC AX4 used iSCSI but other than that there doesn’t seem to be much difference here.

(SCIESRP100429-005) (c) 2010 Silverton Consulting, All Rights Reserved

Finally, Figure 5 shows the time it takes to play back a 1MB log file against the database. This chart provides a view of performance for recovery scenarios. Here IBM’s SVC write performance helps it dominate this category.

Conclusions

Exchange 2010 has introduced Data Availability Groups (DAG) and as such, changed some of the redundancy/availability capabilities available in Exchange 2007. This has forced some subtle modifications to the Jetstress ESRP driver. For example, when hardware configurations can be split in two equal parts, Jetstress now simulates DAG redundant writes to half of the hardware configuration that would have corresponded to the other half’s database updates. These redundant writes constitute the only I/O needing to be done to this half of the storage for the other half’s user activity.

As such, vendors can now submit ESRP reports using half the hardware needed to support a given workload as long as its configuration could be cut into identical halves. This reduction in hardware requirements for ESRP reports should encourage more vendors to submit results. However, it made for confusing reading when we first saw it. (If you understand all this on first reading you get an A+.)

As more ESRP submissions appear we will once again break our analysis up into multiple categories but until then we will keep to normalized comparisons as much as possible. As stated before, please realize that any normalization may imply subsystem performance beyond its present capabilities.

ESRP/Jetstress results are inherently difficult to compare but are worth the effort in our view. We strive to improve our analysis with each report. As such, feel free to contact us with any ideas or improvements to this analysis, our contact information can be found below.

This performance dispatch was sent out to our newsletter subscribers in April of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 April 29 Latest ESRP V3.0 for Exchange 2010 results (PDF 847.0 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 29 April 2010

[2] All prior SCI ESRP Dispatches can be found at http://silvertonconsulting.com/cms1/dispatches/

SCI 2010 Jan 28 last ESRP v2.1 (Exchange 2007) results analysis – 1K-to-5K mailboxes

Jan 282010

This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] performance results for the over 1000 to 5000 mailboxes results category. Prior reports discussed the over 5000 mailboxes and 1000 and under 1000 mailboxes result categories[2].

ESRP was never intended to compare subsystem performance but rather as a proof of concept for Microsoft and storage vendors to depict a configuration supporting a given workload. Hence, any comparisons necessarily come with some caveats and may not be real. Nonetheless, SCI feels comparisons can well serve both the vendor and end-user storage community and thus, worth noting.

Latest ESRP V2.1 results

Our first chart is new to this analysis and shows aggregate database transfers per second per physical spindle. An astute reader requested we include performance per spindle. This metric doesn’t correlate well to any other ESRP performance parameter. Note, some of these subsystems use only 4 spindles (e.g., both AX4-5i) while #1 Netapp FAS2040 used 12 and #5 Dell MD1000 used 20.

(SCIESRP100128-001) (c) 2010 Silverton Consulting, All Rights Reserved

The nice thing about transfers per spindle is the wide range of subsystems that perform well on it, e.g., 5 of these subsystems are iSCSI attached, 3 are SAS attached and the remainder are FC. In addition, the number of mailboxes supported spans almost the whole range from 1400 to 5000. We do not show the speed of the drives (15 or 10Krpm) or their interfaces. Nonetheless, if you want to attain the most from a set of spindles one would do well by going with either the FAS2040 or AX4-5i.

(SCIESRP100128-002) (c) 2010 Silverton Consulting, All Rights Reserved

Figure 2 shows the total number of database operations per second done by each subsystem. This is mostly correlated to the number of mailboxes but the HP MSA60 seems to do quite well for only having 2500 mailboxes. Even so the Dell and HP MSA70 dominate these top 10 results. HP MSA2000sa G2 and Fujitsu ETERNUS DX80 were the only new additions to this chart.

(SCIESRP100128-003) (c) 2010 Silverton Consulting, All Rights Reserved

Figure 3 is similar to Figure 2 except we have normalized subsystem database transfers/second performance by number of mailboxes (actual 1000 mailboxes). If it weren’t for the Dell MD1120, the top 5 would all be HP MSA storage. New entries to this chart are the HP MSA2000sa G2, Fujitsu ETERNUS DX80 and the EMC Celerra NS-480. As discussed in prior reports, normalized results may or may not scale up beyond their actual mailbox counts reported. For example, the subsystem results for 2KMbx may not hold up when pushed to support 5KMbx.

(SCIESRP100128-004) (c) 2010 Silverton Consulting, All Rights Reserved

Next, we show the Top 10 database backup throughput results. Both Dell’s MD1120 and MD1000 did well in this category. Once again the two new subsystems on this chart were the Fujitsu DX80 and the HP MSA2000sa G2. We like this chart because it’s a good surrogate for raw subsystem read throughput (although it’s database reads). For subsystems in this mid-range category to break 1GB/second seems very impressive.

(SCIESRP100128-005) (c) 2010 Silverton Consulting, All Rights Reserved

Figure 5 Top 10 normalized log write/second

Finally, we show the aggregate log writes per second chart. This view of subsystem performance shows the write IOPs that each subsystem can perform. The new subsystems on this chart include HP MSA2000sa G2, NetApp FAS2040, Fujitsu ETERNUS DX80, and the EMC Celerra NS-480. We have a normalized view of this activity but it looks almost the same and does not show the top end nearly as well.

Conclusions

From our perspective, ESRP results in this mid range category seem to be getting more competitive. There were 6 new ESRP results in this category over the last 9 months, and at least 2 over the last quarter. In almost every chart one can see at least 2 and in most cases 3 or more new results showing up in the top 10.

We have always liked ESRP results because they show a real worldview of subsystem performance. Additionally, there seems to be much more willingness on the part of vendors to submit results to ESRP than some of the other, standard benchmarks. Also, iSCSI, FC and SAS attached storage results are available. Given all that, it’s a great way to compare subsystem performance.

ESRP/Jetstress results are inherently difficult to compare but are worth the effort in our view. Our next ESRP/Jetstress report will return to the 1K mailbox and under tier. We added a new database transfers per spindle chart based on feedback we received and continue to welcome any feedback on how to do better. As such, feel free to contact us with any ideas, our contact information can be found below.

This performance dispatch was sent out to our newsletter subscribers in January of 2010. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 January 28 Analysis on recent ESRP results (PDF 773.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 28 January 2010

[2] All prior SCI ESRP Dispatches can be found at http://silvertonconsulting.com/cms1/dispatches/

SCI 2009 Oct 30 latest ESRP results analysis – over-5K mailboxes

Oct 302009

This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] performance results for the over 5000 mailboxes results category. Prior reports discussed the over 1000 to 5000 mailboxes and the under 1000 mailboxes result categories[2].

Latest ESRP V2.0 results

The first chart refers to aggregate average database backup throughput across all storage groups. This value correlates moderately to aggregate database transfers per second. (See figure 1).

(SCIESRP091030-001) (c) 2009 Silverton Consulting, All Rights Reserved

As can be seen from Figure 1, backup activity is dominated by HDS and HP (an HDS OEM). Partly HDS’s commanding result is due to the overall number of drives and mailboxes being serviced. However, 3PAR’s 96Kmbx and Dell’s Equal Logic iSCSI result for 90Kmbx results only came in at number five and seven respectively. Also, the number nine result for Pillar Data’s Axiom 600 supports only 34K mailboxes and stands almost as well as the other results supporting many more mailboxes. A couple of caveats worth noting here database backup performance can be impacted by

Number of disk drives in a configuration
How message store databases are split across those spindles
Subsystem RAID level

(SCIESRP091030-002) 2009 (c) Silverton Consulting, All Rights Reserved

This chart shows ESRP reported database latencies for read, write and log write operations. SCI and others feel the read latency metric best shows what an end-user experience would be from a subsystem configuration. The list is sorted by read latency. New results from Pillar Data with a read latency of 3ms and Sun’s J4400 at 8ms showed up very well here. A couple of considerations to note:

While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated. For example if one examines the EMC SRDF/S in the number two position, its write latency is pretty high. However if one considers that SRDF/S was active this means the data has to be written to the secondary subsystem in parallel to being written to the primary subsystem and as such its write latency does not look that bad.
There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage. As ESRP reports are intended to show a viable performing solution to handle a simulated user workload we assume that these products are all optimizing cost and performance, so believe this is not an issue here.
For an ESRP benchmark to be accepted, read latency must be under 20 msecs. Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware. As such, those vendors may not show up well on this chart.

(SCIESRP091030-003) 2009 (c) Silverton Consulting, All Rights Reserved

The new result for Pillar Data broke into the top 10 log playback time. The top 4 systems LSI 399x, Pillar Data Axiom 600, Lefthand SANIQ, and IBM DS4800 would all be considered mid range storage subsystems although HP’s Lefthand was supporting a heavy workload at 50K mailboxes and was configured accordingly. The range for the top ten subsystems is fairly large, over 2X from lowest to highest. It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations. Similar to the backup discussions above, playback performance can be impacted by:

Number of disk drives in a solution,
How message store databases are split across those spindles,
Subsystem RAID level,
Replication type

(SCIESRP091030-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to normalized database transfer results and the Top 3 normalized ESRP/Jetstress results belong to HDS AMS2100, HDS AMS2300 and IBM XIV, all midrange systems. A few considerations are warranted on normalized results:

Normalized results do not always scale well. Although four of these results were for 20,000 mailboxes or over, (HDS USP-V, Sun StorageTex 6540, 3PAR Inserv T800, and EMC Symmetrix at 60KMbx) the top result from HDS supported only 5400 mailboxes and may not scale much beyond that quantity of mailboxes.
One surprise here is the close running of everyone behind the top five results and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox. But the workload simulated for these results varied considerably (0.3 to 1.0 iops/sec/mbx).

Conclusions

From our perspective, ESRP results in this over 5K mailbox tier are getting more competitive. There were a number of new ESRP results in this category over the last 9 months, and at least 3 over the last quarter. Seeing HDS, EMC and IBM highend systems, all running the same performance tests is a good indicator of their willingness to show their products in the best light as well as high customer interest in Exchange solutions. Most likely, these vendors do not see their individual results as entirely comparable and arguably they may have a point, but we differ with them on this assessment. Moreover, seeing DAS (HP MSA), SATA (IBM XIV), SAS (Sun, HDS AMS2100, & HP MSA), iSCSI (EMC, HP, NetApp, & Dell) and FC storage subsystem results compete in this top-tier at least on various metrics indicates to us that all storage interfaces can be competitive in mission critical applications.

ESRP/Jetstress results are inherently difficult to compare. Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases. Our next ESRP/Jetstress report will return to the 1K to 5K mailbox tier. We continue to welcome any feedback on how to do better.

This performance dispatch was sent out to our newsletter subscribers in October of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 October 30 Latest ESRP results (PDF 681.0 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 27 January 2009

[2] All of prior SCI ESRP Dispatches can be found at ofhttp://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2009 Jul 28 latest ESRP results analysis – 1K-to-5K mailboxes category

Jul 282009

This is SCI’s third report in our new series on Microsoft Exchange Solution Review Program (ESRP)[1] performance results and reports on the 1000 and under mailbox results category. Prior reports discussed the 1001 to 5000 mailboxes and the over 5000 mailboxes result categories[2]. For this class of results we no longer provide normalized results.

Latest ESRP V2.0 results

We now use a ranking for this analysis depicting the average database backup throughput across all storage groups, which roughly corresponds to an IT or datacenter view of Exchange performance. This value correlates moderately to aggregate database transfers per second. (See figure 1).

(SCIESRP090728-001) (c) 2009 Silverton Consulting, All Rights Reserved

We double checked our data and have determined that HP StorageWorks MSA 2000fc G2 is tied for first place with Dell PowerEdge 2900 III and the Dell PowerEdge 3000 comes in at third place. All these results supported 1000 mailboxes except the EMC NS20, which only had 700 mailboxes. Also the MSA result used FC while the others in the top four used SAS and fifth place used iSCSI. A couple of caveats worth noting here database backup performance can be impacted by

Number of disk drives in a configuration
How message store databases are split across those spindles
Subsystem RAID level

(SCIESRP090728-002) (c) 2009 Silverton Consulting, All Rights Reserved

Another new chart we added shows the ESRP reported database latencies for read, write and log write operations. SCI and others feel the read latency metric may best show end-user experience attainable from a storage subsystem configuration. The list is sorted by read latency. A couple of considerations to note:

While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated.
There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage. As of V2.1, all ESRP results now publish the number of simulated IOPS/mailbox used during the Jetstress run and as more V2.1 results become available we may be better able to detect gaming of latency factors, if present. At the moment, ESRP V2.1 reports are scarce and only one of the top 10 have reported on this value.
For an ESRP benchmark to be accepted, read latency must be under 20 msecs. Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware. Such vendors may not show up well on a top 10-latency ranking.

(SCIESRP090728-003) (c) 2009 Silverton Consulting, All Rights Reserved

We have also started to examine the log playback ranking. This is a rather complex workload that encompasses log reading, database reading and updating (or writing). The timings are reported as the average time in seconds it takes to playback or process a 1 MB log. The top 3 systems are Huawei-Symantec’s OceanStor S5500, Dell’s PowerEdge 2900 III, and EMC CX-3-20. The range for the top ten subsystems is not large, ~0.39 to 0.55 seconds from lowest to highest. It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations. Similar to the backup discussions above, some caveats would include:

The number of disk drives can significantly impact playback performance.
How message store databases are split across those spindles also can impact this
Subsystem RAID level may also impact playback performance
Replication type may also impact log playback performance

(SCIESRP090728-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to overall database transfer results and the Top 3 un-normalized ESRP/Jetstress results belong to HP MSA 2000fc G2, IBM DS3300, and Dell PowerEdge 2900 III. A few considerations are warranted on un-normalized results:

While all these results were for 1000 mailboxes, it’s unlikely that these systems will scale much above this.
One surprise here is the close running of everyone behind the top three results and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox. We saw a similar narrow result range for our last report on the over 5Kmbx submissions as well.

Conclusions

From our perspective, ESRP results in this 0 to 1K-mailbox tier are becoming less interesting. There were only a few new ESRP results in this category over the last 9 months, and only one over the last quarter. Most new ESRP submissions have been in the over 5Kmbx category and will be discussed in next quarters ESRP review.

However, it is interesting to see FC, SAS, and iSCSI subsystem results in this category all fighting for prominence. One would think that FC would have a natural advantage across the board here but iSCSI and SAS also place very well at this level. Such competitiveness probably says real world workloads (like Exchange) are less dependent on interface protocols and more dependent on subsystem architecture or other factors.

ESRP/Jetstress results are inherently difficult to compare. Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases. Our next ESRP/Jetstress report will return to over 5K-mailbox tier. We continue to welcome any feedback on how to do better.

This performance dispatch was sent out to our newsletter subscribers in July of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 July 28 Update to ESRP performance results (PDF 569.2 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 28 July 2009

[2] All prior SCI ESRP Dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2009 Apr 29 latest ESRP results analysis – 1K-to-5K mailboxes category

Apr 292009

We now turn to analyzing Exchange Solution Review Program (ESRP) [1] results in the 1001 to 5000 mailbox category. Our January report[2] discussed the over 5000 mailboxes category, and our next ESRP report will discuss the under 1001 mailboxes result category. To better compare ESRP/Jetstress results SCI reports on both normalized and un-normalized results. For normalized results in this mid-tier category we use operations per 1000 mailbox (1Kmbx). Un-normalized results are in the appendix.

Latest ESRP V2.0 results

(SCIESRP090429-001) (c) 2009 Silverton Consulting, All Rights Reserved

As can be seen from the above chart, the Dell’s MD1000 report wins this category. Partly Dell’s result is due to the overall number of mailboxes being serviced. However, Fuji Siemens result, also for 5K mailboxes only came in at number five. A couple of caveats worth noting here database backup performance can be impacted by:

Number of disk drives in a configuration
How message store databases are split across those spindles
Subsystem RAID level

(SCIESRP090429-002) (c) 2009 Silverton Consulting, All Rights Reserved

This chart shows the ESRP reported database latencies for read, write and log write operations. SCI and others feel the read latency metric best shows what an Exchange end-user experience would be from a subsystem configuration. The list is sorted by read latency. Not sure why Dot Hill’s write latency is so poor here but it could be an artifact of their caching activity. A couple of other considerations to note:

While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated.
There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage. As ESRP reports are intended to show a viable performing solution to handle a simulated user workload we assume that these products are all optimizing cost and performance, so believe this is not an issue here.
For an ESRP benchmark to be accepted, read latency must be under 20 msecs. Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware. As such, those vendors may not show up well on this top 10 chart.

(SCIESRP090429-003) (c) 2009 Silverton Consulting, All Rights Reserved

The log playback is a rather complex workload that encompasses log reading and database reading and updating (or writing). The timings are reported as the average time in seconds it takes to playback or process a 1 MB log. The top 3 systems Dot Hill SystemR, and the two Dell Power vault MD1000 results would all be considered small to medium range storage subsystems. The range for the top ten subsystems is fairly large over 3X from lowest to highest. It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations. Similar to the backup discussions above, some caveats would include:

The number of disk drives can significantly impact playback performance.
How message store databases are split across those spindles also can impact this
Subsystem RAID level may also impact playback performance
Replication type may also impact log playback performance

(SCIESRP090429-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to overall database transfer results and the Top 3 normalized ESRP/Jetstress results belong to HP MSA70, HP MSA60, and HP MSA2000sa. One consideration is warranted on normalized results, namely that normalized results do not always scale well. Although only one of these results were for 5,000 mailboxes (Fuji Siemens FibreCat), the top result from HP supported only 4000 mailboxes and may not scale much beyond that quantity of mailboxes.

Conclusions

From our perspective, ESRP results in this 1K to 5K mailbox tier are heating up since we reported on this category last July. There were a number of new ESRP results in this category over the last 9 months, and at least 2 over the last quarter. For some reason HP seems to have a lock on this category performance. Also the category supports SAS, FC and iSCSI solutions very evenly.

ESRP/Jetstress results are inherently difficult to compare. Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases. Our next ESRP/Jetstress report will return to the 1K and under mailbox tier. We continue to welcome any feedback on how to do this analysis better.

A PDF version of this can be found at

SCI 2009 April 29 Update to ESRP performance results (PDF 684.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

Appendix

(SCIESRP090429-005) (c) 2009 Silverton Consulting, All Rights Reserved

Aggregate or un-normalized database transfer results are highly correlated to number of mailboxes in service and as such, are relatively less useful metrics in our opinion. How HP MSA70, a SAS result, dominates this category is difficult to discern.

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 29 January 2009

[2] Available at http://www.silvertonconsulting.com

SCI 2009 Jan 27 latest ESRP results analysis – over 5K mailboxes

Jan 272009

This dispatch restarts SCI’s series on Microsoft Exchange Solution Review Program (ESRP)[1] performance results and reports on the over 5000 mailboxes results category. Prior reports discussed the over 1000 to 5000 mailboxes and the under 1000 mailboxes result categories[2]. As such, to better compare ESRP/Jetstress results SCI reports on both normalized and un-normalized results. For normalized results in this highest-tier category we use operations per 5000 mailbox (5Kmbx). Un-normalized results are in the appendix.

Latest ESRP V2.0 results

We have added a new ranking for this analysis, which depicts the average database backup throughput across all storage groups. This value correlates moderately to aggregate database transfers per second. (See figure 1).

(SCIESRP090127-001) (c) 2009 Silverton Consulting, All Rights Reserved

As can be seen from the above chart, the new report on HP XP24000 dominates this category. Partly HP’s commanding result is due to the overall number of mailboxes being serviced. However, Dell’s Equal Logic iSCSI result for 90K mailboxes only came in at number four. Also the number ten result, for HP EVA4400 supports only 8K mailboxes and stands almost as well as the other results supporting many more mailboxes. A couple of caveats worth noting here database backup performance can be impacted by

Number of disk drives in a configuration
How message store databases are split across those spindles
Subsystem RAID level

(SCIESRP090127-002) (c) 2009 Silverton Consulting, All Rights Reserved

We add another new chart for this analysis showing the ESRP reported database latencies for read, write and log write operations. SCI and others feel the read latency metric best shows what an end-user experience would be from a subsystem configuration. The list is sorted by read latency. A couple of considerations to note:

While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated. For example if one examines the EMC SRDF/S in the number two position, its write latency is pretty high. However if one considers that SRDF/S was active this means the data has to be written to the secondary subsystem in parallel to being written to the primary subsystem and as such its write latency does not look that bad.
There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage. As ESRP reports are intended to show a viable performing solution to handle a simulated user workload we assume that these products are all optimizing cost and performance, so believe this is not an issue here.
For an ESRP benchmark to be accepted, read latency must be under 20 msecs. Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware. As such, those vendors may not show up well on this top 10 chart.

(SCIESRP090127-003) (c) 2009 Silverton Consulting, All Rights Reserved

Another new chart for this report is the log playback ranking. This is a rather complex workload which encompasses log reading and database reading and updating (or writing). The timings are reported as the average time in seconds it takes to playback or process a 1 MB log. The top 3 systems LSI 399x, Lefthand SANIQ, and IBM DS4800 would all be considered mid range storage subsystems although Lefthand was supporting a heavy workload at 50K mailboxes and was configured accordingly. The range for the top ten subsystems is fairly large over 2X from lowest to highest. It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations. Similar to the backup discussions above, some caveats would include:

Playback performance can significantly be impacted by the number of disk drives.
How message store databases are split across those spindles also can impact this
Subsystem RAID level may also impact playback performance
Replication type may also impact log playback performance

(SCIESRP090127-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to overall database transfer results and the Top 3 normalized ESRP/Jetstress results belong to HP MSA70, Sun StorageTex 6540 and EMC Symmetrix DMX-4 4500. A few considerations are warranted on normalized results:

Normalized results do not often scale well. Although four of these results were for 20,000 mailboxes or over, (Sun 6540 at 20KMbx, EMC Symmetrix at 60KMbx, HDS AMS1000 at 25KMbx, and IBM DS8100 at 24KMbx) the top result from HP supported only 6000 mailboxes and may not scale much beyond that quantity of mailboxes.
One surprise here is the close running of everyone behind the top result and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox. However, the over 5K mailbox tier is the only category that shows results this close to one another. (see prior ESRP StorInt Dispatchen for more information).

Conclusions

From our perspective, ESRP results in this over 5K mailbox tier are getting more competitive. There were a number of new ESRP results in this category over the last 9 months, and at least 4 over the last quarter. Seeing HP (an OEM version of HDS’s top product), EMC Symmetrix and IBM DS8300 running the same performance tests is a good indicator of their willingness to show their products in the best light as well as high customer interest in Exchange solutions. Probably these vendors don’t see their individual results as entirely comparable and arguably they may have a point, but we would differ with them on this assessment. Moreover, seeing SAS and iSCSI results compete in this top-tier at least on various metrics indicates these interfaces can provide some competition to FC storage in mission critical applications.

This performance dispatch was sent out to our newsletter subscribers in January of 2009. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 January 27 Update to ESRP/Jetstress benchmark performance results (PDF 1.4 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

Appendix

(SCIESRP090127-005) (c) 2009 Silverton Consulting, All Rights Reserved

Aggregate or un-normalized database transfer results are highly correlated to number of mailboxes in service and as such, are relatively less useful metrics in our opinion. Nonetheless, it’s probably no surprise that the top results belong to HP, EMC and IBM. Aside from the iSCSI results, this ranking is similar to one ranked purely on number of mailboxes.

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 27 January 2009

[2] All of prior SCI ESRP Dispatches can be found at ofhttp://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

SCI 2008 Oct 21 latest ESRP v2 (Exchange 2007) results analysis – under 1K mailboxes

Oct 212008

This is SCI’s third report on Microsoft Exchange Solution Review Program (ESRP)[1] performance results. This report focuses on 0 to 1000 mailboxes category and our prior reports discussed over 5000 and 1000 to 5000 mailboxes result categories. See our prior ESRP/Jetstress dispatches for a better description of ESRP benchmarks and our reported metrics.

Latest ESRP V2.0 results

(SCIESRP081021-001) (c) 2008 Silverton Consulting, All Rights Reserved

In order to better compare Jetstress results we report on both normalized and un-normalized results. For normalized results in this bottom-tier category we use operations per mailbox (mbx). As an example, Compellent reports results for 600 mailboxes at ~331, but normalized to per mbx their results are 0.5 database transfers/sec/mbx (see Figure 1).

We have added a new chart for this dispatch ranking the log file playback.. This metric depicts the average time a 1MB log file takes to be played against the database and is not always specified for ESRP results but all of these ESRP results in this bottom-tier include this metric.

The Top 3 normalized Jetstress results for the 0 to 1000 mailbox category were IBM DS3300 in Clustered Continuous Replication (CCR), Dell PowerEdge 2900 CCR and Sun Fire X4150 (see Figure 1). A few considerations on normalized results

Most of these results were for iSCSI subsystems, Dell and Sun results were SAS attached storage and the Compellent result was for FC attached storage.
Normalized results don’t always scale well. Although 5 of these results were for 1000 mailboxes (IBM, Dell, SUN, EMC NS20, Compellent, and one of the EMC CX3-20) the rest had an average of ~730 mailboxes. For example, although the AMI StorTrends 3200i may do well at 600 mbx it may or may not scale much beyond that.
Another surprise is that the normalized results span such a large range. The top 10 normalized results range from over 1.8 down to around .5 database operations per second per mbx over a factor of 3X, for relatively similar numbers of mailboxes. The IBM DS3300 is the clear winner here.

(SCIESRP081021-002) (c) 2008 Silverton Consulting, All Rights Reserved

The top 3 subsystems in this category are IBM DS3300 CCR, Dell and the Sun subsystems. In contrast to the normalized results above all of the unnormalized top 10 reported results are for 1000 mailboxes. There was one less iSCSI result here in comparison to the normalized results as the other Dell PowerEdge 3000 result was also SAS.

(SCIESRP081021-003) (c) 2008 Silverton Consulting, All Rights Reserved

There should be some correlation between the other results and the database backup results but apparently there is none. The top 3 results were all SAS subsystems and the next two were iSCSI and FC respectively. This implies that SAS has an advantage when it comes to raw data reading over iSCSI. Not sure why the FC product didn’t do better but that’s subject for another discussion. Again the storage group results bear little relation to the aggregate database backup results but the number of storage groups is one of those vendor-optimized variables that confound ESRP/Jetstress result comparisons.

(SCIESRP081021-004) (c) 2008 Silverton Consulting, All Rights Reserved

Our first report on Log playout results does not appear that interesting. However, the Log results don’t correlate that well with any of the previous results. As the ESRP defines this metric, this sort of activity would typically occur during a crash recovery. Given these results all the subsystems are pretty comparable. But if crash recovery time is important perhaps the Dell PowerEdge 2900 or EMC CX3-20 may be worth a look.

Conclusions

From our view on ESRP results in this bottom-tier competition is abating. The only new benchmark during the past quarter for this tier was the AMI result at 600 mailboxes. In fact, if one closely examined this and our prior ESRP result dispatches they would notice that some results showed up twice. All results for 1000 mailboxes were reported on in both this tier and the next higher tier (1K to 5K mailboxes). We did this because there are so few results <1000 mailboxes.

iSCSI storage represents the majority of reported results in this category although the few SAS subsystems do almost as well. SAS does seem to have an advantage in backup speeds but other than that iSCSI handles the other workloads better. FC is just not well represented in this tier probably due to cost.

This is our third ESRP report and we have nov analyzed top 10 results for every category. We continue to welcome any feedback on how to do better. Jetstress results are inherently un-comparable. Nonetheless we believe Exchange results deserve some comparison and result rankings so that the public can be properly informed and as such make better storage purchases. Our next ESRP/Jetstress report will return to the >5K mailbox tier.

This performance dispatch was sent out to our newsletter subscribers in October of 2008. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports. Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2008 October 21 Update to ESRP/Jetstress benchmark performance results (PDF 1003.7 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 21 October 2008

Older Entries

Data Storage, Stategic Planning & Systems Development products and services | Silverton Consulting

SCI 2011 Apr 28 Latest Microsoft ESRP v3.0 (Exchange 2010) results

Latest ESRP V3.0 results

SCI 2010 Oct 25 latest Microsoft ESRP v3 (Exchange 2010) results – 1K to 5K mboxes

Latest ESRP V3.0 results

Conclusions

SCI 2010 Jul 28 latest ESRP results analysis – over-5K mailboxes

Latest ESRP V3.0 results

Conclusions

SCI 2010 Apr 29 First ESRP v3 (Exchange 2010) results analysis – all mailbox counts

Latest ESRP V3.0 results

Conclusions

SCI 2010 Jan 28 last ESRP v2.1 (Exchange 2007) results analysis – 1K-to-5K mailboxes

Latest ESRP V2.1 results

Conclusions

SCI 2009 Oct 30 latest ESRP results analysis – over-5K mailboxes

Latest ESRP V2.0 results

Conclusions

SCI 2009 Jul 28 latest ESRP results analysis – 1K-to-5K mailboxes category

Latest ESRP V2.0 results

Conclusions

SCI 2009 Apr 29 latest ESRP results analysis – 1K-to-5K mailboxes category

Latest ESRP V2.0 results

Conclusions

Appendix

SCI 2009 Jan 27 latest ESRP results analysis – over 5K mailboxes

Latest ESRP V2.0 results

Conclusions

SCI 2008 Oct 21 latest ESRP v2 (Exchange 2007) results analysis – under 1K mailboxes

Latest ESRP V2.0 results

NEWS and EVENTS

TOP DOWNLOADS

TWITTER FEED...

POPULAR POSTS

NEWSLETTER

TRANSLATE THIS PAGE

Dispatch categories

Dispatch archives

Latest ESRP V3.0 results

Latest ESRP V3.0 results

Conclusions

Latest ESRP V3.0 results

Conclusions

Latest ESRP V3.0 results

Conclusions

Latest ESRP V2.1 results

Conclusions

Latest ESRP V2.0 results

Conclusions

Latest ESRP V2.0 results

Conclusions

Latest ESRP V2.0 results

Conclusions

Appendix

Latest ESRP V2.0 results

Conclusions

Latest ESRP V2.0 results

NEWS and EVENTS

TOP DOWNLOADS

TWITTER FEED...

POPULAR POSTS

NEWSLETTER

TRANSLATE THIS PAGE

Dispatch Tag Cloud

Dispatch categories

Dispatch archives