Tag: Database backup performance

SCI’s Latest ESRP (v3) Performance Analysis for Over 5K mailboxes – chart of the month

Posted on by Ray in Block Storage, Disk storage, ESRP, ESRPv3/Exchange 2010, Storage, Storage Backup, Storage performance, System effectiveness
Bar chart showing ESRP Top 10 total database backup throughput results
(SCIESRP120125-001) (c) 2012 Silverton Consulting, All Rights Reserved

This chart comes from our analysis of Microsoft Exchange Reviewed Solutions Program (ESRP) v3 (Exchange 2010) performance results for the over 5000 mailbox category, a report sent out to SCI Storage Intelligence Newsletter subscribers last month.

The total database backup throughput reported here is calculated based on the MB read per second per database times the number of databases in a storage configuration. ESRP currently reports two metrics for database backups the first used in our calculation above is the backup throughput on a database basis and the second is backup throughput on a server basis.  I find neither of these that useful from a storage system perspective so we have calculated a new metric for our ESRP analysis which represents the total Exchange database backup per storage system.

I find three submissions (#1, #3 & #8 above) for the same storage system (HDS USP-V) somewhat unusual in any top 10 chart and as such, provides a unique opportunity to understand how to optimize storage for Exchange 2010.

For example, the first thing I noticed when looking at the details behind the above chart is that disk speed doesn’t speed up database throughput.  The #1, #3 and #8 submissions above (HDS USP-V using Dynamic or thin provisioning) had 7200rpm, 15Krpm and 7200rpm disks respectively with 512 disk each.

So what were the significant differences between the USP-V submissions (#1, #3 and #8) aside from mailbox counts and disk speed:

  • Mailboxes per server differed from 7000 to 6000 to 4700 respectively, with the top 10 median = 7500
  • Mailboxes per database differed from 583 to 1500 to 392, with the top 10 median = 604
  • Number of databases per host (Exchange server) differed from 12 to 4 to 12, with the top 10 median = 12
  • Disk size differed from 2TB to 600GB to 2TB, with the top 10 median = 2TB
  • Mailbox size differed from 1GB to 1GB to 3GB, with the top 10 median = 1.0 GB
  • % storage capacity used by Exchange databases differed from 27.4% to 80.0% to 55.1%, with the top 10 median = 60.9%

If I had to guess, the reason the HDS USP-V system with faster disks didn’t backup as well as the #1 system is that its mailbox databases spanned multiple physical disks.  For instance, in the case of the (#3) 15Krpm/600GB FC disk system it took at least 3 disks to hold a 1.5TB mailbox database.  For the #1 performing 7200rpm/2TB SATA disk system, a single disk could hold almost 4-583GB databases on a single disk.  The slowest performer (#8) also with 7200rpm/2TB SATA disks could hold at most 1-1.2TB mailbox database per disk.

One other thing that might be a factor between the #1 and #3 submissions is that the data being backed up per host was significantly different.  Specifically for a host in the #1 HDS USP-V solution they only backed up  ~4TB but for a host in the #3 submission they had to backup ~9TB.   However, this doesn’t help explain the #8 solution, which only had to backup 5.5TB per host.

How thin provisioning and average space utilization might have messed all this up is another question entirely.  RAID 10 was used for all the USP-V configurations, with a 2d+2d disk configuration per RAID group.  The LDEV configuration in the RAID groups was pretty different, i.e., for #1 & #8 there were two LDEVs one 2.99TB and the other .58TB whereas for the #3 there was one LDEV of 1.05TB.  These LDEVs were then added to Dynamic Provisioning pools for database and log files.  (There might be something in how the LDEVs were mapped to V-VOL groups but I can’t seem to figure it out.)

Probably something else I might be missing here but I believe a detailed study of these three HDS USP-V systems ESRP performance would illustrate some principles on how to accelerate Exchange storage performance.

I suppose the obvious points to come away with here are to keep your Exchange databases  smaller than your physical disk sizes and don’t overburden your Exchange servers.

~~~~

The full ESRP performance report went out to our newsletter subscriber’s this past January.  A copy of the full report will be up on the dispatches page of our website sometime next month (if all goes well). However, you can get the full ESRP performance analysis now and subscribe to future newsletters by just sending us an email or using the signup form above right.

For a more extensive discussion of block or SAN storage performance covering SPC-1&-2 (top 30) and ESRP (top 20) results please consider purchasing SCI’s SAN Storage Buying Guide available on our website.

As always, we welcome any suggestions on how to improve our analysis of ESRP results or any of our other storage performance analyses.

Comments?

 

Latest Microsoft ESRP v3 (Exchange 2010) 1K to 5K mailbox performance results – chart of the month

Posted on by Ray in Block Storage, Disk storage, ESRP, ESRPv3/Exchange 2010, Ethernet, FC, Storage Backup, Storage performance, System effectiveness
SCIESRP110726-004 (c) 2011 Silverton Consulting, All Rights Reserved
SCIESRP110726-004 (c) 2011 Silverton Consulting, All Rights Reserved

Microsoft specifies two different metrics on sequential read rates for database backup activity in their Exchange Solution Reviewed Program (ESRP) reports

  • MB read/sec per database
  • MB read/sec total per server

Our problem with these metrics is that they don’t say much about the storage systems performance.  Some ESRP submissions could have a single database while others can have 100s of databases.  And the same thing applies to servers, although 20 servers seems to be about the max we have seen.  So as one can see the MB/s/DB or MB/s/server can vary all over the place depending on the Exchange configuration that one uses, even for the same exact storage system.

In the above chart, we  have attempted to move beyond some of these problems and use the information supplied in the ESRP reports to aggregate DB backups across all databases.  As such, we have derived a new metric called “total database backup”.  (Pretty simple actually just multiply the MB/s/DB times the number of databases in the Exchange configuration).

A couple of problems with our approach.

  • Current ESRP reports typically utilize a shadow storage system and shadow Exchange servers which host 50% of the databases and email activity. So what I am showing for those ESRP reports is what two storage systems can accomplish not one.
  • Another potential way to get the same result would be to use the number of servers times the MB/sec/server metric. (But try as I might these two approaches didn’t work to get the same answer so I am using the computation above – must be the way I am recording the number of [shadow] servers).
  • Although ESRP reports the average MB/sec/database to backup a single database it’s not clear that these measurements were taken while backing up all active databases at the same time, especially for those submissions with 100s of databases.

Probably the last is the most problematic critique to our new measure but may not be that harmful for smaller configurations. Nonetheless, we produced the above chart and published it in our last months review of ESRP results for the 1001 to 5000 mailbox category.

One item we discussed in our report was that numbers of disk drives didn’t seem to correlate well with high positions on this chart.  The number ten position (Fujitsu ETERNUS JX40) used over 140 disks, the number two position (Dell PowerEdge R510) had only 12 disk drives, and the number one solution (HP E5700) consisted of 56 drives, close to the average for this category.

One striking finding using this measure is that performance varies considerably from the top providing over 1600 MB/sec of database backup to the lowest of the group providing only ~800 MB/sec of backup performance. What with Exchange 2010 and lagged DAGs, some people feel that backup activity is no longer needed but we would disagree. We continue to believe that taking backups of Exchange data still makes a whole lot of sense and shouldn’t go away, ever.

It’s our hope that this or some similar follow-on metric will remove some of the Exchange configuration parameters from confounding ESRP reported storage system performance results.  We realize that this quixotic quest may never be entirely successful nevertheless we perform this duty in the hope that it will benefit today and future storage performance analysts everywhere.

Comments?

—–

The full ESRP report went out to our newsletter subscribers last month.  A copy of the full report will be up on the dispatches page of our website later next month. However, you can get this information now and subscribe to future newsletters to receive these reports even earlier by just emailing us at SubscribeNews@SilvertonConsulting.com?Subject=Subscribe_to_NewsletterR or using the signup form above and to the right.

As always, we welcome any suggestions on how to improve our analysis of ESRP or any of our other storage system performance discussions.