SCI 2010 Jul 28 latest ESRP results analysis – over-5K mailboxes

Jul 282010
 

This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] V3.0 for Exchange 2010. There have been a large number of submissions this past quarter with at least ten new ones in the over 5K mailbox category discussed here.  Future reports will cover the 1001 to 5K and the 1K and under mailbox categories.  Previous ESRP V2 and ESRP V3.0 analysis reports are available on our website[2].

Latest ESRP V3.0 results

We start our analysis with Exchange database access latency results.  Recall that this chart lists the top 10 database-read latencies reported by ESRP for the over-5K mailbox category.

(SCIESRP100728-001) (c) 2010 Silverton Consulting, All Rights Reserved

(SCIESRP100728-001) (c) 2010 Silverton Consulting, All Rights Reserved

For database reads the HDS AMS 2100 and 2300 take the top two spots with HP’s EVA coming in a close third.  However, I find the HP Smart Array results (#4) very interesting.

HP’s Smart Array consists of just a bunch of SAS interfaced disks, connected to Exchange Servers.  Its log-write latency is almost immeasurable (~0.1msec).  As log writes are primarily sequential writes and we would expect a JBOD to do well here but this seems too good.  That this (sub-)system had excellent database read and log-write latencies implies that both disks and controllers were well tuned for random and sequential I/O – in my experience, a hard thing to do without cache.  HP’s Smart Array used a mailbox database size = disk drive size, which may have resulted in good access times, but it’s unclear why.  As a counter example, Dell’s submission (#7) also used direct connected SAS drives but had a database size ~2X the size of their disks.

The other odd result in Figure 1 is the variability in Exchange 2010 database write access times.  One would think that caching subsystems would accommodate most database writes at high speed.  But, we believe this write variability is an affect of Exchange 2010’s larger database blocksizes forcing more destage activity and simulated DAG I/O activity to replicate database data for each write operation.

DAG I/O results from all the over 5K mailbox results supporting Exchange 2010 mailbox (database) resiliency.  As such, any Exchange database write must be replicated to alternate (usually 2 or 3) copies.  It’s unclear how Jetstress measures simulated replication I/O vis-a-vis database write latencies but we assume that in real Exchange 2010 environments, any database write could not complete until all DAG copies were updated.

Next we turn to database transfer counts. As ESRP mailbox counts for the over 5K mailbox category span such a wide spectrum we have elected to normalize database transfer counts to accesses per 1,000 mailboxes (1Kmbx).

(SCIESRP100728-002) (c) 2010 Silverton Consulting, All Rights Reserved

(SCIESRP100728-002) (c) 2010 Silverton Consulting, All Rights Reserved

The #1 result was the same HP Smart Array discussed above an HP Prolient server SAS connected to an D2600 disk array.  In addition, there were two IBM XIV submissions (#2 and #6) at the same mailbox count (40K) with substantive differences between the two being the mailbox size (3- vs. 1-GB for slower), drive capacity (1- vs. 2-TB for slower), and storage used (40% vs. 88% for slower).  As discussed in prior reports, the read database transfer counts are significantly higher than the write transfers, in some cases almost 2X the rate.

A couple of caveats for normalized results:

  • Normalized results may or may not scale much beyond the reported mailbox counts.  For example, the #1 result supported 6K mailboxes and may not support much more than that.
  • Normalized results can be impacted by over provisioning.  For example, the #2 result only used 40% of its storage for Exchange services allowing it to use more spindles than necessary for the workload.
(SCIESRP100728-003) (c) 2010 Silverton Consulting, All Rights Reserved

(SCIESRP100728-003) (c) 2010 Silverton Consulting, All Rights Reserved

Speaking of over provisioning, another way to look at Exchange storage performance is to normalize it over the number of disk drives or spindles used for the configuration.  Figure 3 above shows the total number of (read and write) database operations per second per drive done by each subsystem.  Here one can see the same two IBM XIV submissions in the same order discussed for Figure 2.

Some caveats for database transfers per spindle results:

  • Drive speed can help one do well on this metric, i.e. 15Krpm drives can perform better than 7.2Krpm drives and four of the top six performers used 15Krpm drives (both XIV results used 7.2Krpm drives).
  • Drive over provisioning usually reduces one’s performance on this metric.   However this was not evident with the XIV placements (the over provisioned XIV did better than the normally provisioned one).
(SCIESRP100728-004) (c) 2010 Silverton Consulting, All Rights Reserved

(SCIESRP100728-004) (c) 2010 Silverton Consulting, All Rights Reserved

Next, we show the top 10 Exchange backup throughput rates in MB/sec/database.  The #1 and #2 positions went to IBM with SVC and XIV.  Tied for #2 spot was Dell PowerVault, the other SAS connected disk system.

With Exchange 2010 and mailbox resiliency through DAGs, database backup activity no longer seems as important.  In fact, there was at least one submission that didn’t even report on this metric.  However, there are many valid reasons for database backups and we continue to believe that there will be an ongoing need for mailbox backups.  As such, reporting on backup speed need to be re-instated and preserved.

Conclusions

ESRP results for the over 5K mailbox category are always difficult to analyze due to the wide span of mailbox counts (from 6K to almost ~69K for current submissions).  That said, with the limited submissions to date, it appears for smaller mailbox counts, a properly configured SAS direct connected storage system may perform well enough.  Above that is subject to some debate but more results should help clarify this.

Nonetheless, it’s still early in ESRP v3 history.  To date there have been only 12 submissions in this category and just over 20 overall (with reports available).  Even so, we were surprised to see this many, since our last report only showed 4 results for all categories.

Finally, ESRP/Jetstress results seem designed to be difficult to compare but merit the effort.  Thus, we strive to improve our analysis with each report.  As always, feel free to contact us with any ideas on how to improve.  In that regard, our contact information can be found below or on our website at SilvertonConsulting.com.

This performance dispatch was sent out to our newsletter subscribers in July of 2010.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2010 July 28 Latest ESRP V3.0 for Exchange 2010 results (PDF 744.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/ff182054.aspx, as of 27 Julyl 2010

 

[2] All prior SCI ESRP Dispatches can be found at http://silvertonconsulting.com/cms1/dispatches/

 ESRP, ESRP v3/Exchange 2010, HP, IBM, MSA2000, XIV  1 Response »  Tagged with: , , , , , , , ,

SCI 2009 Oct 30 latest ESRP results analysis – over-5K mailboxes

Oct 302009
 

This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] performance results for the over 5000 mailboxes results category.  Prior reports discussed the over 1000 to 5000 mailboxes and the under 1000 mailboxes result categories[2].

ESRP was never intended to compare subsystem performance but rather as a proof of concept for Microsoft and storage vendors to depict a configuration supporting a given workload.  Hence, any comparisons necessarily come with some caveats and may not be real.  Nonetheless, SCI feels comparisons can well serve both the vendor and end-user storage community and thus, worth noting.

Latest ESRP V2.0 results

The first chart refers to aggregate average database backup throughput across all storage groups.  This value correlates moderately to aggregate database transfers per second.  (See figure 1).

(SCIESRP091030-001) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP091030-001) (c) 2009 Silverton Consulting, All Rights Reserved

As can be seen from Figure 1, backup activity is dominated by HDS and HP (an HDS OEM).  Partly HDS’s commanding result is due to the overall number of drives and mailboxes being serviced.  However, 3PAR’s 96Kmbx and Dell’s Equal Logic iSCSI result for 90Kmbx results only came in at number five and seven respectively.  Also, the number nine result for Pillar Data’s Axiom 600 supports only 34K mailboxes and stands almost as well as the other results supporting many more mailboxes.  A couple of caveats worth noting here database backup performance can be impacted by

  • Number of disk drives in a configuration
  • How message store databases are split across those spindles
  • Subsystem RAID level
(SCIESRP091030-002) 2009 (c) Silverton Consulting, All Rights Reserved

(SCIESRP091030-002) 2009 (c) Silverton Consulting, All Rights Reserved

This chart shows ESRP reported database latencies for read, write and log write operations.  SCI and others feel the read latency metric best shows what an end-user experience would be from a subsystem configuration.  The list is sorted by read latency.  New results from Pillar Data with a read latency of 3ms and Sun’s J4400 at 8ms showed up very well here.  A couple of considerations to note:

  • While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated.  For example if one examines the EMC SRDF/S in the number two position, its write latency is pretty high.  However if one considers that SRDF/S was active this means the data has to be written to the secondary subsystem in parallel to being written to the primary subsystem and as such its write latency does not look that bad.
  • There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage.  As ESRP reports are intended to show a viable performing solution to handle a simulated user workload we assume that these products are all optimizing cost and performance, so believe this is not an issue here.
  • For an ESRP benchmark to be accepted, read latency must be under 20 msecs.  Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware.  As such, those vendors may not show up well on this chart.
(SCIESRP091030-003) 2009 (c) Silverton Consulting, All Rights Reserved

(SCIESRP091030-003) 2009 (c) Silverton Consulting, All Rights Reserved

The new result for Pillar Data broke into the top 10 log playback time.  The top 4 systems LSI 399x, Pillar Data Axiom 600, Lefthand SANIQ, and IBM DS4800 would all be considered mid range storage subsystems although HP’s Lefthand was supporting a heavy workload at 50K mailboxes and was configured accordingly.   The range for the top ten subsystems is fairly large, over 2X from lowest to highest.  It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations.  Similar to the backup discussions above, playback performance can be impacted by:

  • Number of disk drives in a solution,
  • How message store databases are split across those spindles,
  • Subsystem RAID level,
  • Replication type
(SCIESRP091030-004) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP091030-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to normalized database transfer results and the Top 3 normalized ESRP/Jetstress results belong to HDS AMS2100, HDS AMS2300 and IBM XIV, all midrange systems. A few considerations are warranted on normalized results:

  • Normalized results do not always scale well.  Although four of these results were for 20,000 mailboxes or over, (HDS USP-V, Sun StorageTex 6540, 3PAR Inserv T800, and EMC Symmetrix at 60KMbx) the top result from HDS supported only 5400 mailboxes and may not scale much beyond that quantity of mailboxes.
  • One surprise here is the close running of everyone behind the top five results and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox.   But the workload simulated for these results varied considerably (0.3 to 1.0 iops/sec/mbx).

Conclusions

From our perspective, ESRP results in this over 5K mailbox tier are getting more competitive.  There were a number of new ESRP results in this category over the last 9 months, and at least 3 over the last quarter.  Seeing HDS, EMC and IBM highend systems, all running the same performance tests is a good indicator of their willingness to show their products in the best light as well as high customer interest in Exchange solutions.  Most likely, these vendors do not see their individual results as entirely comparable and arguably they may have a point, but we differ with them on this assessment.  Moreover, seeing DAS (HP MSA), SATA (IBM XIV), SAS (Sun, HDS AMS2100, & HP MSA), iSCSI (EMC, HP, NetApp, & Dell) and FC storage subsystem results compete in this top-tier at least on various metrics indicates to us that all storage interfaces can be competitive in mission critical applications.

ESRP/Jetstress results are inherently difficult to compare.  Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases.  Our next ESRP/Jetstress report will return to the 1K to 5K mailbox tier.  We continue to welcome any feedback on how to do better.

This performance dispatch was sent out to our newsletter subscribers in October of 2009.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 October 30 Latest ESRP results (PDF 681.0 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 27 January 2009

 

[2] All of prior SCI ESRP Dispatches can be found at ofhttp://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

 DMX-3, DMX-4, EMC, ESRP, ESRP v2/Exchange 2007, HP, IBM, Symmetrix, XP24000  No Responses »  Tagged with: , , , , , , , , , , , , , , , , , ,

SCI 2009 May 27 latest SPC benchmark results analysis

May 272009
 

We once again return to discuss the latest Storage Performance Council (SPC) results*.

SPC-1*results

There have been only five new SPC-1 results these past quarter, one for IBM DS5300, one for 3PAR InServ F400, and three for HDS AMS2500, AMS2300, and AMS2100.  None of these made the top 10 in IOPS™ or $/IOPS™ so these charts can be found in prior dispatches#.  However, two of these products did make the top 10 for IOPS/$/GB (see Figure 1 below).

(SCISPC090527-001) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090527-001) (c) 2009 Silverton Consulting, All Rights Reserved

The two new subsystems here are the 3PAR InServ F400 and the HDS AMS2500, both of which would be considered mid-class storage subsystems.  We calculate this metric as = IOPS / ($/GB) and created this metric as another way to factor in performance against cost and capacity.  Although none of these metrics can tell the complete story.

(SCISPC090527-002) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090527-002) (c) 2009 Silverton Consulting, All Rights Reserved

As for response time, the latest IBM DS5300 managed to break into the top 10 at an average least response time (LRT) of 1.77 msec.  I have been told this metric is not as important anymore but given all the interest in SSDs these days I find that hard to reconcile.

(SCISPC090527-003) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090527-003) (c) 2009 Silverton Consulting, All Rights Reserved

We have redone the IO operations/second/Drive (IOPS/Drv) chart so that it now only include drives over 140GB as the smaller drives held too great an advantage here.  All of the latest benchmark results show up in this Top 10 IOPS/Drv with HDS holding the number 4, 7, and 10 spots, IBM DS5300 at number 8, and 3PAR F400 at number 9.  Also, as mentioned in a prior report, both the J4400 and J4200 results had no RAID protection so may not be suitable comparisons for normal customer environments.

Previously this chart had an error for the ETERNUS2000 that caused us to report double its actual IOPS/Drv rate. We have fixed that error and it now shows the correct IOPS/Drive for the ETERNUS2000.

(SCISPC090527-004) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090527-004) (c) 2009 Silverton Consulting, All Rights Reserved

Figure 4 SPC-1 Bubble chart of IOPS against LRT, bubble size proportional to subsystem price

Always looking for an interesting cut on SPC-1 data we came up with this new bubble chart.  It shows a scatter plot view of subsystem performance with the x-axis as LRT and the y-axis as IOPS.  The one thing always missing from a pure performance analysis is subsystem cost, added here as bubble size.  To keep this interesting we capped the subsystems shown on the chart to a $100/GB maximum to eliminate the higher priced subsystems.

From an end-user perspective it’s interesting to note that one can obtain a reasonably performing subsystem (~100K IOPS with LRT <2msec) for about 1/4th or less the price of truly high performance subsystem.  Also, similarly priced or even more expensive subsystems can have much worse performance on an IOPS and/or a LRT basis indicating that pricing isn’t always the best factor in subsystem selection.

SPC-2, SPC-1C and SPC-2C results

There were no new SPC-2, SPC-1C and SPC-2C benchmarks released for this update and as such we stand with our last report SPC StorInt Dispatch&.

Significance

As we show above midrange subsystems can be high performers when configured properly.  Although higher end subsystems dominate the Top 10 IOPS chart, the midrange subsystems discussed here have all managed to do well on most of the other performance metrics.

Over the years we have tried to come up with ways to compare performance to price and have used both $/IOPS and IOPS/$/GB (and others on occasion) as attempts to further this analysis.  In the end we find that both of these metrics, although interesting on their own, leaves something out.  The new bubble chart (see Figure 4 above) is our latest attempt to incorporate pricing information with subsystem performance.  Hopefully, the reader will view this as a worthy addition to our SPC analysis.  As always we welcome  your feedback on how to do this better.

This performance dispatch was sent out to our newsletter subscribers in May of 2009.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 May 27 Update to SPC benchmark results (PDF 616.1 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 23 February 2009

 

# Prior SPC performance dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

& Available at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

 DS5300, HP, IBM, SPC, SPC-1  No Responses »  Tagged with: , , , , , , ,

SCI 2009 Feb 24 latest SPC benchmark results analysis

Feb 242009
 

We once again return to discuss the latest Storage Performance Council (SPC) results*.  Also, SPC has recently introduced new versions of its benchmarks to measure storage component performance.  We report on these new results below.

SPC-1*results

There have been only two new SPC-1 results these past three months one for Pillar Data Axiom 600 and the other for Sun Storage 6780.  Neither of these two made the top 10 in IOPS™, LRT™, or $/IOPS™ so these charts can be found in prior dispatches#.  However, both of these products did make the top 10 for IOPS/$/GB (see Figure 1 below)

(SCISPC090224-001) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090224-001) (c) 2009 Silverton Consulting, All Rights Reserved

The big two players here from last report are 3PAR and SVC 4.3. Again, we calculate this metric as = IOPS / ($/GB) and created this metric as another way to factor in performance against cost and capacity.

On the other hand, as has been pointed out to SCI, this metric may unfairly advantage big, monolithic subsystems at the expense of smaller subsystems.  The monolithic subsystems generate such high IOPS counts that they’re relatively expensive $/GB doesn’t impact their ranking on this chart.  In contrast, smaller

subsystems, such as Xiotech’s Emprise may be capable of putting up high IOPS rates by aggregating a number of smaller subsystems but in their current instantiation, their relatively modest $/GB doesn’t compensate for the resultant IOPS and hence they cannot compete on this chart. In such a configuration, even when taking additional switch port costs into account, may still be significantly less costly than the systems shown here.

Again it would seem the FAS3170, Sun Storage 6780 and Pillar Axiom 600 look out of place here with these multi-million $ subsystems (3PAR, IBM SVC 4.3&4.2, Sun T9990V, HDS USP-V, and HP XP2400) but seems to provide relatively good performance for its price and capacity.  Also, once again the Sun J4400 has no RAID protection whatsoever and probably does not belong here.

SPC-2 results

There were also two new SPC-2 benchmarks recorded for this update, both from Sun, one the SUN 6780 with RAID5 and the other the same system with RAID6.  Both these systems are new entries in the top 10 MBPS™

(SCISPC090224-002) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090224-002) (c) 2009 Silverton Consulting, All Rights Reserved

There was a slight degradation in performance for the RAID6 over the RAID5 version of the Sun Storage 6780 of roughly 3%.  All the other systems were reported on in prior SPC performance dispatches.

(SCISPC090224-003) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090224-003) (c) 2009 Silverton Consulting, All Rights Reserved

With this report SCI is introducing a new way to show subsystem performance for each of the three workloads that constitute the composite MPBS™ score.  Here one can see some variability in the scale of workloads each vendor’s product can attain.  There seems to be some interesting difference between the LDQ (large database query) workloads and the other two LFP (large file processing) and VOD (video on demand).

It’s unclear why the HP XP24000, HDS USP-V and the SVC 4.2 do so well on the LDQ workload as compared to the other two workloads.  SCI believes the secret lies somewhere in their respective caching algorithms being optimized for LDQ and not as well optimized for the other two workloads.  This may say that the other products such as the Fujitsu ETERNUS8000, Sun Storage 6780 and IBM DS5300 could improve their LDQ performance if they tweaked their caching algorithms somewhat.

New SPC-1C and -2C Results

SPC has been busy and have created new versions of their SPC-1 and SPC-2 benchmarks specifically to measure storage component performance.  Currently there are not many released results.  Also, SCI is having some difficulty trying to compare a 24 drive RAID subsystem against a single drive SATA system but other than that we believe the results are worthy of elaboration.

SPC-1C Results

SPC-1 results have always reported details for 10%, 50%, 80%, 90%, 95%, and 100% load levels with IOPS counts and response times in msec.  100% is based on exceeding a pre-determined response time cap, which SCI believes to be greater than 30msec.

(SCISPC090224-004) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090224-004) (c) 2009 Silverton Consulting, All Rights Reserved

For the single drive results instead of reporting IOPS™ which is at the 100% level SCI decided to use the 50% load level and plot error bars up to 80% load and down to 10%.  At the 100% load point many of these single drives are producing response times just shy of the 30msec cut off.  Unclear whether anyone would run these drives at that slow a response time.  Thus, SCI decided to show the 50% load level with error bars.  Somewhere around 80% load, the drive busy-ness takes over and how it manages its seek queue drives performance rather than seek speed.

All of these devices are running 7200RPM and have 1TB of storage.  The first drive listed uses a SAS interface, all the others use SATA.  LSI SAS3041X-R HBAs were used for all these runs. A couple of caveats here

  • Seagate sponsored all of these results.  But the results as reported were from benchmark runs executed at the SPC lab with sponsored supplied hardware.
  • There are three other SPC-1C results, for 12, 24, and 15 drive RAID systems that don’t seem comparable to these results so they have been left out of this analysis.
(SCISPC090224-005) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090224-005) (c) 2009 Silverton Consulting, All Rights Reserved

Many end users are more interested in what a storage subsystem could do at a constant 10msec response time.  As such, we introduce another new chart for the SPC-1C that depicts the expected IOPS at an average response time to 10msec.  For most drives, the IOPS shown in the chart is an interpolated value.  However for the Samsung Spinpoint F1 HE103UJ the value shown represents what they actually achieved at the 100% workload value.

For some reason both the Samsung devices seem to do well at 10msec response time while the others fall off.  This could possibly be due to just different seek optimization profiles targeted at varying workload levels.  The 50% workload level is typically closer to the 10msec mark for all the other drives.  For the other Samsung drive, the 10msec response time is closer to its 95% load.

SPC-2C results

(SCISPC090224-006) (c) 2009 Silverton Consulting, All Rights Reserved

(SCISPC090224-006) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, for the new SPC-2C results we show the various workloads in a radar chart format.  Here we can obviously see the slowness of the VOD workload vis-à-vis the other workloads.  Also there appears to be something amiss with the HDS Ultrastar drive during the LFP workload and the Samsung non-RAID class drive during the VOD workload.  Possibly another artifact of varying seek optimization profiles.

Aside from the fact that these are 7200RPM drives, SCI would have assumed this chart would have looked closer to the subsystem SPC-2 chart shown earlier.  Obviously, subsystem caching and data striping has an impact and can radically alter subsystem performance in comparison to drive performance on the same workloads.  However, the difference between all these drives for the LDQ vs. VOD workloads are significantly different than what was shown on the earlier chart.  For example, a maximum of 30% difference in performance between LDQ and VOD workloads on the subsystem chart vs. a minimum of almost 50% difference in performance for the current chart seems disquieting and worthy of more research.

The same caveats apply to SPC-2C that applied to SPC-1C results (see above).

Significance

SCI is always interested in understanding subsystem performance.  The new SPC-1C and -2C benchmark results open up a new way to judge storage components rather than storage subsystems.  Seagate has taken a shot at 7200RPM drives.  It would be even more interesting to show 10K and 15KRPM drives, as well as HBA’s.

As much as we like the new SPC-1C and -2C benchmarks, some method needs to be implemented to insure comparable benchmarks are run.  It’s unclear to SCI how to compare a 12, 15, and 24 drive storage subsystem against a single drive subsystem other than on a per drive basis.  Of course this problem also occurs with the SPC-1 and SPC-2 benchmarks as well but at least there we are talking about complete storage subsystems rather than storage components.  Perhaps as more multi-drive SPC-1C and -2C results are released we can use the per drive basis as a comparison.

Finally, SPC has once again allowed a sponsor to supply results for other vendor products (Seagate for SPC-1C and -2C results).  SCI believes that while this may be expedient, it may hurt the long run value of SPC benchmarks.  Because some non-sponsoring vendors feel slighted when not getting an opportunity to review, submit their own benchmark results, and/or reject the results entirely.  Unclear how to solve this dilemma, given SPC’s stated policy but perhaps allowing the non-SPC member vendors to review and dispute benchmark results ahead of publication may be a necessary first step.

This performance dispatch was sent out to our newsletter subscribers in Febuary of 2009.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 February 24 Update to SPC performance results (PDF 823.1 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 23 February 2009

 

# Prior SPC performance dispatches can be found at http://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

 SPC, SPC-1, SPC-2  No Responses »  Tagged with: , , , , , , , , , , ,

SCI 2009 Jan 27 latest ESRP results analysis – over 5K mailboxes

Jan 272009
 

This dispatch restarts SCI’s series on Microsoft Exchange Solution Review Program (ESRP)[1] performance results and reports on the over 5000 mailboxes results category.  Prior reports discussed the over 1000 to 5000 mailboxes and the under 1000 mailboxes result categories[2].  As such, to better compare ESRP/Jetstress results SCI reports on both normalized and un-normalized results.  For normalized results in this highest-tier category we use operations per 5000 mailbox (5Kmbx). Un-normalized results are in the appendix.

ESRP was never intended to compare subsystem performance but rather as a proof of concept for Microsoft and storage vendors to depict a configuration supporting a given workload.  Hence, any comparisons necessarily come with some caveats and may not be real.  Nonetheless, SCI feels comparisons can well serve both the vendor and end-user storage community and thus, worth noting.

Latest ESRP V2.0 results

We have added a new ranking for this analysis, which depicts the average database backup throughput across all storage groups.  This value correlates moderately to aggregate database transfers per second.  (See figure 1).

(SCIESRP090127-001) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090127-001) (c) 2009 Silverton Consulting, All Rights Reserved

As can be seen from the above chart, the new report on HP XP24000 dominates this category.  Partly HP’s commanding result is due to the overall number of mailboxes being serviced.  However, Dell’s Equal Logic iSCSI result for 90K mailboxes only came in at number four.  Also the number ten result, for HP EVA4400 supports only 8K mailboxes and stands almost as well as the other results supporting many more mailboxes.  A couple of caveats worth noting here database backup performance can be impacted by

  • Number of disk drives in a configuration
  • How message store databases are split across those spindles
  • Subsystem RAID level
(SCIESRP090127-002) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090127-002) (c) 2009 Silverton Consulting, All Rights Reserved

We add another new chart for this analysis showing the ESRP reported database latencies for read, write and log write operations.  SCI and others feel the read latency metric best shows what an end-user experience would be from a subsystem configuration.  The list is sorted by read latency.  A couple of considerations to note:

  • While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated.  For example if one examines the EMC SRDF/S in the number two position, its write latency is pretty high.  However if one considers that SRDF/S was active this means the data has to be written to the secondary subsystem in parallel to being written to the primary subsystem and as such its write latency does not look that bad.
  • There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage.  As ESRP reports are intended to show a viable performing solution to handle a simulated user workload we assume that these products are all optimizing cost and performance, so believe this is not an issue here.
  • For an ESRP benchmark to be accepted, read latency must be under 20 msecs.  Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware.  As such, those vendors may not show up well on this top 10 chart.
(SCIESRP090127-003) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090127-003) (c) 2009 Silverton Consulting, All Rights Reserved

Another new chart for this report is the log playback ranking.  This is a rather complex workload which encompasses log reading and database reading and updating (or writing).  The timings are reported as the average time in seconds it takes to playback or process a 1 MB log.  The top 3 systems LSI 399x, Lefthand SANIQ, and IBM DS4800 would all be considered mid range storage subsystems although Lefthand was supporting a heavy workload at 50K mailboxes and was configured accordingly.   The range for the top ten subsystems is fairly large over 2X from lowest to highest.  It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations.  Similar to the backup discussions above, some caveats would include:

  • Playback performance can significantly be impacted by the number of disk drives.
  • How message store databases are split across those spindles also can impact this
  • Subsystem RAID level may also impact playback performance
  • Replication type may also impact log playback performance
(SCIESRP090127-004) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090127-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to overall database transfer results and the Top 3 normalized ESRP/Jetstress results belong to HP MSA70, Sun StorageTex 6540 and EMC Symmetrix DMX-4 4500. A few considerations are warranted on normalized results:

  • Normalized results do not often scale well.  Although four of these results were for 20,000 mailboxes or over, (Sun 6540 at 20KMbx, EMC Symmetrix at 60KMbx, HDS AMS1000 at 25KMbx, and IBM DS8100 at 24KMbx) the top result from HP supported only 6000 mailboxes and may not scale much beyond that quantity of mailboxes.
  • One surprise here is the close running of everyone behind the top result and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox.  However, the over 5K mailbox tier is the only category that shows results this close to one another. (see prior ESRP StorInt Dispatchen for more information).

Conclusions

From our perspective, ESRP results in this over 5K mailbox tier are getting more competitive.  There were a number of new ESRP results in this category over the last 9 months, and at least 4 over the last quarter.  Seeing HP (an OEM version of HDS’s top product), EMC Symmetrix and IBM DS8300 running the same performance tests is a good indicator of their willingness to show their products in the best light as well as high customer interest in Exchange solutions.  Probably these vendors don’t see their individual results as entirely comparable and arguably they may have a point, but we would differ with them on this assessment.  Moreover, seeing SAS and iSCSI results compete in this top-tier at least on various metrics indicates these interfaces can provide some competition to FC storage in mission critical applications.

ESRP/Jetstress results are inherently difficult to compare.  Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases.  Our next ESRP/Jetstress report will return to the 1K to 5K mailbox tier.  We continue to welcome any feedback on how to do better.

This performance dispatch was sent out to our newsletter subscribers in January of 2009.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2009 January 27 Update to ESRP/Jetstress benchmark performance results (PDF 1.4 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

Appendix

(SCIESRP090127-005) (c) 2009 Silverton Consulting, All Rights Reserved

(SCIESRP090127-005) (c) 2009 Silverton Consulting, All Rights Reserved

Aggregate or un-normalized database transfer results are highly correlated to number of mailboxes in service and as such, are relatively less useful metrics in our opinion.  Nonetheless, it’s probably no surprise that the top results belong to HP, EMC and IBM.  Aside from the iSCSI results, this ranking is similar to one ranked purely on number of mailboxes.

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 27 January 2009

 

[2] All of prior SCI ESRP Dispatches can be found at ofhttp://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html

 DMX-4, EMC, ESRP, ESRP v2/Exchange 2007, EVA4400, HP, IBM, Symmetrix, XP24000  No Responses »  Tagged with: , , , , , , , , , , , , ,

SCI 2008 Nov 18 Latest SPC-1 and SPC-2 results analysis

Nov 182008
 

We once again return to our quarterly SPC results and as such we report on the latest benchmark submissions below.

SPC-1*results

(SCISPC081118-001) (c) 2008 Silverton Consulting, All Rights Reserved

(SCISPC081118-001) (c) 2008 Silverton Consulting, All Rights Reserved

There have been only four new SPC-1 results these past three months. IBM released a new SVC 4.3 and a new DS5300 benchmark, 3PAR released an Inserv T800 and Fujitsu has released a new ETERNUS Model 200 benchmark.  None of these results changed the LRT Top 10 chart which still stands from last May.

However, the new benchmarks did impact some of the other charts SCI maintains for SPC-1 data.  For example when we look at my favorite metric, IOPS™ and these new results have changed the top 10.

The new IBM SVC 4.3 and 3PAR results have moved into the top 10 IOPS™.  The SVC 4.3 has an 8 node configuration with 8 separate DS4700’s behind it with a total of 1536 disks.  3PAR’s Inserv T800 also had an 8 controller node configuration with only 280 disks but also costs less than the IBM SVC 4.3 system.  Of course the top result is still provided by TMS and their DRAM SSD subsystem.

(SCISPC081118-002) (c) 2008 Silverton Consulting, All Rights Reserved

(SCISPC081118-002) (c) 2008 Silverton Consulting, All Rights Reserved

As for other metrics, the SPC-1 $/IOPS™ Top 10 results have also changed but this time the culprit was the lowend Fujitsu ETERNUS2000 Model 200.   Considering the fact that it is mirrored 146GB drive storage it’s not clear why the other systems couldn’t compete at this level other than expense.  Again both the Sun J4400 and J4200 have no RAID protection and probably should not be compared to the rest of the subsystems.

(SCISPC081118-003) (c) 2008 Silverton Consulting, All Rights Reserved

(SCISPC081118-003) (c) 2008 Silverton Consulting, All Rights Reserved

Finally, the big jump up here is entirely in 3PARs favor but IBM’s SVC 4.3 has also made a significant uptick.  Again, we calculate this metric as = IOPS / ($/GB) and created this metric as another way to factor in performance against cost and capacity.  Both these products have significantly increased this metrics performance.

On the other hand, as has been pointed out to SCI, this metric may unfairly advantage big, monolithic subsystems at the expense of smaller subsystems.  The monolithic subsystems generate such high IOPS counts that their relatively expensive $/GB doesn’t impact their ranking on this chart.  In contrast, smaller subsystems, such as Xiotech’s Emprise may capable of putting up high IOPS rates by aggregating a number of smaller subsystems but in their current instantiation, their relatively modest $/GB doesn’t compensate for the resultant IOPS and hence they cannot compete on this chart. In such a configuration, even when taking additional switch port costs into account may still be significantly less costly than the systems shown on this chart.

Again the FAS3170 looks out of place here with these multi-million $ subsystems (3PAR, IBM SVC 4.3&4.2, Sun T9990V, HDS USP-V, and HP XP2400) but seems to provide relatively good performance for its price and capacity.  Also, once again the Sun J4400 has no RAID protection whatsoever and probably should not be listed here.

SPC-2 results

(SCISPC081118-004) (c) 2008 Silverton Consulting, All Rights Reserved

(SCISPC081118-004) (c) 2008 Silverton Consulting, All Rights Reserved

There were also four new SPC-2 benchmarks recorded for this update, two for the IBM DS5300 (RAID 5&6) and one from HDS USP-V and the last from HP the XP24000 (an OEM version of the HDS USP-V).  All four of these new SPC-2 results have cracked into the top 10 MPBS™.

As shown above the two HDS and HP benchmarks now reach over 8700 MBPS™ and the two DS5300s hit over 4600 MBPS™.  Once again it is somewhat surprising that a relatively low-cost subsystem such as the IBM DS5300 can compete with all these million dollar subsystems (HP XP24000, HDS USP-V, and IBM SVC 4.2, 4.1, & 3.1)

(SCISPC081118-005) (c) 2008 Silverton Consulting, All Rights Reserved

(SCISPC081118-005) (c) 2008 Silverton Consulting, All Rights Reserved

 

Another place where the latest SPC-2 submissions have broken into the top 10 is in $/MBPS™.  Perhaps it’s not surprising that the IBM’s DS5300 RAID 5 would beat out the RAID 6 version but it’s still somewhat confounding that the two HDS products with RAID 1 protection would crack into the top 10 here.  Again, both the Sun J4200 and J4400 have no data protection and should probably not be compared to these other products.

This performance dispatch was sent out to our newsletter subscribers in November of 2008.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

SCI 2008 November 18 SPC performance update (PDF 1.0 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* All results from www.storageperformance.org as of 18 November 2008

 

 DS5300, IBM, SPC, SPC-1, SPC-2, SVC  No Responses »  Tagged with: , , , , , , , , , ,

SCI 2008 Apr 23 first ESRP v2 (Exchange 2007) performance results analysis dispatch – over-5K mailboxes

Apr 232008
 

This is SCI’s first report on Microsoft Exchange Solution Review Program (ESRP)[1] performance results.  SCI found ESRP reports as a group inconsistent and somewhat difficult to compare.   However, upon further examination, we came to a clear understanding of the results and discuss our comparisons below.

Latest ESRP V2.0 results

(SCI080423-001) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080423-001) (c) 2008 Silverton Consulting, All Rights Reserved

ESRP uses a Jetstress benchmark together with a formalized reporting framework to publish results.  Jetstress benchmark variables are many, involving not only mail store database and log layouts, but also the number of Exchange servers and numerous other storage, host and software configuration options.  We have chosen to ignore most of these variables in our analysis and only use those few we deem important for comparing subsystems.

As a starting place this dispatch focuses on results for the 5000 mailboxes and over ESRP category.  This is the top and most challenging category in ESRP reports.  Future SCI performance reports will discuss the 1001 to 5000 mailbox and the 1000 and under mailbox result categories.

Moreover, Jetstress comparisons are confounded by the high correlation between database operations and the number of mailboxes tested.  In order to explore more fully this correlation effect, SCI reports on both normalized and un-normalized results.  For normalized results we use operations per 5000 mailboxes (5Kmbx) in this dispatch.  As an example, EMC Symmetrix originally reported results on 84,000 mailboxes (see Figure 2), but normalized they perform around 2800-database operations/second/5Kmbx. (see Figure 1).

Both the normalized and un-normalized results are for database transfer activity only and do not include log file transfers or other characteristics.   Also ‘DB wt/sec/5Kmbx’ is database write transfers per second per 5000 mailboxes and similarly, ‘DB rd/sec/5Kmbx’ is database read transfers per second per 5000 mailboxes.

The Top 3 normalized Jetstress results were Sun StorageTek 6540, IBM DS4800 and HDS AMS1000 storage subsystems reporting on 20-, 8-, and 25-Kmbx respectively (see Figure 1).  A few considerations on normalized results:

  • Normalized results can’t always scale all the way up to 50Kmbx.  For example while the HP MSA60 may do well at 6Kmbx it is unlikely to scale much beyond that.  On the other hand, the NetApp FAS3070 FC should scale well up to 26Kmbx.
  • Most subsystems in the top 10 use FC interfaces, the lone exception being number 4, HP MSA60 that is SAS attached.  The closest iSCSI attached subsystem came in at number 11, the DELL PowerVault MD3000i.
  • Overall the top 10 performance results span a small range, about 600-database transfers/second/5Kmbx.  The span for the top 10 un-normalized results is much larger.
(SCI080423-002) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080423-002) (c) 2008 Silverton Consulting, All Rights Reserved

As shown in the above chart un-normalized results suffer somewhat from mailbox correlation – the more mailboxes under test the higher the database operations achieved.  Although not all rankings are a function of mailbox counts, certainly the top 3 ranked results show the impact of more mailboxes.

The top 3 subsystems in this category are EMC Symmetrix, DELL Equal Logic, and Lefthand SANIQ® at 84-, 65- and 50-Kmbx respectively.  A few considerations about un-normalized results:

  • There seems to be a performance clustering around 20Kmbx as shown in the last 7 of the top 10.  The performance range for this group of results narrows, only around 6000-database transfer difference from highest to lowest.  Also, the mailbox-database transfer correlation breaks down for this group.
  • Here iSCSI results compare well against FC subsystems (e.g., see both FAS3070 results above).
  • Equal Logic and Lefthand results are also iSCSI so for the top 10 un-normalized results, 7 are FC and the rest, iSCSI.
  • We might add that these iSCSI results were with Gige interfaces.  The closest 10Gbe iSCSI result came in at number 14 for the Intransa IP SAN subsystem.

Conclusions

It seemed like normalized results was the only valid comparison to use but total database results were also of interest, especially where the number of mailboxes are similar.  Also, ESRP reports on other activity, namely log writes, database backups, latencies, and log replays.  SCI chose to focus on database activity because it seemed most valid for comparing normal Exchange mailbox activity.  Perhaps future versions of this dispatch will report on some of these other reported results as well.

As this is our first ESRP report we welcome any feedback on how to do better.  Jetstress results are inherently difficult to compare.  Hopefully our approach will prove to have merit and if so, look to future ESRP SCI StorInt™ Performance Result Dispatches to follow this lead.

This performance dispatch was sent out to our newsletter subscribers in April of 2008.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.A PDF version of this can be found at

SCI 2008 April 23 First performance results dispatch on ESRP results (PDF 711.6 KiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 23 April 2008

 

 EMC, ESRP, ESRP v2/Exchange 2007, HP, IBM, Symmetrix  No Responses »  Tagged with: , , , , , , , , , , ,

SCI 2008 Feb 19 latest SPC-1 and SPC-2 performance results analysis

Feb 192008
 

After our last performance result dispatch proved so popular we decided to split out the SpecSFS results into a separate dispatch.  Look for the new SpecSFS performance results dispatch to be released in March.

SPC-1 results

(SCI080219-001) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-001) (c) 2008 Silverton Consulting, All Rights Reserved

The latest SPC-1 results (see Figure 1) show significant improvement across many performance metrics.  HDS released SPC-1 benchmark results last October and HP and Sun, HDS OEM partners followed releasing their own results on HDS USP-V products.  TMS published new SPC-1 benchmark with #1 performance at over 290K IOPS™ and a never before seen LRT™ of under 100 microseconds.  The other major new benchmark published since our last report was for the ETERNUS8000-M1100 and it came in at #8 in the top 10 IOPS™.

In addition to these new benchmarks, NetApp has published some interesting SPC results.  NetApp’s IOPS™ results are good for a mid-range product but even more interesting is that they also published results with SnapShots active.  This marks the first time that an SPC or for that matter any formal audited benchmark result has been published with an enhanced storage service features active.   Even more provocative, NetApp released parallel results on non-NetApp equipment, namely EMC CX3 with and without active SnapViews.  More about this below.

(SCI080219-002) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-002) (c) 2008 Silverton Consulting, All Rights Reserved

As for LRT™ results (see Figure 2), TMS now holds the first two positions for the fastest IO in the industry for their current and previous generation products, with the current product reporting in with a blistering 90 microsecond LRT™.  The next two slots are held by IBM D8300 products and then coming in at #5, the newly benchmarked ETERNUS8000-M1100.  EMC CX3 with SnapView active showed surprising LRT™ results that came in under 2 msec. and is the 11th fastest storage subsystem on record.

(SCI080219-003) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-003) (c) 2008 Silverton Consulting, All Rights Reserved

In the $/IOPS™ results (see Figure 3) TMS once again dominates with their latest TMS RamSan 400 coming in at less than $0.67/IO operation.  The $/IOPS™ metric is confounded by two factors that favor TMS, IOPS rate (291K) and subsystem cost (~$200K).  The fact that the TMS subsystem is only 137GB of storage is not a factor for this metric.  After TMS, DataCore and the rest of the top 5 all come in over $4/IO operation.

(SCI080219-004) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-004) (c) 2008 Silverton Consulting, All Rights Reserved

In contrast to the $/IOPS™ results, another way to look at IO value is to use the cost of a GB storage as a factor such as IOPS/$/GB (see Figure 4).  Here, TMS’s high priced storage ($1.4K/GB) places them at a significant disadvantage and out of the top 10.  The top 5 for this metric are IBM SVC 4.2, IBM DS4700, and then the 3 HDS USP-V variants.

SPC-2 results

(SCI080219-005) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-005) (c) 2008 Silverton Consulting, All Rights Reserved

As for SPC-2 results there have only been a few new results since our last report, namely the Sun StorageTek 2530 arrays with and without RAID5.  For the second SCI report in a row three generations of IBM’s SVC take the top three MPBS™ results (see Figure 5).  The ETERNUS8000 and IBM DS8300 round out the rest of the top 5.

Neither new Sun StorageTek result came in the top 20 for MBPS™ but for $/MBPS (see Figure 6) they came in as #1 & 2 cheapest bandwidth available.  The rest of the top 6 slots in this SCI created category are dominated by Sun StorageTek products (2540 RAID 5 & mirrored and 6140 RAID5 and mirrored).

Comments on the NetApp results

(SCI080219-006) (c) 2008 Silverton Consulting, All Rights Reserved

(SCI080219-006) (c) 2008 Silverton Consulting, All Rights Reserved

SPC allows members to submit audited results on any vendor’s equipment, but NetApp’s results are the first time SPC has done this.  EMC was notified the morning of the release and had only a few hours to react.  EMC is not a current member of the Storage Performance Consortium and now probably will never be one.  SCI believes the EMC CX3 results with SnapView activity deserves some further discussion.  Consider Table 1

Table 1 LRT™ results as NetApp reported&

LRT™ results Without SnapShot and SnapView With SnapShot and SnapView
NetApp FAS3040 2.85 msec. 3.45 msec.
NetApp reported EMC CX3-M40 4.34 msec. 1.86 msec.

Something’s wrong here.  While NetApp’s response time degraded with SnapShot enabled, EMC’s reported response time improved.  After informal talks with NetApp and the Storage Performance Council, SCI realizes that the results reported for the LRT™ metric did not have any SnapView active.   NetApp and SPC state this was to EMC’s benefit and showing up as the 11th fastest LRT™, it definitely helped.  However, it’s an unfair to other LRT™ results to do this.

All this presents an interesting problem – what is the proper way to benchmark storage subsystem with advanced features active?  Snapshot is a relatively straightforward feature but its use is dictated by a number of concerns and its operational profile can become quite complex to figure out.  For example, what’s the right frequency to use snapshot, when do you deactivate/delete a snapshot volume(s), how much storage should be snapshot, etc.  Operational profiles for snapshot use in backup are vastly different than snapshot use in testbed creation.  Are NetApp’s usage models appropriate?

In all fairness, the full disclosure reports for both NetApp’s FAS3040 and NetApp reported EMC CX-M40 results described how SnapShot and SnapView were used during the testing and they were different.  (NetApp issued SnapShots every 15 minutes on FAS3040 and issued SnapView every hour on EMC CX3-M40 and as stated above did not issue SnapView during the LRT™ testing.)  Once again, this was to EMC’s advantage, but how can they be truly compared unless SnapShot/SnapViews are used at the same frequency.

The benchmark business is fraught with vendors sparing no expense to field a system showing the best results.  Is it any wonder that aside from TMS, the other top 5 IOPS™ results were on storage subsystems costing more than $3 million each.  Although NetApp made an attempt to optimize EMC’s subsystem, is this what EMC would have done?  We will never know.  SCI believes the right answer here is to not allow one vendor to publish results for another and when publishing results with storage features.  When publishing benchmarks with advanced storage features operational profiles have to be established ahead of time and adhered to throughout the benchmark’s various phases and if that’s impossible the benchmark needs to be withdrawn.

This performance dispatch was sent out to our newsletter subscribers in February of 2008.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.A PDF version of this can be found at

SCI 2008 February 19 SPC Performance Results Update (PDF 1.4 MiB)

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

* SPC-1 and SPC-2 results are from the Storage Performance Council website http://www.storageperformance.org.

 

& Full disclosure reports are in file numbers A00057, A00058, A00059, and A00060, available at http://www.storageperformance.org as of 14 Feb 2008.

 CLARiion, CX3, EMC, HDS, NetApp, SPC, SPC-1, SPC-2  No Responses »  Tagged with: , , , , , , , , , ,
For a no risk, free 30 minute consultation please give us a call at +1-720-221-7270

If you are interested in receiving our free monthly newsletter on major storage announcements and storage performance results please subscribe by email
Silverton Consulting, Inc., P.O. Box 492
Broomfield, CO, 80038-0492, USA		 © 2004-2012 Silverton Consulting, Inc., All Rights Reserved
Contact|SCI Privacy Policy|Sitemap		 Suffusion theme by Sayontan Sinha