What is it about storage benchmarks that speaks to me? Is it the fact that they always present new data on current products, that there are always some surprises, or that they always reveal another facet of storage performance.
There are some that say benchmarks have lost their way, become too politicized, and as a result, become less realistic. All these faults can and do happen but it doesn’t have to be this way. Vendors can do the right thing if enough of them are engaged and end-users can play an important part as well.
Benchmarks exist mainly to serve the end-user community, by supplying an independent, audit-able, comparison of storage subsystem performance. To make benchmarks more useful, end users can help insure that they model real-world workloads. But this only happens when end-users participate in benchmark organizations, understand benchmark workloads, and understand in detail, their own I/O workloads. Which end-users can afford to do this, especially today?
As a result, storage vendors take up the cause. They argue amongst themselves to define “realistic end-user workloads”, put some approximation out as a benchmark and tweak it over time. The more storage vendors, the better this process becomes.
When I was a manager of storage subsystem development, I hated benchmark results. Often it meant there was more work to do. Somewhere, somehow or someway we weren’t getting the right level of performance from our subsystem. Something had to change. We would end up experimenting until we convinced ourselves we were on the right track. That lasted until we exhausted that track and executed the benchmark again. It almost got to the point where I didn’t really want to know the results – almost but not quite. In the end, benchmarks caused us to create better storage, to understand the best of the storage world, and to look outside ourselves at what others could accomplish.
Is storage performance still important today? I was talking with a storage vendor a couple of months back who said that storage subsystems today perform so well that performance is no longer a major differentiator or a significant buying consideration. I immediately thought why all the interest in SSDs and 8GFC. To some extent I suppose, raw storage performance is not as much a concern today but it will never go away completely.
Consider the automobile, it’s over a century old now (see Wikipedia) and we still talk about car performance. Perhaps it’s no longer raw speed, but a car’s performance still matters to most of us. What’s happened over time is that the definition of car performance has become more differentiated, more complex – top speed is not the only metric anymore. I am convinced that similar differentiation will happen to storage performance and storage benchmarks must lead the way.
So my answer is yes, storage performance still matters and benchmarks ultimately define storage performance. It’s up to all of us to keep benchmarks evolving to match the needs of end-users.
Nowadays, I can enjoy looking at storage benchmarks and leave the hard work to others.
