Testing storage systems – Intel’s SSD fix

Intel’s latest (35nm NAND) SSD shipments were halted today because a problem was identified when modifying BIOS passwords (see IT PRO story). At least they gave a timeframe for a fix – a couple of weeks.

The real question is can products be tested sufficiently these days to insure they work in the field. Many companies today will ship product to end-user beta testers to work out the bugs before the product reaches the field. But beta-testing has got to be complemented with active product testing and validation. As such, unless you plan to get 100s or perhaps 1000s of beta testers you could have a serious problem with field deployment.

And therein lies the problem, software products are relatively cheap and easy to beta test, just set up a download site and have at it. But with hardware products beta testing actually involves sending product to end-users which costs quite a bit more $’s to support. So I understand why Intel might be having problems with field deployment.

So if you can’t beta test hardware products as easily as software – then you have to have a much more effective test process. Functional testing and validation is more of an art than a science and can cost significant $’s and more importantly, time. All of which brings us back to some form of beta testing.

Perhaps Intel could use their own employees as beta testers rotating new hardware products from one organization to another, over time to get some variability in the use of a new product. Many companies use their new product hardware extensively in their own data centers to validate functionality prior to shipment. In the case of Intel’s SSD drives these could be placed in the in-numberable servers/desktops that Intel no-doubt has throughout it’s corporation.

One can argue whether beta testing takes longer than extensive functional testing. However given today’s diverse deployments, I believe beta testing can be a more cost effective process when done well.

Intel is probably trying to figure out just what went wrong in their overall testing process today. I am sure, given their history, they will do better next time.

On Storage Benchmarks

What is it about storage benchmarks that speaks to me? Is it the fact that they always present new data on current products, that there are always some surprises, or that they always reveal another facet of storage performance.

There are some that say benchmarks have lost their way, become too politicized, and as a result, become less realistic. All these faults can and do happen but it doesn’t have to be this way. Vendors can do the right thing if enough of them are engaged and end-users can play an important part as well.

Benchmarks exist mainly to serve the end-user community, by supplying an independent, audit-able, comparison of storage subsystem performance. To make benchmarks more useful, end users can help insure that they model real-world workloads. But this only happens when end-users participate in benchmark organizations, understand benchmark workloads, and understand in detail, their own I/O workloads. Which end-users can afford to do this, especially today?

As a result, storage vendors take up the cause. They argue amongst themselves to define “realistic end-user workloads”, put some approximation out as a benchmark and tweak it over time. The more storage vendors, the better this process becomes.

When I was a manager of storage subsystem development, I hated benchmark results. Often it meant there was more work to do. Somewhere, somehow or someway we weren’t getting the right level of performance from our subsystem. Something had to change. We would end up experimenting until we convinced ourselves we were on the right track. That lasted until we exhausted that track and executed the benchmark again. It almost got to the point where I didn’t really want to know the results – almost but not quite. In the end, benchmarks caused us to create better storage, to understand the best of the storage world, and to look outside ourselves at what others could accomplish.

Is storage performance still important today? I was talking with a storage vendor a couple of months back who said that storage subsystems today perform so well that performance is no longer a major differentiator or a significant buying consideration. I immediately thought why all the interest in SSDs and 8GFC. To some extent I suppose, raw storage performance is not as much a concern today but it will never go away completely.

Consider the automobile, it’s over a century old now (see Wikipedia) and we still talk about car performance. Perhaps it’s no longer raw speed, but a car’s performance still matters to most of us. What’s happened over time is that the definition of car performance has become more differentiated, more complex – top speed is not the only metric anymore. I am convinced that similar differentiation will happen to storage performance and storage benchmarks must lead the way.

So my answer is yes, storage performance still matters and benchmarks ultimately define storage performance. It’s up to all of us to keep benchmarks evolving to match the needs of end-users.

Nowadays, I can enjoy looking at storage benchmarks and leave the hard work to others.