Although deduplication for primary storage has been shipping on the Hitachi NAS (HNAS) product line since last January, we only heard about it this past month.
Deduplication capabilities have been added to Version 11 of the HNAS software. Any current 3080, 3090 or Hitachi Unified Storage File Module (HUS FM)customers can upgrade up to this level to get the new functionality. HNAS already uses its FPGA to offload some performance intensive file/storage services today. Now they’ve added automated, auto-throttling deduplication capabilities to the FPGA. This way there is little to no impact on write performance. Hitachi’s patented deduplication technology leverages an advanced cryptographic hashing algorithm on a fixed block basis (4KB or 32KB blocks) to ensure data integrity.
As for read performance, you may recall the HNAS implements an object storage system as its underlying storage architecture. Which means that it already processes lists of pointers to blocks of data, so reconstructing a data block for reading has no impact on performance.
The major function of deduplication is space reclamation, where redundant data is eliminated and wasted capacity is freed up. With HNAS, space reclamation is done in the background and the system throttles this activity down if it’s running at more than 50% of its IOPS capabilities to avoid impacting file serving performance.
The base HNAS Deduplication license, which enables one of four deduplication engines, is provided at no cost and HDS encourages all current customers to try it out. If you need more deduplication engines for larger amounts of data, they can be licensed at a small additional cost. Also, deduplication can be enabled on a file system basis.
While not every primary file storage provider supports deduplication, more are supplying this option over time. Although the capacity savings provided by deduplication can be substantial, it should not come at the cost of impacting performance. And while HDS is a little late to this party they are here now and they have delivered a solution that their customers can take advantage of today to substantially increase their storage capacity utilization without impacting file serving performance.
[This storage announcement dispatch was originally sent out to our newsletter subscribers in April of 2013. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports.]
Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community.