I have blogged about the poor deduplication ratios seen when using Oracle 10G RMAN compression before (see my prior post) but not everyone uses compressed backupsets. As such, the question naturally arises as how well RMAN non-compressed backupsets deduplicate.
RMAN backup types
Oracle 10G RMAN supports both full and incremental backups. The main potential for deduplication would come when using full backups. However, 10G also supports something called RMAN cumulative incremental backups in addition to the more normal differential backups. Cumulative incrementals backs up all changes since the last full and as such, could readily duplicate many changes which occur between full backups also leading to higher deduplication rates.
RMAN multi-threading
In any event, the other issue with RMAN backups is Oracle’s ability to multi-thread or multiplex backup data. This capability was originally designed to keep tape drives busy and streaming when backing up data. But the problem with file multiplexing is that file data is intermixed with blocks from other files within a single data backup stream, thus losing all context and potentially reducing deduplication ability. Luckily, 10G RMAN file multiplexing can be disabled by setting FILESPERSET=1, telling Oracle to provide only a single file per data stream.
Oracle’s use of meta-data in RMAN backups also makes them more difficult to deduplicate but some vendors provide workarounds to increase RMAN deduplication (see Quantum DXI, EMC Data Domain and others).
—-
So deduplication of RMAN backups will vary depending on vendor capabilities as well as admin RMAN backup specifications. As such, to obtain the best data deduplication of RMAN backups follow deduplication vendor best practices, use periodic full and/or cumulative incremental backups, don’t use compressed backupsets, and set FILESPERSET=1.
Aurora's Perception or I Schrive When I See Technology by Wonderlane (cc) (from Flickr)
Some of these technologies were in development prior to 2000, some were available in other domains but not in storage, and some were in a few subsystems but had yet to become popular as they are today. In no particular order here are my top 10 storage technologies for the decade:
NAND based SSDs– DRAM and other technology solid state drives (SSDs) were available last century but over the last decade NAND Flash based devices have dominated SSD technology and have altered the storage industry forever more. Today, it’s nigh impossible to find enterprise class storage that doesn’t support NAND SSDs.
GMR head– Giant Magneto Resistance disk heads have become common place over the last decade and have allowed disk drive manufacturers to double data density every 18-24 months. Now GMR heads are starting to transition over to tape storage and will enable that technology to increase data density dramatically
Data Deduplication – Deduplication technologies emerged over the last decade as a complement to higher density disk drives as a means to more efficiently backup data. Deduplication technology can be found in many different forms today, ranging from file and block storage systems, backup storage systems, to backup software only solutions.
Thin provisioning – No one would argue that thin provisioning emerged last century but it took the last decade to really find its place in the storage pantheon. One almost cannot find a data center class storage device that does not support thin provisioning today.
Scale-out storage – Last century if you wanted to get higher IOPS from a storage subsystem you could add cache or disk drives but at some point you hit a subsystem performance wall. With scale-out storage, one can now add more processing elements to a storage system cluster without having to replace the controller to obtain more IO processing power. The link reference talks about the use of commodity hardware to provide added performance but scale-out storage can also be done with non-commodity hardware (see Hitachi’s VSP vs. VMAX).
Storage virtualization – server virtualization has taken off as the dominant data center paradigm over the last decade but a counterpart to this in storage has also become more viable as well. Storage virtualization was originally used to migrate data from old subsystems to new storage but today can be used to manage and migrate data over PBs of physical storage dynamically optimizing data placement for cost and/or performance.
LTO tape – When IBM dominated IT in the mid to late last century, the tape format dejour always matched IBM’s tape technology. As the decade dawned, IBM was no longer the dominant player and tape technology was starting to diverge into a babble of differing formats. As a result, IBM, Quantum, and HP put their technology together and created a standard tape format, called LTO, which has become the new dominant tape format for the data center.
Cloud storage – Unclear just when over the last decade cloud storage emerged but it seemed to be a supplement to cloud computing that also appeared this past decade. Storage service providers had existed earlier but due to bandwidth limitations and storage costs didn’t survive the dotcom bubble. But over this past decade both bandwidth and storage costs have come down considerably and cloud storage has now become a viable technological solution to many data center issues.
iSCSI – SCSI has taken on many forms over the last couple of decades but iSCSI has the altered the dominant block storage paradigm from a single, pure FC based SAN to a plurality of technologies. Nowadays, SMB shops can have block storage without the cost and complexity of FC SANs over the LAN networking technology they already use.
FCoE – One could argue that this technology is still maturing today but once again SCSI has taken opened up another way to access storage. FCoE has the potential to offer all the robustness and performance of FC SANs over data center Ethernet hardware simplifying and unifying data center networking onto one technology.
No doubt others would differ on their top 10 storage technologies over the last decade but I strived to find technologies that significantly changed data storage that existed in 2000 vs. today. These 10 seemed to me to fit the bill better than most.
Macroblocking: demolish the eerie ▼oid by Rosa Menkman (cc) (from Flickr)
Last week NetApp announced the availability of data compression on many of their unified storage platforms, which includes block and file storage. Earlier this year EMC announced data compression for LUNs on CLARiion and Celerra. I must commend both of them for re-integrating data compression back into primary storage systems, missing since IBM and Sun stopped marketing RVA and SVA.
Data compression algorithms
Essentially data compression is an algorithm that eliminates redundancy in data streams. Data compression can be “lossy” or “loss-less”. Data compression in storage subsystems is typically loss-less which means that the original data can be reconstructed without any loss of information. One sees lossy algorithms in video/audio data compression which doesn’t impact video/audio fidelity unless it results in significant loss of data.
One simple example of loss-less data compression is Run-Length Encoding which substitutes a trigger, count, and character string for any character repeated more than 4 times in a block of data. This compresses well any text strings with lots of blanks, numerical data with lot’s of 0’s and initial format data written with a repeating character.
There are other, more sophisticated compression algorithms like Huffman Coding, which identify the most frequent bytes in a block of data and replace these bytes with shorter bit patterns. For example if ~50% of the characters in a text file are the letters “a”, “e”, “i”, “o”, “t”, and “n” (see Wikipedia, Frequency Analysis) then these characters can take up much less space if we encode them 4 or less bits rather than the 8-bits in a byte.
I am certain that both EMC and NetApp are using much more sophisticated algorithms than either of these and it wouldn’t surprise me to know they are using something like the open source algorithm like Zlib (gzip) or Bzip2 (see my Poor deduplication with Oracle RMAN compressed backups post for an explanation) which uses Huffman Coding and adds even more sophistication. Data compression algorithms like these could offer something like 50% compression, i.e., your data could be stored in 50% less space.
Data compression is often confused with Data Deduplication but it’s not the same. Deduplication looks for duplicate data across different data blocks and files while data compression is strictly only examining the data stream within a block or file and doesn’t depend on any other data.
Storage system data compression
In the past, data compression was relegated to a separate appliance, tape storage systems, and/or host software. By integrating these algorithms into their main storage engines, both NetApp and EMC are taking advantage of the recent processor speed increases being embedding into their systems to offer offline functionality for online data.
Historically, compression algorithms such as these were implemented in hardware but nowadays they can easily be done in software by being relegated to operate during off-peak IO time or execute as the lowest priority task in the storage system. As such, there can be no guarantee when your data will finally be compressed but it will be compressed eventually.
Data compression like this is great for data that isn’t modified frequently. It takes some processing time to compress data and as such, would need to be repeated after every modification of a compressed block or file. So if the data isn’t modified that much, compression’s processing cost could be amortized over longer data lifetimes.
Further, data compression must be undone at read time, i.e., the data needs to be de-compressed and handed off to the IO requesting it. De-compression is a much faster algorithm than compression because in the case of something like Huffman Coding the character dictionary is already known and as such, it’s just a matter of table lookup and bit field isolation. It would be convenient if this data were sitting in the system DRAM someplace but lacking that, moving it from cache to DRAM could be done quickly enough, processed there, and then moved back before final transfer to the requesting IO.
As such, data compression may impact response time for compressed data reads. How much of an impact is yet TBD.
Data writes will not be impacted at all because the compression activity is done much later. Whether the data stays in cache until compressed or is brought back in at some later time is another algorithm question which may impact cache hit rates/compression performance but this doesn’t have to be a serious impediment.
NetApp is able to offer this capability for both block and file storage because of it’s WAFL backend data structure which essentially allows it to create variable length blocks for file and block data. EMC only offers this for LUN data (block storage) as of yet but it’s probably just a matter of time before it’s available for other data as well.
