(Back of) Western Digital's Thunderbolt Duo (from their website)
Western Digital (WD) just released their new Digital MyBook Thunderbold Duo the other day and it features 2-2TB or -3TB disks and of course you can daisy chain up to 6 of these together just in case, for up to 36TB on a Mac.
I have been happy with my desktop storage which has been running about 80% full. Plus I have a 1TB time machine external drive for online backups which I use more than I care to admit. But what the heck am I going to do with 36TB.
Enter Apple TV
Well, now that the new Apple TV is out and it supports 1080p video that problem might be solved. I am starting to think of transfering my entire DVD/BlueRay collection to digital format and loading it all on iTunes. That way I could use Airplay and Apple TV to play it to a TV.
This is where the 6 to 36TB of storage could come in handy. Especially if I wasn’t interested in streaming everything off of iCloud and having a local iTunes repository onsite for all my videos.
Digital video for the iPad
Today, I don’t have a lot of videos on my desktop, mostly ones I wanted to view on my iPad so, they are highly compressed and only take up about 1GB per video (Handbrake encoded from DVDs).
I am thinking the new 1080p iTunes encoded videos would take up more space at least 4-5GB per video but would still be considerably better than 9GB for DVD and ~36GB for BluRay, high definition videos.
Given current storage I could probably handle converting my current iPad videos over to the 1080p version (if I actually owned them in hi-def) but if I wanted to put the rest of my video library on my desktop I don’t have enough space.
Bulk storage meet the Mac
Then WD came out with their new Thunderbolt Duo drives. It seems to have it all, Thunderbolt I/O at 10Gbps, with all the storage I could possibly need. Presumably the 2 or 3TB drives are 5400 or 7200 SATA 3.0 drives. But they are user swappable, so could concievably be changed out to whatever comes out next but probably in pairs.
Of course with SATA 3.0 they can only go 6Gbps to the disks, but it’s not a bad match to have 2 drives per single bi-directional Thunderbolt channel. Although whether 6 of these daisy chained on a single Thunderbolt cable would generate decent performance is another question. Then again, how much performance can one Mac use?
I suppose my next steps are to upgrade my Mac to hardware that supports Thunderbolt, get Apple TV, buy a Duo drive or two and then start encoding my DVD/BluRay library.
But that’s too logical, instead maybe I’ll just get Apple TV and give iCloud a try, at least for awhile and save the WD Duo for the next evolution. Maybe by then WD have come out with their 4TB drives, providing 8TB per Duo.
I must admit, even though I have disparaged DVD archive life (see CDs and DVDs longevity questioned) I still backup my work desktops/family computers to DVD and DVDdl disks. It’s cheap (on sale 100 DVDs cost about $30 and DVDdl ~2.5 that much) and it’s convenient (no need for additional software, outside storage fees, or additional drives). For offsite backups I take the monthly backups and store them in a safety deposit box.
But my partner (and wife) said “Your time is worth something, every time you have to swap DVDs you could be doing something else.” (… like helping around the house.)
She followed up by saying “Couldn’t you use something that was start it and forget it til it was done.”
Well this got me to thinking (as well as having multiple media errors in my latest DVDdl full backup), there’s got to be a better way.
The options for SOHO (small office/home office) Offsite backups look to be as follows: (from sexiest to least sexy)
Cloud storage for backup – Mozy, Norton Backup, Gladinet, Nasuni, and no doubt many others can provide secure, cloud based backup of desktop, laptop data for Macs and Window systems. Some of these would require a separate VM or server to connect to the cloud while others would not. Using the cloud might require the office systems to be left on at nite but that would be a small price to pay to backup your data offsite. Benefits to cloud storage approaches are that it would get the backups offsite, could be automatically scheduled/scripted to take place off-hours and would require no (or minimal) user intervention to perform. Disadvantages to this approach is that the office systems would need to be left powered on, backup data is out of your control and bandwidth and storage fees would need to be paid.
RDX devices – these are removable NFS accessed disk storage which can support from 40GB to 640GB per cartridge. The devices claim 30yr archive life, which should be fine for SOHO purposes. Cost of cartridges is probably RDX greatest issue BUT, unlike DVDs you can reuse RDX media if you want to. Benefits are that RDX would require minimal operator intervention for anything less than 640GB of backup data, backups would be faster (45MB/s), and the data would be under your control. Disadvantages are the cost of the media (640GB Imation RDX cartridge ~$310) and drives (?), data would not be encrypted unless encrypted at the host, and you would need to move the cartridge data offsite.
LTO tape – To my knowledge there is only one vendor out there that makes an iSCSI LTO tape and that is my friends at Spectra Logic but they also make a SAS (6Gb/s) attached LTO-5 tape drive. It’s unclear which level of LTO technology is supported with the iSCSI drive but even one or two generations down would work for many SOHO shops. Benefits of LTO tape are minimal operator intervention, long archive life, enterprise class backup technology, faster backups and drive data encryption. Disadvantages are the cost of the media ($27-$30 for LTO-4 cartridges), drive costs(?), interface costs (if any) and the need to move the cartridges offsite. I like the iSCSI drive because all one would need is a iSCSI initiator software which can be had easily enough for most desktop systems.
DAT tape – I thought these were dead but my good friend John Obeto informed me they are alive and well. DAT drives support USB 2.0, SAS or parallel SCSI interfaces. Although it’s unclear whether they have drivers for Mac OS/X, Windows shops could probably use them without problem. Benefits are similar to LTO tape above but not as fast and not as long a archive life. Disadvantages are cartridge cost (320GB DAT cartridge ~$37), drive costs (?) and one would have to move the media offsite.
(Blu-ray, Blu-ray dl), DVD, or DVDdl – These are ok but their archive life is miserable (under 2yrs for DVDs at best, see post link above). Benefits are they’res very cheap to use, lowest cost removable media (100GB of data would take ~22 DVDs or 12 DVDdls which at $0.30/ DVD or $0.75 for DVDdl thats ~$6.60 to $9 per backup), and lowest cost drive (comes optional on most desktops today). Disadvantages are high operator intervention (to swap out disks), more complexity to keep track of each DVDs portion of the backup, more complex media storage (you have a lot more of it), it takes forever (burning 7.7GB to a DVDdl takes around an hour or ~2.1MB/sec.), data encryption would need to be done at the host, and one has to take the media offsite. I don’t have similar performance data for using Blu-ray for backups other than Blu-ray dl media costs about $11.50 each (50GB).
Please note this post only discusses Offsite backups. Many SOHOs do not provide offsite backup (risky??) and for online backups I use a spare disk drive attached to every office and family desktop.
Probably other alternatives exist for offsite backups, not the least of which is NAS data replication. I didn’t list this as most SOHO customers are unlikely to have a secondary location where they could host the replicated data copy and the cost of a 2nd NAS box would need to be added along with the bandwidth between the primary and secondary site. BUT for those sophisticated SOHO customers out there already using a NAS box for onsite shared storage maybe data replication might make sense. Deduplication backup appliances are another possibility but suffer similar disadvantages to NAS box replication and are even less likely to be already used by SOHO customers.
—-
Ok where to now. Given all this I M hoping to get a Blu-ray dl writer in my next iMac. Let’s see that would cut my DVDdl swaps down by ~3.2X for single layer and ~6.5X for dl Blu-ray. I could easily live with that until I quadrupled my data storage, again.
Although an iSCSI LTO-5 tape transport would make a real nice addition to the office…
A recent article in Rolling Stone (File Not Found: The Record Industry’s Digital Storage Crisis) laments the fact that digital recordings can go out of service due to format changes, plugin changes, and/or files not being readable (file not found).
In olden days, multi-track masters were recorded on audio tape and kept in vaults. Audio tape formats never seemed to change or at least changed infrequently, and thus, re-usable years or decades after being recorded. And the audio tape drives seemed to last forever.
Digital audio recordings on the other hand, are typically stored in book cases/file cabinets/drawers, on media that can easily become out-of-date technology (i.e., un-readable) and in digital formats that seem to change with every new version of software.
Consumer grade media doesn’t archive very well
The article talks about using hard drives for digital recordings and trying to read them decades after they were recorded. I would be surprised if they still spin up (due to stiction) let alone still readable. But even if these were CDs or DVDs, the lifetime of consumer grade media is not that long, maybe a couple of years at best, if treated well and if abused by writing on them or by bad handling, it’s considerably less than that.
Digital audio formats change frequently
The other problem with digital audio recordings is that formats go out of date. I am no expert but let’s take Apple’s Garage Band as an example. I would be surprised if 15 years down the line that a 2010 Garage Band session recorded today was readable/usable with Garage Band 2025, assuming it even existed. Sounds like a long time but it’s probably nothing for popular music coming out today.
Solutions to digital audio media problems
Audio recordings must use archive grade media if it’s to survive for longer than 18-36 months. I am aware of archive grade DVD disks but have never tested any, so cannot speak to their viability in this application. However, for an interesting discussion on archive quality CD&DVD media see How to choose CD/DVD archival media. But, there are other alternatives.
Removable data center class archive media today includes magnetic tape, removable magnetic disks or removable MO disks.
Magnetic tape – LTO media vendors specify archive life on the order of 30 years, however this assumes a drive exists that can read the media. The LTO consortium states that current generation drives will read back two generations (LTO-5 drive today reads LTO-4 and LTO-3 media) and write back one generation (LTO-5 drive can write on LTO-4 media [in LTO-4 format]). With LTO generations coming every 2 years or so, it would only take 6 years for a LTO volume, recorded today to be unreadable by current drives. Naturally, one could keep an old drive around but maintenance/service would no longer be available for it after a couple of years. LTO drives are available from a number of vendors.
Magnetic disk – The RDX Storage Alliance claims a media archive life of 30 years but I wonder whether a RDX drive would exist that could read it and the other question is how archive life was validated. Today’s removable disk typically imitates a magnetic tape drive/format. The most prominent removable disk vendor is ProStor Systems but there are others.
Magneto-optical (MO) media – Plasmon UDO claims a media life of 50+ years for their magneto-optical media. UDO has been used for years to record check images, medical information and other data. Nonetheless, recently UDO technology has not been able to keep up with other digital archive solutions and have gained a pretty bad rap for usability problems. However, they plan to release a new generation of UDO product line in 2010 which may shake things up if it arrives and can address their usability issues.
Finally, one could use non-removable, high density disk drives and migrate the audio data every 2-3 years to new generation disks. This would keep the data readable and continuously accessible. Modern storage systems with RAID and other advanced protection schemes can protect data from any single and potentially double drive failure but as drives age, their error rate goes up. This is why the data needs to be moved to new disks periodically. Naturally, this is more frequently than magnetic tape, but given disk drive usability and capacity gains, might make sense in certain applications.
As for removable USB sticks – unclear what the archive life is for these consumer devices but potentially some version that went after the archive market might make sense. It would need to be robust, have a long archive life and be cheap enough to compete with all the above. I just don’t see anything here yet.
Solutions to digital audio format problems
There needs to be an XML-like description of a master recording that reduces everything to a more self-defined level which describes the hierarchy of the recording, and provides object buckets for various audio tracks/assets. Plugins that create special effects would need to convert their effects to something akin to a MPEG-like track that could be mixed with the other tracks, surrounded by meta-data describing where it starts, ends and other important info.
Baring that, some form of standardization on a master recording format would work. Such a standard could be supported by all major recording tools and would allow a master recording to be exported and imported across software tools/versions. As this format evolved, migration/conversion products could be supplied to upgrade old formats to new ones.
Another approach is to have some repository for current master audio recording formats. As software packages go out of date/business, their recording format could be stored in some “format repository”, funded by the recording industry and maintained in perpetuity. Plug-in use would need to be documented similarly. With a repository like this around and “some amount” of coding, no master recording need be lost to out-of-date software formats.
Nonetheless, If your audio archive needs to be migrated periodically, it be a convenient time to upgrade the audio format as well.
—-
I have written about these problems before in a more general sense (see Today’s data and the 1000 year archive) but the recording industry seems to be “leading edge” for these issues. When Producer T Bone Burnett testifies at a hearing that “Digital is a feeble storage medium” it’s time to step up and take action.
Digital storage is no more feeble than analog storage – they each have their strengths and weaknesses. Analog storage has gone away because it couldn’t keep up with digital recording densities, pricing, and increased functionality. Just because data is recorded digitally doesn’t mean it has to be impermanent, hard to read 15-35 years hence, or in formats that are no longer supported. But it does take some careful thought on what storage media you use and on how you format your data.
In a recent article from BBC on Should you store treasured data on (optical) disk the conclusion was that CDs and DVDs have significantly worse archive life than advertised or even suspected until recently. The study done by the French National Centre for Scientific Research discovered that the reliability of a few optical disks was just over one year and most “rarely lasted longer than five to 10 years” although they were advertised to last significantly more.
There was not much detail in the BBC article and searching (in English) for the original research yielded nothing pertaining to the topic. However, the article did say that the centre used accelerated life testing with heat, water vapor and light (standard IT industry practice) to determine point of failure and that products under the same brand had significant archive life variability due to multiple manufacturers. They also stated that branding the discs might be impacting longevity as well. And that it appeared that the more than seven miles of (probably DVD) data recorded on the discs is deteriating faster than anticipated.
As a result, they suggested that data on optical disks should be copied every two to three years and maybe as time moves on, this can be done less frequently assuming optical disk lifespans improve. Also important data should be spread across multiple storage formats.
The case for (IT) tape in video archives
Nonetheless, the article did mention that a 52 minute documentary typically requires about 500GB of high definition video to be recorded and at the moment that video is normally stored on data (tape) cassettes and hard drives. In my experience these (video) tapes were specific to the recording equipment vendor, i.e. Panasonic, Sony, or others and as such, relatively expensive. But nowadays, this data can also be stored on LTO or other IT tapes. In contrast to the above, LTO tape has an archival storage life of around 30 years (depending on vendor) and can be had at reasonable cost.
Also, in the past I was aware of a number of TV broadcasters that had an archive of finished broadcasts residing only on DVDs. They typically took one additional copy of a DVD and stored them both in their desks or file cabinets. Many of these people will be very surprised when five years down the line, they go to access their archived broadcasts and find that they can no longer be read. Of course, I have made the same mistake with my family video archive stored on DVDs.
Video archives whether of raw video or finished broadcasts require large capacity, sequentially accessed storage which seems ideal for automated LTO or other magnetic tape storage. By using IT tape data storage for video archives, one can benefit from technology advances in density and throughput that happen every couple of years, benefit from volume manufacturing available to IT product manufacturers, and benefit from a significantly longer archive life.
Now if I can just find a USB LTO tape drive that works on the Mac for my home videos and family backups I would feel much better, …
