We were talking with Ursheet Parikh at StorSimple today about their new cloud gateway product (to be covered in a future post) when at the end of the talk he described some IP they have to handle cloud storage’s “eventual consistency“. Dumbfounded, I asked him to clarify, having never heard this term before.
Apparently, eventual data consistency is what you get when you use most cloud storage providers. With eventual consistency they will not guarantee that when you read back an object that has been recently updated that you will get the latest copy.
In contrast, “immediate consistency” means that if you update an object the cloud storage provider guarantees the latest version will be supplied for any and all subsequent read backs. To me all storage up until cloud storage guaranteed immediate consistency otherwise it was considered a data integrity failure.
To explain, cloud storage providers have multiple copies of any object replicated about that must be updated throughout their environment. As such, they cannot guarantee that you will read back an updated version versus one of the downlevel one(s)- Yikes!
What does this mean for your cloud storage?
First, Microsoft’s Azure cloud storage is the only provider that guarantees immediate consistency but in order to do so has made some restrictions on object size. But this means all the other cloud storage providers only guarantee eventual consistency.
Second, cloud storage with eventual consistency guarantee should not be used for data that’s updated frequently and then read back. It’s probably ok for archive or backup storage (that’s not restored for awhile) BUT it’s not ok for “normal” file or block data which is updated frequently and then read back expecting to see the updates.
According to Ursheet, the cloud storage providers have been completely up-front about their consistency level and as such his product, StorSimple, has been specifically designed to accommodate variable levels of consistency. We would need to ask the other providers how they handle cloud storage consistency-ness to understand whether they have tried to deal with this as well.
However, from my perspective eventual consistency is scary. It appears that cloud storage has redefined what we mean by storage or at the very least eliminating data integrity. Moreover, this seriously limits the usability of raw cloud storage to very archive-like, infrequently updated data storage.
And I thought cloud storage was going to take over the data center – not like this…
Who says there are no clouds today by akakumo (cc) (from Flickr)
One problem holding back general purpose cloud storage has been the lack of a “standard” way to get data in and out of the cloud. Most cloud storage providers supply a REST interface, an object file interface or other proprietary ways to use their facilities. The problem with this is that they all require some form of code development on the part of the cloud storage customer in order to make use of these interfaces.
It would be much easier if cloud storage could just talk iSCSI, FCoE, FC, NFS, CIFS, FTP, etc. access protocols. Then any data center could use the cloud with a NIC/HBA/CNA and just configure the cloud storage as a bunch of LUNs or file systems/mount points/shares. Probably FCoE or FC might be difficult to use due to timeouts or other QoS (quality of service) issues but iSCSI and the file level protocols should be able to support cloud storage access without such concerns.
So which cloud storage support these protocols today? Nirvanix supports CloudNAS used to access their facilities via NFS, CIFS and FTP, ParaScale supports NFS and FTP, while Amazon S3 and Rackspace CloudFiles do not seem to support any of these interfaces. There are probably other general purpose cloud storage providers I am missing here but these will suffice for now. Wouldn’t it be better if some independent vendor supplied one way to talk to all of these storage environments.
How can gateways help?
For one example, Nasuni recently emerged from stealth mode, releasing a beta version of a cloud storage gateway that supports file access to a number of providers. Currently, Nasuni supports CIFS file protocol as a front end for Amazon S3, IronMountain ASP, Nirvanix, and coming soon Rackspace CloudFile.
However, Nasuni is more than just a file protocol converter for cloud storage. It also supplies a data cache, file snapshot services, data compression/encryption, and other cloud storage management tools. Specifically,
Cloud data cache – their gateway maintains a disk cache of frequently accessed data that can be accessed directly without having to go out to the cloud storage. File data is chunked by the gateway and flushed out of cache to the backend provider. How such a disk cache is maintained coherently across multiple gateway nodes was not discussed.
File snapshot services – their gateway supports a point-in-time copy of file date used for backup and other purposes. The snapshot is created on a time schedule and provides an incremental backup of cloud file data. Presumably these snapshot chunks are also stored in the cloud.
Data compression/encryption services – their gateway compresses file chunks and then encrypts it before sending them to the cloud. Encryption keys can optionally be maintained by the customer or be automatically maintained by the gateway
Cloud storage management services – the gateway configures the cloud storage services needed to define volumes, monitors cloud and network performance and provides a single bill for all cloud storage used by the customer.
By chunking the files and caching them, data read from the cloud should be accessible much faster than normal cloud file access. Also by providing a form of snapshot, cloud data should be easier to backup and subsequently restore. Although Nasuni’s website didn’t provide much information on the snapshot service, such capabilities have been around for a long time and found very useful in other storage systems.
Nasuni is provided as a software only solution. Once installed and activated on your server hardware, it’s billed for as a service and ultimately is charged for on top of any cloud storage you use. You sign up for supported cloud storage providers through Nasuni’s service portal.
How well all this works is open for discussion. We have discussed caching appliances before both from EMC and others. Two issues have emerged from our discussions, how well caching coherence is maintained across nodes is non-trivial and the economics of a caching appliance are subject to some debate. However, cloud gateways are more than just caching appliances and as a way of advancing cloud storage adoption, such gateways can only help.
Full disclosure: I currently do no business with Nasuni.
The Night Lights of Europe (as seen from space) by woodleywonderworks (cc) (from flickr)
Yesterday, twitterland was buzzing about EMC’s latest enhancement to their Atmos Cloud Storage platform called GeoProtect. This new capability improves cloud data protection by supporting erasure code data protection rather than just pure object replication.
Erasure coding has been used for over a decade in storage and some of the common algorithms are Reed-Solomon, Cauchy Reed-Soloman, EVENODD coding, etc. All these algorithms provide a way for splitting up customer data into data instances and parity (encoding) to allow some number of data or parity instances to be erased (or lost) while still providing customer data. For example, a R-S encoding scheme we used in the past (called RAID 6+) had 13 data fragments and 2 parity fragments. Such an encoding scheme supported the simultaneous failure of any two drives and could still supply (reconstruct) customer data.
But how does RAID differ from something like GeoProtect.
RAID is typically within a storage array and not across storage arrays
RAID is typically limited to a small number of alternative configurations of data disks and parity disks which cannot be altered in the field, and
Currently, RAID typically doesn’t support more than two disk failures while still being able to recover customer data (see Are RAIDs days numbered?)
As I understand it GeoProtect currently supports only two different encoding schemes which can provide for different levels of data instance failures while still protecting customer data. And with GeoProtect you are protecting data across Atmos nodes and potentially across different geographic locations not just within storage arrays. Also, with Atmos this is all policy driven and data that comes into the system can use any object replication policy or either of the two GeoProtect policies supported today.
Although the nice thing about R-S encoding is that it doesn’t have to be fixed to two different encoding schemes. And as it’s all software, new coding schemes could easily be released over time, possibly someday being entirely something a user could dial up or down at their whim.
But this would seem much more like what Cleversafe has been offering in their SliceStor product. With Cleversafe the user can specify exactly how much redundancy they want to support and the system takes care of everything else. In addition, Cleversafe has implemented a more fine grained approach (with many more fragments) and data and parity are intermingled in each stored fragment.
It’s not a big stretch for Atmos to go from two GeoProtect configurations to four or more. Unclear to me what the right number would be but once you get past 3 or so, it might be easier to just code a generic R-S routine that can handle any configuration the customer wants but I may be oversimplifying the mathematics here.
Nonetheless, in future versions of Atmos I wouldn’t be surprised if it’s possible that through policy management the way data is protected could change over time. Specifically, while data is being frequently accessed, one could use object replication or less compressed encoding to speed up access but once access frequency diminishes (or time passes), data can then protected with more storage efficient encoding schemes which would reduce the data footprint in the cloud while still offering similar resiliency to data loss.
Full disclosure I have worked for Cleversafe in the past and although I am currently working with EMC, I have had no work from EMC’s Atmos team.
What would happen if somebody came up with a peer-to-peer cloud (P2P-Cloud) storage or computing service. I see this as
Operating a little like Napster/Gnutella where many people come together and share out their storage/computing resources.
It could operate in a centralized or decentralized fashion
It would allow access to data/computing resources anywhere from the internet
Everyone joining the P2P-cloud would need to set aside computing and/or storage resources they were willing to devote to the cloud. By doing so, they would gain access to an equivalent amount (minus overhead) of other nodes computing and storage resources to use as they see fit.
P2P-Cloud Storage
For cloud storage the P2P-Cloud would create a common cloud data repository spread across all nodes in the network:
Data would be distributed across the network in such a way that would allow reconstruction within any reasonable time frame and would handle any reasonable amount of node outages without loss of data.
Data would be encrypted before being sent to the cloud rendering the data unreadable without the key.
Data would NOT necessarily be shared, but would be hosted on other users systems.
As such, if I were to offer up 100GB of storage to the P2P-Cloud, I would get at least a 100GB (less overhead) of protected storage elsewhere on the cloud to use as I see fit. Some % of this would be lost to administration say 1-3% and redundancy protection say ~25% but the remaining 72GB of off-site storage could be very useful for DR purposes.
P2P-Cloud storage would provide a reliable, secure, distributed file repository that could be easily accessible from any internet location. At a minimum, the service would be free and equivalent to what someone supplies (less overhead) to the P2P-Cloud Storage service. If storage needs exceeded your commitment, more cloud storage could be provided at a modest cost to the consumer. Such fees would be shared by all the participants offering excess [=offered – (consumed + overhead)] storage to the cloud .
P2P-Cloud Computing
Cloud computing is definitely more complex, but generally follows the Seti@HOME/BOINC model:
P2P-Cloud computing suppliers would agree to use something like a “new screensaver” which would perform computation while generating a viable screensaver.
Whenever the screensaver was invoked, it would start execution on the last assigned processing unit. Intermediate work results would need to be saved and when completed, the answer could be sent to the requester and a new processing unit assigned.
Processing units would be assigned by the P2P-Cloud computing consumer, would be timeout-able and re-assignable at will.
Computing users won’t gain much if the computing time they consume is <= the computing time they offer (less overhead). However, computing time offset may be worth something, i.e., computing time now might be more valuable than computing time tonite. Which may offer a slight margin of value to help get this off the ground. As such, P2P-Cloud computing suppliers would need to be able to specify when computing resources might be mostly available along with the type, quality and quantity.
Unclear how to secure the processing unit and this makes legal issues more prevalent. That may not be much of a problem, as a complex distributed computing task makes little sense in isolation. But the (il-)legality of some data processing activities could conceivably put the provider in a precarious position. (Somebody from the legal profession would need clarify all this, but I would think that some “Amazon C2” like licensing might offer safe harbor here).
P2P-Cloud computing services wouldn’t necessarily be amenable to the more normal, non-distributed or linear computing tasks but one could view these as just a primitive version of distributed computing tasks. In either case, any data needed for computation would need to be sent along with the computing software to be run on a distributed node. Whether it’s worth the effort is something for the users to debate.
BOINC can provide a useful model here. Also, the Condor(R) project at U. of Wisconsin/Madison can provide a similar framework for scheduling the work of a “less distributed” computing task model. In my mind, both types of services ultimately need to be provided.
To generate more compute servers, the SETI@Home and similar BOINC projects rely on doing good deeds. As such, if you can make your computing task do something of value to most users then maybe that’s enough. In that case, I would suggest joining up as a BOINC project. For the rest of us, doing more mundane data processing, just offering our compute services to the P2P-Cloud will need to suffice.
Starting up the P2P-Cloud
Bootstrapping the P2P-Cloud might take some effort but once going it should be self sustaining (assuming no centralized infrastructure). I envision an open source solution, taking off from the work done on Napster&Gnutella and/or Boinc&Condor.
I believe the P2P-Cloud Storage service would be the easiest to get started. BOINC and SETI@home (list of active Boinc projects) have been around a lot longer than cloud storage but their existence suggests that with the right incentives, even the P2P-Cloud Computing service can make sense.
Facebook friend carrousel by antjeverena (cc) (from flickr)
Cloud storage has emerged as a viable business service in the last couple of years, but what does cloud storage really do for the data center. Moving data out to the cloud makes for unpredictable access times with potentially unsecured and unprotected data. So what does the data center gain by using cloud storage?
Speed – it often takes a long time (day-weeks-months) to add storage to in-house data center infrastructure. In this case, having a cloud storage provider where one can buy additional storage by the GB/Month may make sense if one is developing/deploying new applications where speed to market is important.
Flexibility – data center storage is often leased or owned for long time periods. If an application’s data storage requirements vary significantly over time then cloud storage, purchase-able or retire-able on a moments notice, may be just right.
Distributed data access – some applications require data to be accessible around the world. Most cloud providers have multiple data centers throughout the world that can be used to host one’s data. Such multi-site data centers can be often be accessed much quicker than going back to a central data center.
Data archive – backing up data that is infrequently accessed wastes time and resources. As such, this data could easily reside in the cloud with little trouble. References to such data would need to be redirected to one’s cloud provider but that’s about all that needs to be done.
Disaster recovery – disaster recovery for many data centers is very low on their priority list. Cloud storage provides an easy, ready made solution to accessing one’s data outside the data center. If you elect to copy all mission critical data out to the cloud on a periodic basis, then this data could theoretically be accessed anywhere, usable in many DR scenarios.
Probably some I am missing here but these will do for now. Most cloud storage providers can provide any and all of these services.
Of course all these capabilities can be done in-house with additional onsite infrastructure, multi-site data centers, archive systems, or offsite backups. But the question then becomes which is more economical. Cloud providers can amortize their multi-site data centers across many customers and as such, may be able to provide these services much cheaper than could be done in-house.
Now if they could only solve that unpredictable access time, …
safe 'n green by Robert S. Donovan (cc) (from flickr)
A recent article from MIT’s Technology Review discussed cloud security (“Security in the Ether”). Most of the article was on how many cloud servers are vulnerable to a particular hack that can uncover private data in server memory/cache. But a good portion of the article was on how to secure data in the cloud and the article discussed a couple of new ideas (to me at least):
Securing cloud data access by using a key hierarchy – in this way a particular file/table/row could have a hierarchy of keys and thus, could have one master key for the whole datum and subset keys which would provide access to segments of the datum. As such, the patient could hold the master key to their electronic health records while their physicians held subset keys that would allow them to access diagnostic results and other information needed to treat the patient.
Securing cloud data search by encrypting meta-data – in this way a search key could be encrypted and then the search could execute in the cloud against the encrypted meta-data. As such, meta data and search keys would need to be encrypted in a static fashion so that they would always encrypt to the same cipher text but this could be done with an MD5 hash. Not sure how this might help sorting but it’s certainly a step in the right direction as searches could be performed completely secure while using cloud resources. Subsequent search results could then be easily delivered back to the end user for decryption and use.
Securing cloud data manipulation by using “ideal lattice” calculations on encrypted data – in this way mathematical manipulations of encrypted data are possible and can be extracted from the cloud for decryption and use. As such, data queries using arithmetic functions such as summing a column of cloud data, can be completely secured and the resultant summation delivered outside the cloud. How this works is beyond me and the mathematics are said to be a bit cumbersome but, it’s still early and may someday become a viable approach.
It seems to me most of this goes way beyond the data archive I would envision for the cloud. With such encryption techniques one could conceivably host one’s data center applications in the cloud and/or use the cloud to serve as data storage for all applications. While this may be the ultimate goal for the cloud it still seems a way off.
So what mathematical functions can be accomplished using an “ideal lattice”?
Who says there are no clouds today by akakumo (cc) (from Flickr)
We have posted previously about the need for backup in cloud storage. So today I would like to start a discussion on securing “data-at-rest” within the cloud.
Depositing data into the cloud seems a little like a chinese laundry to me – you deposit data in the cloud and receive a ticket or token used to retrieve the data. Today’s cloud data security depends entirely on this token.
Threats to the token
If one only looks at external security threats, two issues to token use seem apparent
Brute force cracking of any token is possible. I envision a set of cloud storage users wherein they use their current storage tokens as seeds to identify other alternate tokens. Such an attack could easily generate token synonyms which may or may not be valid. Detecting a brute force attack could be easily accomplished, but distributing this attack across 1000s of compromised PCs would be much harder to detect.
Tokens could be intercepted in the clear. Cloud data often may need to be accessed in locations outside the data center of origin. This would require sending tokens to others. These data tokens could inadvertently be sent in the clear and as such, intercepted.
Probably other external exposures beyond these two exist as well but these will suffice.
Securing cloud data-at-rest
Given the potential external and internal threats to data tokens, securing such data can eliminate any data loss from token exposure. I see at least three approaches to securing data in the cloud.
The data dispersal approach – Cleversafe’s product splits a data stream into byte segments and disburses these segments across storage locations. Their approach is oriented around Reed Solomon logic and results in no one location having any recognizable portion of the data. This requires multiple sites, or multiple systems at one site to implement but is essentially securing data through segmenting it. The advantages of this approach is that its fast and automatic but the disadvantage is that it only is supported via Cleversafe.
The software data encryption approach – there are plenty of software packages out such as GnuPG (GNU Privacy Guard) or PGP which can be used to encrypt your data prior to sending it to the cloud. It’s sort of brute force, software only approach but its advantage is that it can be used with any cloud storage provider. Its disadvantages are that it’s slow, processor intensive, and key management is sporadic.
The hardware data encryption approach – there are also plenty of data encryption appliances and/or hardware options out there which can be used to encrypt data. Some of these are available at the FC switch level, some are standalone appliances, and some exist at the storage subsystem level. The problem with most of these is that they only apply to FC storage and are not readily useable by Cloud Storage (unless the provider uses FC storage as its backing store). The advantages are that it’s fast and key management is generally built into the product.
One disadvantage to any of the encryption approaches is that now one needs the encryption keys and the token to access the data. Yet one more thing to protect.
Nothing says that hardware data encryption couldn’t also work for data flowing to the cloud but they would have to support IP plus the cloud specific REST interface. Such support would depend on cloud storage provider market share, but perhaps some cloud vendor could fund a security appliance vendor to support their interface directly, providing a cloud data security option.
The software approach suffers from performance problems but supports anybody’s cloud storage. It might be useful if Cloud storage providers started offering hooks into GnuPG or PGP to directly encrypt cloud data. However, most REST interfaces require some programming to use and it’s not too much of a stretch to program in encryption into this.
I like the data dispersal approach, but most argue that security is not guaranteed as reverse engineering the dispersal algorithm allows one to reconstruct the data stream. But the other more serious problem is that it only applies to Cleversafe storage, perhaps the dispersal algorithm should be open sourced (which it already is) and/or standardized.
There are possibly other approaches which I have missed here but these can easily be used to secure cloud data-at-rest. Possibly adding more security around the data token could also help alleviate this concern. Thoughts?
Post disclosure: I am not currently working with Cleversafe, any data security appliance provider, or cloud storage provider.
Cloud whisps (sic) by turtlemom4bacon (cc) (from flickr)
When I first heard about cloud storage I wondered just what exactly it was trying to solve. There are many storage problems within the IT shop nowadays days, cloud storage can solve a few of them but introduces more and leaves a few unsolved.
Storage problems solved by cloud storage
Dynamic capacity – storage capacity is fixed once purchased/leased. Cloud storage provides an almost infinite amount of storage for your data. One pays for this storage, in GB or TB per month increments, with added storage services (multi-site replication, high availability, etc.) at extra charge. Such capacity can be reduced or expanded at a moments notice.
Offsite DR – disaster recovery for many small shops is often non-existent or rudimentary at best. Using cloud storage, data can be copied to the cloud and accessed anywhere via the internet. Such data copies can easily support rudimentary DR for a primary data center outage.
Access anywhere – storage is typically local to the IT shop and can normally only be accessed at that location. Cloud storage can be accessed from any internet access point. Applications that are designed to operate all over the world can easily take advantage of such storage.
Data replication – data should be replicated for high availability. Cloud storage providers can replicate your data to multiple sites so that if one site goes down other sites can still provide service.
Storage problems introduced by the cloud
Variable access times – local storage access times vary from 1 and 100 milleseconds. However, accessing cloud storage can take from 100’s of milleseconds to minutes depending on network connectivity. Many applications cannot endure such variable access times.
Different access protocols – local storage support fairly standard access protocols like FC, iSCSI, NFS, and/or CIFS/SMB. Barring the few (but lately increasing) cloud providers that provide NFS access protocol, most cloud storage requires rewriting applications to use new protocols such as REST to store and access cloud file data.
Governance over data – local storage is by definition all located inside one data center. Many countries do not allow personal and/or financial data to be stored outside the country of origin. Some cloud storage providers will not guarantee that data stored in the cloud couldn’t be stored outside the country and jurisdiction of a single country.
Storage problems not solved by the cloud:
Data backups – data protection via some form of backup is essential. Nothing says that cloud storage providers cannot provide backup of data in the cloud but few if any provide such service. See my Are backups needed in the cloud post.
Data security – data security remains an ongoing problem for the local data center moving the data to the cloud just makes security more difficult. Many cloud storage providers provide rudimentary security for data stored but none seem to have integrated strong authentication and encryption services that might provide true data security.
Energy consumption – today’s storage consumes power and cooling. Although, cloud storage can be more efficient than onsite storage, this does not eliminate the environmental cost of storage.
Data longevity – data stored in the cloud can just as easily go obsolete as data stored locally.
Probably some I have missed here but these are a good start.
