Category: storage economics

What eMLC and eSLC do for SSD longevity

Posted on by Ray in storage economics, Storage longevity, System quality
Enterprise NAND from Micron.com (c) 2010 Micron Technology, Inc.
Enterprise NAND from Micron.com (c) 2010 Micron Technology, Inc.

I talked last week with some folks from Nimbus Data who were discussing their new storage subsystem.  Apparently it uses eMLC (enterprise Multi-Level Cell) NAND SSDs for its storage and has no SLC (Single Level Cell) NAND at all.

Nimbus believes with eMLC they can keep the price/GB down and still supply the reliability required for data center storage applications.  I had never heard of eMLC before but later that week I was scheduled to meet with Texas Memory Systems and Micron Technologies that helped get me up to speed on this new technology.

eMLC/eSLC defined

eMLC and its cousin, eSLC are high durability NAND parts which supply more erase/program cycles than generally available from MLC and SLC respectively.  If today’s NAND technology can supply 10K erase/program cycles for MLC and similarly, 100K erase/program cycles for SLC then, eMLC can supply 30K.  Never heard a quote for eSLC but 300K erase/program cycles before failure might be a good working assumption.

The problem is that NAND wears out, and can only sustain so many erase/program cycles before it fails.  By having more durable parts, one can either take the same technology parts (from MLC to eMLC) to use them longer or move to cheaper parts (from SLC to eMLC) to use them in new applications.

This is what Nimbus Data has done with eMLC.  Most data center class SSD or cache NAND storage these days are based on SLC. But SLC, with only on bit per cell, is very expensive storage.  MLC has two (or three) bits per cell and can easily halve the cost of SLC NAND storage.

Moreover, the consumer market which currently drives NAND manufacturing depends on MLC technology for cameras, video recorders, USB sticks, etc.  As such, MLC volumes are significantly higher than SLC and hence, the cost of manufacturing MLC parts is considerably cheaper.

But the historic problem with MLC NAND is the reduction in durability.  eMLC addresses that problem by lengthening the page programming (tProg) cycle which creates a better, more lasting data write, but slows write performance.

The fact that NAND technology already has ~5X faster random write performance than rotating media (hard disk drives) makes this slightly slower write rate less of an issue. If eMLC took this to only ~2.5X disk writes it still would be significantly faster.  Also, there are a number of architectural techniques that can be used to speed up drive write speeds easily incorporated into any eMLC SSD.

How long will SLC be around?

The industry view is that SLC will go away eventually and be replaced with some form of MLC technology because the consumer market uses MLC and drives NAND manufacturing.  The volumes for SLC technology will just be too low to entice manufacturers to support it, driving the price up and volumes even lower – creating a vicious cycle which kills off SLC technology.  Not sure how much I believe this, but that’s conventional wisdom.

The problem with this prognosis is that by all accounts the next generation MLC will be even less durable than today’s generation (not sure I understand why but as feature geometry shrinks, they don’t hold charge as well).  So if today’s generation (25nm) MLC supports 10K erase/program cycles, most assume the next generation (~18nm) will only support 3K erase/program cycles. If eMLC then can still support 30K or even 10K erase/program cycles that will be a significant differentiator.

—-

Technology marches on.  Something will replace hard disk drives over the next quarter century or so and that something is bound to be based on transistorized logic of some kind, not the magnetized media used in disks today. Given todays technology trends, it’s unlikely that this will continue to be NAND but something else will most certainly crop up – stay tuned.

Anything I missed in this analysis?

Cirtas surfaces

Posted on by Ray in Block Storage, storage economics, Storage performance
Cirtas system (from www.Cirtas.com)
Cirtas system (from http://www.Cirtas.com)

Yesterday, Cirtas came out of stealth mode and into the lime-light with their new Bluejet cloud storage controller hardware system.  Cirtas joins a number of other products offering cloud storage to the enterprise by supplying a more standard interface which we have discussed before (see Cloud storage gateways surface).

With Cirtas, the interface to the backend cloud storage is supplied as iSCSI, similar to StorSimple‘s product we reviewed previously (see More cloud storage gateways …).  However, StorSimple is focused on Microsoft environments only and select applications, namely Sharepoint, Exchange and Microsoft file services.  Cirtas seems aimed at the more general purpose application environment that uses iSCSI storage protocols.  The only other iSCSI cloud storage gateway providers appear to be TwinStrata and Panzura but the information on Panzura’s website is sketchy.

In addition, Cirtas, StorSimple (and Panzura) provide hardware appliances whereas most of the other cloud storage gateways (Nasuni, Gladinet, TwinStrata) only come as software  packages.  Although Gladinet appears to be targeted at the home office environment.

Cirtas’s Bluejet controller includes onboard RAM cache, SSD flash drives and SAS drives (5TB total) that is used to provide higher performing cloud storage access.  Bluejet also supports space efficient snapshots, data encryption, thin provisioning, data deduplication, and data compression. The Cirtas team comes out of the WAN optimization space so they have incorporated some of these data saving technologies into their product to reduce bandwidth requirement and cloud storage demand.

Cirtas currently supports Amazon S3 and IronMountain cloud storage but more are on the way.  They also recently completed their Series A round of funding which included NEA and Amazon.

Cirtas says they can support local storage performance but have no benchmarks to prove this out.  With iSCSI there aren’t many benchmark options but one could use iSCSI to support Microsoft Exchange and submit something on the Exchange Solution Review Program (ESRP) which might show off this capability.

Nonetheless, cloud storage can be considerably cheaper than primary storage ($/GB basis) and no doubt even with the ~$70K Cirtas Bluejet cloud storage controller, Cirtas supports a significant cost advantage.   With the appliance purchase, you get a basic storage key which allows you to store up to 20TB of data on (through) the appliance, if you have more data to store, additional storage keys can be purchased separately.  This 20TB license does not include the cloud storage costs for storing data on the cloud nor the bandwidth costs to upload and/or access the data on the cloud.

Seems like interest in cloud storage gateways/controllers is heating up, with the addition of Cirtas I count at least 4 that target the enterprise space and when Panzura releases a product that will add another.

Anything I missed?

Why cloud, why now?

Posted on by Ray in Business economics, Cloud services, Market dynamics, storage economics, Strategy, Systems
Moore’s Law by Marcin Wichary (cc) (from Flickr)
Moore’s Law by Marcin Wichary (cc) (from Flickr)

I have been struggling for sometime now to understand why cloud computing and cloud storage have suddenly become so popular.  We have previously discussed some of cloud problems (here and here) but we have never touched on why cloud has become so popular.

In my view, SaaS or ASPs and MSPs have been around for a decade or more now and have been renamed cloud computing and storage but they have rapidly taken over the IT discussion.  Why now?

At first I thought this new popularity was due to the prevalence of higher bandwidth today. But later I determined that this was too simplistic.  Now I would say the reasons cloud services have become so popular, include

  • Bandwidth costs have decreased substantially
  • Hardware costs have decreased substantially
  • Software costs remain flat

Given the above one would think that non-cloud computing/storage would also be more popular today and you would be right.  But, there is something about the pricing reduction available from cloud services which substantially increases interest.

For example, at $10,000 per widget, a market size may be ok, at $100/widget the market becomes larger still, and at $1/widget the market can be huge.  This is what seems to have happened to Cloud services.  Pricing has gradually decreased, brought about through hardware and bandwidth cost reductions and has finally reached a point where the market has grown significantly.

Take email for example:

Now with Google or Exchange Online you have to supply internet access or the bandwidth required to access the email account.  For Exchange, you would also need to provide the internet access to get email in and out of your environment, servers and storage to run Exchange server, and would use internal LAN resources to distribute that email to internally attached clients.  I would venture to say the similar pricing differences applies to CRM, ERP, storage etc. which could be hosted in your data center or used as a cloud service.  Also, over the last decade these prices have been coming down for cloud services but have remained (relatively) flat for on premises services.

How does such pricing affect market size?

Well, when it costs ~$1034 (+ server costs + admin time) to field 5 Exchange email accounts vs.  $250 for 5 Gmail ($300 for 5 Exchange Online) accounts the assumption is that the market will increase, maybe not ~12X but certainly 3X or more.  At ~$3000 or more, I need a substantially larger justification to introduce enterprise email services but at $250,  justification becomes much simpler.

Moreover, the fact that the entry pricing is substantially smaller, i.e.,  $~2800 for one Exchange Standard Edition account vs $50 for one (Gmail) email account, justification becomes almost a non-issue and the market size grows geometrically.  In the past, pricing for such services may have prohibited small business use, but today cloud pricing makes them very affordable and as such, more widely adopted.

I suppose there is another inflection point at  $0.50/mail user that would increase market size even more.  However, at some point anybody in the world with internet access could afford enterprise email services and I don’t think the market could grow much larger.

So there you have it.  Why cloud, why now – the reasons are hardware and bandwidth pricing have come down giving rise to much more affordable cloud services opening up more market participants at the low end.  But it’s not just SMB customers that can now take advantage of these lower priced services, large companies can also now afford to implement applications which were too costly to introduce before.

Yes, cloud services can be slow and yes, cloud services can be insecure but, the price can’t be beat.

As to why software pricing has remained flat must remain a mystery for now but may be treated in some future post.

Any other thoughts as to why cloud’s popularity has increased so much?

Primary storage compression can work

Posted on by Ray in Block Storage, storage economics, Storage Features
Dans la nuit des images (Grand Palais) by dalbera (cc) (from flickr)
Dans la nuit des images (Grand Palais) by dalbera (cc) (from flickr)

Since IBM’s announced their intent to purchase StorWize there has been much discussion on whether primary storage data compression can be made to work.  As far as I know StorWize only offered primary storage compression for file data but there is nothing that prohibits doing something similar for block storage as long as you have some control over how blocks are laid down on disk.

Although secondary block  data compression has been around for years in enterprise tape and more recently with some deduplication appliances, primary storage compression pre-dates secondary storage compression.  STK delivered primary storage data compression with Iceberg in the early 90’s but it wasn’t until a couple of years later that they introduced compression on tape.

In both primary and secondary storage, data compression works to reduce the space needed to store data.  Of course, not all data compresses well, most notably image data (as it’s already compressed) but compression ratios of 2:1 were common for primary storage of that time and are normal for today’s secondary storage.  I see no reason why such ratios couldn’t be achieved for current primary storage block data.

Implementing primary block storage data compression

There is significant interest in implementing deduplication for primary storage as NetApp has done but supporting data compression is not much harder.  I believe much of the effort to deduplicate primary storage lies in creating a method to address partial blocks out of order, which I would call data block virtual addressing which requires some sort of storage pool.  The remaining effort to deduplicate data involves implementing the chosen (dedupe) algorithm, indexing/hashing, and other administrative activities.  These later activities aren’t readily transferable to data compression but the virtual addressing and space pooling should be usable by data compression.

Furthermore, block storage thin provisioning requires some sort of virtual addressing as does automated storage tiering.  So in my view, once you have implemented some of these advanced capabilities, implementing data compression is not that big a deal.

The one question that remains is does one implement compression with hardware or software (see Better storage through hardware for more). Considering that most deduplication is done via software today it seems that data compression in software should be doable.  The compression phase could run in the background sometime after the data has been stored.  Real time decompression using software might take some work, but would cost considerably less than any hardware solution.  Although the intensive bit fiddling required to perform data compression/decompression may argue for some sort of hardware assist.

Data compression complements deduplication

The problem with deduplication is that it needs duplicate data.  This is why it works so well for secondary storage (backing up the same data over and over) and for VDI/VMware primary storage (with duplicated O/S data).

But data compression is an orthogonal or complementary technique which uses the inherent redundancy in information to reduce storage requirements.  For instance, something like LZ compression takes advantage of the fact that in text some letters occur more often than others (see letter frequency). For instance, in English, ‘e’, ‘t’, ‘a’, ‘o’, ‘i’, and ‘n, represent over 50% of the characters in most text documents.  By using shorter bit combinations to encode these letters one can reduce the bit-length of any (English) text string substantially.  Another example is run length encoding which takes any repeated character and substitutes a trigger character, the character itself, and a count of the number of repetitions for the repeated string.

Moreover, the nice thing about data compression is that all these techniques can be readily combined to generate even better compression rates.  And of course compression could be applied after deduplication to reduce storage footprint even more.

Why would any vendor compress data?

For a couple of reasons:

  • Compression not only reduces storage footprint but with hardware assist it can also increase storage throughput. For example, if 10GB of data compresses down to 5GB, it should take ~1/2 the time to read.
  • Compression reduces the time it would take time to clone, mirror or replicate.
  • Compression increases the amount of data that could be stored which should incentivise them to pay more for your storage.

In contrast, with data compression vendors might may sell less storage.  But the advantages of enterprise storage is in the advanced functionality/features and higher reliability/availability/performance that are available.  I see data compression as just another advantages to enterprise class storage and as a feature, the user could enable or disable it and see how well it works for there data.

What do you think?

What’s wrong with tape?

Posted on by Ray in Storage, storage economics, Tape storage
StorageTek Automated Cartridge System by brewbooks (cc) (from Flickr)
StorageTek Automated Cartridge System by brewbooks (cc) (from Flickr)

Was on a conference call today with Oracle’s marketing discussing their tape business.  Fred Moore (from Horison Information Systems) was on the call and mentioned something which surprised me.  What’s missing in open and distributed systems was some standalone mechanism to stack volumes onto a single tape cartridge.

The advantages of tape are significant, namely:

  • Low power utilization for offline or nearline storage
  • Cheap media, drives, and automation systems
  • Good sequential throughput
  • Good cartridge density

But most of these advantages fade when cartridge capacity utilization drops.  One way to increase cartridge capacity utilization is to stack multiple tape volumes on a single cartridge.

Mainframes (like system/z) have had cartridge stacking since the late 90’s.  Such capabilities came about due to the increasing cartridge capacities then available. Advance a decade and the problem still exists, Oracle’s StorageTek T10000 has a 1TB cartridge capacity and LTO-5 supports 1.5TB per cartridge both uncompressed.  Nonetheless, open or distributed systems still have no tape stacking capability.

Although I agree with Fred that volume stacking is missing in open systems, but does it really need such a thing.  Currently it seems open systems uses tape for backups, archive data and the occasional batch run.  Automated hierarchical storage management can readily fill up tape cartridges by holding their data movement to tape until enough data is ready to be moved.  On the other hand, backups by their very nature create large sequential streams of data which should result in high capacity utilization except for the last tape in a series.  Which only leaves the problem of occasional batch runs using large datasets or files.

I believe most batch processing today already takes place on the mainframe, leaving relatively little for open or distributed systems.  There are certainly some verticals that do lots of batch processing, for example banks and telcos.  But most heavy batch users grew up in the heyday of the mainframe and are still using them today.

Condor notwithstanding, open and distributed systems never had any sophisticated batch processing capabilities readily available on the mainframe. As such, of those new companies that need batch processing, my guess is that they start with open and as their needs for batch grow move these applications to mainframe.

So the real question becomes how do we increase open systems batch processing.   I don’t think a tape volume stacking system solves that problem.

Given all the above, I see tape use in open being relegated to backup and archive and used less and less for any other activities.

What do you think?

5 killer apps for $0.10/TB/year

Posted on by Ray in Data, Data growth, Data index, Data search, storage economics
iblioteca José Vasconcelos / Vasconcelos Library by * CliNKer * (from flickr) (cc)
iblioteca José Vasconcelos / Vasconcelos Library by * CliNKer * (from flickr) (cc)

Cloud storage keeps getting more viable and I see storage pricing going down considerably over time.  All of which got me thinking what could be done with a dime per TB per year storage ($0.10/TB/yr).  Now most cloud providers charge 10 cents or more per GB per month so this is at least 12,000 times less expensive but it’s inevitable at some point in time.

So here are my 5 killer apps for $0.10/TB/yr cloud storage:

  1. Photo record of life – something akin to glasses which would record a wide angle, high mega-pixel video record of everything I looked at, for every second of my waking life.  I think at a photo shot every second for 12hrs/day 365days/yr would be about ~16M photos and at 4 MB per photo this would be about ~64TB per person year.  For my 4 person family this would cost ~$26/year for each year of family life and for a 40 year family time span, the last payment for this would be ~$1040 or an average payment of $520/year.
  2. Audio recording of life – something akin to a always on bluetooth headset which would record an audio feed to go with the semi-video or photo record above.  By being an always on bluetooth headset it would automatically catch cell phone as well as  spoken conversations but it would need to plug to landlines as well.  As discussed in my YB by 2015 archive post, one minute of MP3 audio recording takes up roughly a MB of storage.  Lets say I converse with someone ~33% of my waking day.  So this would be about 4 hrs of MP3 audio/day 365days/yr or about 21TB per year per person.  For my family this would cost or ~$8.40/year for storage and for a 40 year family life span my last payment would be ~$336 or an average of $168/yr.
  3. Home security cameras – with ethernet based security cameras, it wouldn’t be hard to record a 360 degree outside as well as inside points of entry coverage video.  The quantities for the photo record of my life would suffice for here as well but one doesn’t need to retain the data for a whole year perhaps a rolling 30 day record would suffice but it would be recorded for 24 hours. Assuming 8 cameras outside and inside,  this could be stored in about 10TB of storage per camera, or  about 80TB of storage or $8/year but would not increase over time.
  4. No more deletes/version everything – if storage were cheap enough we would never delete data.  Normal data change activity is in the 5 to 10% per week rate, but this does not account for duplicating deleted data.  So let’s say we would need to store an additional 20% of your primary/active data per week for deleted data.  For a 1TB primary storage working set, a ~20% deletion rate per week would be 10TB of deleted data per year per person and for my family ~$4/yr and my last yearly payment would be ~$160.  If we were to factor in data growth rates of ~20%/year, this would go up substantially averaging ~$7.3k/yr over 40 years.
  5. Customized search engines – if storage AND bandwidth were cheap enough it would be nice to have my own customized search engine. Such a capability would follow all my web clicks, spawning a search spider for every website I traverse and provide customized “deep” searching for every web page I view.   Such an index might take 50% of the size of a page and on average my old website used ~18KB per page, so at 50% this index would require 9KB. Assuming, I look at ~250 web pages per business day of which maybe ~170 are unique and each unique page probably links to 2 more unique pages, which links to two more, which links to two more, … If we go 10 pages deep, then for 170 pages viewed, an average branching factor of 2,  we would need to index ~174K pages/day and for a year, this would represent about represent about 0.6TB of page index.  For my household, a customized search engine would cost  ~$0.25 of additional storage per year and for 40 years my last payment would be $10.

I struggled with coming with ideas that would cost between $10 and $500 a year as every other storage use came out significantly less than $1/year for a family of four.  This seems to say that there might be plenty of applications in the range of under a $10 per TB per year, still 1200X current cloud storage costs.

Any other applications out there that could take  advantage of a dime/TB/year?

Describing Dedupe

Posted on by Ray in Data, Data efficiency, storage economics, Storage Features, Storage performance, Systems
Hard Disk 4 by Alpha six (cc) (from flickr)
Hard Disk 4 by Alpha six (cc) (from flickr)

Deduplication is a mechanism to reduce the amount of data stored on disk for backup, archive or even primary storage.  For any storage, data is often duplicated and any system that eliminates storing duplicate data will be more utilize storage more efficiently.

Essentially, deduplication systems identify duplicate data and only store one copy of such data.  It uses pointers to incorporate the duplicate data at the right point in the data stream. Such services can be provided at the source, at the target, or even at the storage subsystem/NAS system level.

The easiest way to understand deduplication is to view a data stream as a book and as such, it can consist of two parts a table of contents and actual chapters of text (or data).  The stream’s table of contents provides chapter titles but more importantly (to us), identifies a page number for the chapter.  A deduplicated data stream looks like a book where chapters can be duplicated within the same book or even across books, and the table of contents can point to any book’s chapter when duplicated. A deduplication service inputs the data stream, searches for duplicate chapters and deletes them, and updates the table of contents accordingly.

There’s more to this of course.  For example, chapters or duplicate data segments must be tagged with how often they are duplicated  so that such data is not lost when modified.  Also, one way to determine if data is duplicated is to take one or more hashes and compare this to other data hashes, but to work quickly, data hashes must be kept in a searchable index.

Types of deduplication

  • Source deduplication involves a repository, a client application, and an operation which copies client data to the repository.  Client software chunks the data, hashes the data chunks, and sends these hashes over to the repository.  On the receiving end, the repository determines which hashes are duplicates and then tells the client to send only the unique data.  The repository stores the unique data chunks and the data stream’s table of contents.
  • Target deduplication involves performing deduplication inline, in-parallel, or post-processing by chunking the data stream as it’s recieved, hashing the chunks, determining which chunks are unique, and storing only the unique data.  Inline refers to doing such processing while receiving data at the target system, before the data is stored on disk.  In-parallel refers to doing a portion of this processing while receiving data, i.e., portions of the data stream will be deduplicated while other portions are being received.  Post-processing refers to data that is completely staged to disk before being deduplicated later.
  • Storage subsystem/NAS system deduplication looks a lot like post-processing, target deduplication.  For NAS systems, deduplicaiot looks at a file of data after it is closed. For general storage subsystems the process looks at blocks of data after they are written.  Whether either system detects duplicate data below these levels is implementation dependent.

Deduplication overhead

Deduplication processes generate most overhead while deduplicating the data stream, essentially during or after the data is written, which is the reason that target deduplication has so many options, some optimize ingestion while others optimize storage use. There is very little additonal overhead for re-constituting (or un-deduplicating) the data for read back as retrieving the unique and/or duplicated data segments can be done quickly.  There may be some minor performance loss because of lack of  sequentiality but that only impacts data throughput and not that much.

Where dedupe makes sense

Deduplication was first implemented for backup data streams.  Because any backup that takes full backups on a monthly or even weekly basis will duplicate lot’s of data.  For example, if one takes a full backup of 100TBs every week and lets say new unique data created each week is ~15%, then at week 0, 100TB of data is stored both for the deduplicated and undeduplicated data versions; at week 1 it takes 115TB to store the deduplicated data but 200TB for the non-deduplicated data; at week 2 it takes ~132TB to store deduplicated data but 300TB for the non-deduplicated data, etc.  As each full backup completes it takes another 100TB of un-deduplicated storage but significantly less deduplicated storage.  After 8 full backups the un-deduplicated storage would require 8ooTB but only ~265TB for deduplicated storage.

Deduplication can also work for secondary or even primary storage.  Most IT shops with 1000’s of users, duplicate lot’s of data.  For example, interim files are sent from one employee to another for review, reports are sent out en-mass to teams, emails are blasted to all employees, etc.  Consequently, any storage (sub)system that can deduplicate data would more efficiently utilize backend storage.

Full disclosure, I have worked for many deduplication vendors in the past.

Free P2P-Cloud Storage and Computing Services?

Posted on by Ray in Distributed computing, Storage, storage economics, Systems
FFT_graph from Seti@home
FFT_graph from Seti@home

What would happen if somebody came up with a peer-to-peer cloud (P2P-Cloud) storage or computing service.  I see this as

  • Operating a little like Napster/Gnutella where many people come together and share out their storage/computing resources.
  • It could operate in a centralized or decentralized fashion
  • It  would allow access to data/computing resources anywhere from the internet

Everyone joining the P2P-cloud would need to set aside computing and/or storage resources they were willing to devote to the cloud.  By doing so, they would gain access to an equivalent amount (minus overhead) of other nodes computing and storage resources to use as they see fit.

P2P-Cloud Storage

For cloud storage the P2P-Cloud would create a common cloud data repository spread across all nodes in the network:

  • Data would be distributed across the network in such a way that would allow reconstruction within any reasonable time frame and would handle any reasonable amount of node outages without loss of data.
  • Data would be encrypted before being sent to the cloud rendering the data unreadable without the key.
  • Data would NOT necessarily be shared, but would be hosted on other users systems.

As such, if I were to offer up 100GB of storage to the P2P-Cloud, I would get at least a 100GB (less overhead) of protected storage elsewhere on the cloud to use as I see fit.  Some % of this would be lost to administration say 1-3% and redundancy protection say ~25% but the remaining 72GB of off-site storage could be very useful for DR purposes.

P2P-Cloud storage would provide a reliable, secure, distributed file repository that could be easily accessible from any internet location.  At a minimum, the service would be free and equivalent to what someone supplies (less overhead) to the P2P-Cloud Storage service.  If storage needs exceeded your commitment, more cloud storage could be provided at a modest cost to the consumer.  Such fees would be shared by all the participants offering excess [=offered – (consumed + overhead)] storage to the cloud .

P2P-Cloud Computing

Cloud computing is definitely more complex, but generally follows the Seti@HOME/BOINC model:

  • P2P-Cloud computing suppliers would agree to use something like a “new screensaver” which would perform computation while generating a viable screensaver.
  • Whenever the screensaver was invoked, it would start execution on the last assigned processing unit.  Intermediate work results would need to be saved and when completed, the answer could be sent to the requester and a new processing unit assigned.
  • Processing units would be assigned by the P2P-Cloud computing consumer, would be timeout-able and re-assignable at will.

Computing users won’t gain much if the computing time they consume is <= the computing time they offer (less overhead).  However, computing time offset may be worth something, i.e., computing time now might be more valuable than computing time tonite.  Which may offer a slight margin of value to help get this off the ground.  As such, P2P-Cloud computing suppliers would need to be able to specify when computing resources might be mostly available along with the type, quality and quantity.

Unclear how to secure the processing unit and this makes legal issues more prevalent.  That may not be much of a problem, as a complex distributed computing task makes little sense in isolation. But the (il-)legality of some data processing activities could conceivably put the provider in a precarious position. (Somebody from the legal profession would need clarify all this, but I would think that some “Amazon C2” like licensing might offer safe harbor here).

P2P-Cloud computing services wouldn’t necessarily be amenable to the more normal, non-distributed or linear computing tasks but one could view these as just a primitive version of distributed computing tasks.  In either case, any data needed for computation would need to be sent along with the computing software to be run on a distributed node.  Whether it’s worth the effort is something for the users to debate.

BOINC can provide a useful model here.  Also, the Condor(R) project at U. of Wisconsin/Madison can provide a similar framework for scheduling the work of a “less distributed” computing task model.  In my mind, both types of services ultimately need to be provided.

To generate more compute servers, the SETI@Home and similar BOINC projects rely on doing good deeds.  As such, if you can make your computing task  do something of value to most users then maybe that’s enough. In that case, I would suggest joining up as a BOINC project. For the rest of us, doing more mundane data processing, just offering our compute services to the P2P-Cloud will need to suffice.

Starting up the P2P-Cloud

Bootstrapping the P2P-Cloud might take some effort but once going it should be self sustaining (assuming no centralized infrastructure).  I envision an open source solution, taking off from the work done on Napster&Gnutella and/or Boinc&Condor.

I believe the P2P-Cloud Storage service would be the easiest to get started.  BOINC and SETI@home (list of active Boinc projects) have been around a lot longer than cloud storage but their existence suggests that with the right incentives, even the P2P-Cloud Computing service can make sense.