My Head in the Clouds

By Rich Wellner

Up until I read Ian Foster’s Cloud Computing post, I had paid little attention to what the term meant to people.  Personally, I had already chalked up the idea as a rebranding of Grid computing.  So I asked a number of friends what they thought the differences between the two were.  Of course many people not actively involved in the community are not familiar with either concept.   (I find the fact that computer professionals know what the latest buzz is around SaaS, and SOA but do not seem to consider how and where they might land these systems peculiar.)  In any event, here is a summary of the answers I received:

• Grid is characterized by more formalized computing arrangements between user and vendor whereas a Cloud is more for ad hoc resource utilization;
• The types of computation on a Grid are typically parallel in nature whereas Clouds are for more simple calculations;
• Grid was usurped by vendors to indicate that their services were distributed for better performance and reliability while Cloud had become the term for a generalized set of distributed resources;
• Clouds are ethereal – anybody who watches them as they cross the sky knows that…

While I found it rather interesting that while there was some overlap of technical perspectives, I did not get any answers that were identical.  I believe that the last description explains this situation nicely while also offering the most interesting take on the topic.  A Cloud is a nuanced term that invokes the idea of something beautiful which also evolves rapidly, contains a lot of power, and then is gone.  Meanwhile the grid, like the utilities it was conceived from, is known for its reliability, ability to tap into reserve power on a moments notice, as well as their accommodating levels of service.  Notice how the first two answers all adhere to this concept?  I don’t think this is an accident nor do I think that this escaped the attention of the marketing departments of industry-leaders like Amazon and Google, both of whom operate in what they term Cloud space.  While it is distinctly possible that the term organically evolved, it is interesting that they chose to stick with it.

Once more, I found it particularly noteworthy that not one person I queried mentioned the amount of data to be processed. Foster and the Business Week article he references, as well as many others, suggest that we need to think in terms of a great deal more data than we have before.  For example, Google wants their people to think in terms of a thousand times more data than that to which they are accustomed.

Heck, I was thinking about writing about the so-called “Data Tsunami” myself – but not in terms of thinking about significantly larger datasets. The datasets we were working with a decade ago were suitably massive for what we were trying to accomplish.  Like today, it was not economically feasible to keep it all online at once.  The fact is that the incredible leaps in computational capacity have led us to build more complicated problems that demand still more data.  As such, a thousand times more data is probably still not enough.  If only the networking and storage companies had kept up with the leaps in processing capacity (I was on a gigabit network five years ago and I am using a gigabit network today >sigh<).

Consequently we still have to use the tried and true standard operating procedure of:

• First carefully selecting a fraction of the data available for storage (i.e. triggering);
• Next this rough dataset is pruned further by pre-processing to find the most interesting records;
• Finally detailed analysis is performed on what is computationally feasible.

For example, the Large Hadron Collider will be examining billions of collisions per second but will only store a few hundred per second for later processing (see recent article on Dr. Heuer). We have always needed to think in terms of a thousand times more data than we can possibly process or to become accustomed.  Basically the scope of what is economically feasible has changed dramatically over the last few decades, while we continue to be quite resource constrained.  Which brings us back to the concept of capacity computing, whether in the form of a transitory Cloud, a steadfast Grid, or even the comfortable @home project.  The key here is that people are continuing to push passed the boundaries of what is feasible.