Virtual Grid Nodes: The Tension

By Roderick Flores

Lately I have been putting a lot of thought into the challenges that grid managers face in building an enterprise grid.  Primarily they must support the various stakeholders throughout the enterprise, each of whom has their own sets of application workflows used to meet their business needs. 

The software packages that each interested group uses may have a significant overlap with one another, but the similarity stops there.  Because each group ostensibly has a different goal, the usage patterns are almost guaranteed to be unique.  This implies that the community as a whole will demand any of the following:

• A wide range of operating systems including Linux, Microsoft Windows, or any of the varied flavors of Unix;
• Support for multiple versions of the same software package; and
• A wide range of operating environments particularly with respect to memory, CPU performance, network usage, and storage.

When you consider users’ needs in more detail, you will recognize that a number of implications further complicate things:

• The set of applications that users wish to run will likely run under a two or more different major OS revisions (e.g. Linux kernel 2.4 versus 2.6 or Windows XP versus Vista);
• Similarly, there are applications that steadfastly refuse to run under a specific patch level.  For example, a minor revision of the Linux kernel that is lacking a specific security patch might be required.  You might be able to force the software to install but then the software is likely to no longer be supported;
• Off-the-shelf installations which seek to upgrade rather than coexist with a previous version;
• Custom software that expects a very specific behavior from a package that has changed in its most recent update;
• Software which requires particular kernel tuning which is not appropriate for general operation; and
• Software packages which have 32/64-bit library compatibility issues;

Meanwhile, grid managers will most likely be focused on providing a stable, secure, and easy to maintain infrastructure that is both cost-effective and capable of meeting the users’ core requirements.  Clearly the priorities between the individual groups and the support team will be at odds much of the time.

The most elegant solution to these issues is to build a grid whose execution environments are all virtualized.  In this situation, each usage pattern would have its own environment tailored to its own unique needs while the core OS would be under the complete control of the infrastructure staff.  Clearly there would be a stakeholder driven set of virtual servers available for use on each node in the grid. 

It seems simple enough: rather than creating a complicated infrastructure that will not accommodate all of the situations your users will require, you simply will give them their own isolated operating environments.  As you might expect, nothing is that straightforward.  The standard tools that you use for grid and virtualization management do not work well in this architecture.

In future posts, we will explore the challenges and possible solutions in detail. In particular we will focus on:

-    Networking
-    Virtual Server Management
-    Job Scheduling
-    Performance Monitoring
-    Security
-    Data Lifecycle