How to Use Grids to Cure Cancer

By Ivo Janssen

Part 11 in the pharmaceutical sector, SOX in
the financial world. These days, businesses have to adhere to a large slew of
regulations, and implementing a production grid has not become any easier. This
article will go into some of the factors that have to be weighed when installing
grids in these audited types of environment, with specific attention to the Part
11 regulation of Desktop Grids.

First a little background. Title 21 CFR Part 11, or
simply "Part 11", defines the criteria under which electronic records and
electronic signatures are considered to be trustworthy, reliable and equivalent
to paper records. Practically speaking, Part 11 requires drug makers, medical
device manufacturers, biotech companies, and other to implement controls,
including audits, validation systems, audit trails, electronic signatures, and
documentation for software and systems. [1]

Unfortunately, there is no such thing as a handbook
or a turnkey "Part 11 compliant system". Part 11 requires both procedural
controls (i.e. notification, training, SOPs, administration) and administrative
controls to be put in place by the customer in addition to the technical
controls that the vendor can offer. At best, the vendor can offer an application
containing the required technical requirements of a compliant system [2]. As
such, a grid vendor must work with each individual customer in deploying a
custom system. In my company, Univa UD, I have been responsible for implementing
Part 11 compliance with our product, Grid MP, in a number of pharmaceutical
companies around the world.

With grids, and especially desktop grids, the
following issues must be addressed:

• User authentication
  
• Application lifecycle
  
• Job tracking
  
• Auditing
  
• External influences on other
  systems

User
authentication
A Grid job will typically run on a desktop as a
different user than the submitting user, thus, an SOP for User Management is
very important here. This SOP should include items such as:

• Creation of users: Who can create users? Is this a
  decentralized process? Linked to a unified logon such as Windows AD or LDAP?
  
  
• Password management: How often do passwords expire?
  Will plaintext passwords in a configuration file for job submission be allowed?
  
  
• User access control: What application binaries and
  input data can a user access?

Application lifecycle
It is of the utmost important in an audit trail
that one can say with certainty that a job was run against a certain specific
version of a binary. For instance, Grid MP offers an abstraction model for
binaries into a "Program" object. The following processes must be
well-defined:

• Application version tracking: Are old versions of an application stored and
  saved in the system?
  
• Jobs: Do jobs
  have a reference to the exact binary or version that was used to compute a
  certain job?
  
• Verification: Have applications gone through a
  proper test/stage/production cycle before being promoted to production? See
  also this entry in this blog.
  
• Paper trail: Do all verification steps and
  application promotions have the proper sign-off and paper trail as required by
  Part 11?

Job tracking
Keeping track of jobs and their data is possibly
one of the most important and straightforward items in a Part 11 compliant
system. These are the aspects that need to be considered:

• Tracking: Are all job attributes tracked? Is it
  enough to just list the job submitter? Does
  one allow modifying of an existing job by another user, and if so, are
  all modifications tracked?
  
• Saving: Is Job meta data (submitter, date, etc)
  tracked? Are results saved? Does one know
  on which devices jobs ran, and what the state of those devices was at that time
  (other software, load, etc.)?

Auditing
Auditing and a paper trail is really the core of
the Part 11 regulation, out of which all previous issues rise.

• Job audit: Is
  there a log of each and every job in your system? In there a log of any modifications to a job?
  Does one track which user created, edited
  or deleted a job? For instance, Grid MP
  has a reporting subsystem called MP Insight that will track each job, data and
  result in the system.
  
• Log files: Not everything is typically saved in such
  reporting systems. Are there other logfiles that need to be retained? Is there a
  way to authenticate these logfiles, and make sure they are not tampered with? A
  digital checksum? A hardcopy with a signature?

External influences
Since Desktop Grids by definition run on hardware
that is already used for other tasks as well, the following question arises:
Even if you Grid is not qualified, does it affect other qualified software
running on these machines? What if your unqualified job runs on a qualified lab
machine doing clinical tests?

• Application certification: Is there a plan or
  methodology to certify new binaries to make sure that they do not affect (i.e.
  crash) other applications outside the sandbox? Is
  there a sandbox or virtualization technique to shield the grid
  application from the rest of the machine? Is this certification signed off with
  the proper paper trail? Which departments should be involved the sign-off?
  
  
• Tracking: Is there an
  audit trail of what job with what data ran on a machine at any given
  time? Is there information on the state of that machine (other
  installed software base, load) at the time of the grid jobs?

By no means is the above list an exhaustive list,
but it does give you an idea of the scope of questions and problems that arise
when trying to incorporate a grid in a qualified environment. I have worked
with various Top-10 pharmas to implement grids in their qualified environment,
and as I stated in the beginning, there is no Right Way to do so, each company
will have different interpretations of the level of log gathering, retention and
scope. If this all seems daunting to you, then rest assured that Univa UD
offers various consulting and service offerings to help you with these decisions. ;-)