Qualifying the Cloud: Fact or Fiction?

There was a great deal of interest in last Wednesday’s webcast “Qualifying the Cloud: Fact or Fiction?”. Cloud Computing is certainly an issue with a number of people and your responses during the session clearly indicate that there are some regulatory concerns.

Despite adding 15 minutes to the originally scheduled session there were still more questions than we could fully answer in the time allowed and as promised we have provided written answers to your questions below.

Q. In your audit and/or customer experience, have you found that a SLA or service level agreements indicating demonstrative control over the infrastructure in the cloud is sufficient to meet GxP regulatory compliance, or are auditors still looking for IQ/OQ "installation operational qualification" checklists against a specific list of requirements

Different auditors look for different things and let’s start by saying that it’s pretty rare for regulatory inspectors to be spending any time in data centers unless there is due cause. Nowadays this is usually because of issues with an uncontrolled system that are encountered during a broader inspection.

When I am auditing on behalf of Life Sciences clients I will always look for evidence that IQ/OQ (or combined  IOQ) is performed properly. By this I mean not just that the as-built/installed infrastructure matches the configuration management records, but that the as-built/installed infrastructure complies with the design specifications and client requirements.

I once audited a major managed services and hosting provider and their processes for building and installing infrastructure platforms were very good and highly automated – which is good for the rapid elasticity required in Cloud Computing. They literally selected the options off a pick list – how much memory, how many CPUs, what disk capacity etc – and the system was built and installed accordingly in their data center accordingly.
However, there was no independent review of the specifications against the client requirements and no independent review of the as built/installed server platform against the specification. Configuration management records were generated directly from the as built/installed server and never compared against the specification.

As Neill described in the webcast, if someone had accidentally selected the wrong build option from the pick list (e.g. 20GB of storage instead of 40GB) no-one would have noticed until the Service Level Agreement requirements were unfulfilled. That’s why I will always check that there is some element of design review and build/install verification.

However, I’ll usually review the specification, design, build and verification procedures as part of the initial audit to check that these reviews are part of the defined process. I’ll also spot check some of the IOQ records to check that the verification has been done. During subsequent surveillance audits I’ll also check the IOQ records as part of whatever sampling approach I’m taking (sometimes I’ll follow the end-to-end specification, design, build/installation and verification for a particular platform or sometimes I’ll focus on the IOQ process). I'm not looking to verify the build/installation of the infrastructure myself, but I am looking for evidence that there is a process to do this and that someone has done it.

IOQ needn’t be a particularly onerous process – the use of checklists and standard templates can help accelerate this process and as long as people are appropriately trained I’m usually prepared to accept a signature of someone to say that the review activity was done i.e. a signed design specification signed by the reviewer.
As we've found in our own data center, if it's an integral part of the process (especially a semi-automated process) it doesn't have a significant impact on timescales and doesn't detract from the 'rapid elasticity' which as an essential characterristic of Cloud Computing. While issues of capacity are less of a problem in a extensible Cloud the process of IOQ does help catch other types of error (patches not being applied, two or three steps in an automated install having failed etc).

Q. Your early descriptions were good but how would you explain the concept of a the cloud to a traditional Quality person with only a very basic knowledge of Network Architecture? 

I don’t think I would!

Trying to explain to a non-IT specialist in the Quality Unit what the Cloud is always going to be difficult if you take the approach of saying that the Cloud is undefined and the Users don’t need to know what’s going on.
The way to explain it is to say that although the Users in the Regulated Company don’t need to know what the Cloud is, the Regulated Companies IT Department and their IT Quality Group do know what is going on in the Cloud, and that they have checked that it is appropriately controlled

You then need to demonstrate to your Quality Unit that you do know what’s going on in the Cloud. If it’s a Private Cloud you do this by showing them diagrams and specifications, qualification documents and so on. If it’s a Public Cloud (or an externally hosted Private Cloud) you do this by showing that you have audited the Cloud Provider to check that they have the diagrams and specifications, qualification documents and so on.
It’s all about perception. It’s okay for the Users not to know what’s going on in the Cloud, but someone clearly has to be in control. This needs to be the appropriate subject matter experts (either your own IT people or the Cloud Service Providers) and your own IT Quality Unit.

If you’re a small company without the resources or technical knowledge to assess your Cloud Providers you can rely on independent consultants for this support, but you have to select the right consultants and demonstrate due diligence in their selection.

Q. In the event of a regulatory audit, when you are using cloud resources (non-private), how does the Cloud Service Providers responsibility factor in?

Basically, you need your Cloud Service Providers to be on the hook with you and this means clearly defining what support they will provide both in terms of day to day service level requirements and in the event of a regulatory inspection.

Again, let’s emphasize that regulatory authorities rarely look in the data center without due cause and although we are prepared for them to come to our data center in Wayne, we’re not aware of any regulators actually having visited a true third party hosting facility. (However, with the concerns the industry is demonstrating around this issue we think that it’s only a matter of time before they visit someone’s third party data center, somewhere).

The worst case scenario is when, during a regulatory inspection and Inspector asks the question “Is the System Validated” and you have to say “We don’t know…” That’s when further questions will be asked, the answers to which will eventually lead to your Cloud Service Provider. A failure to have properly assessed your Provider will clearly demonstrate to the regulatory authorities a lack of control.

We know of a LOT of Life Sciences Regulated Companies who have outsourced based solely on cost, with the process driven by IT management and the accountants. They usually accept the Providers standard service levels and any involvement from quality/regulatory is often late and sometimes ignored. The result is that ‘compliance’ then becomes an added activity with added costs, the promised cost savings disappear and there is often no right to adequately audit or provide support for regulatory inspections including in the Service Level Agreement.

• Always conduct a full audit well before signing a contract (at least two days on-site, at least a month before the contract is due for signing).
• Agree in the contract how and when any quality/compliance/control ‘gaps’ from the audit (and any surveillance audits) will be addressed.
• Identify the penalties for not addressing any quality/compliance/control ‘gaps’ in the contract (this might include reducing service charges to cover the cost of the Regulated Companies additional quality/compliance/control activities or even cancellation of the contract – which we know one pharmaceutical company actually did).
• Include the right for surveillance audits in the contract.
• Include the need to support any regulatory inspections in the contract (this may never happen so can be a justifiable additional cost).

If the Cloud Service Provider won’t agree to these things we would recommend looking elsewhere or building your own Private Cloud in-house. Regulated Companies should remember that they are accountable for the control of systems they are using and that although they are the customer, the most power you’ll have in the relationship is immediately before the contract is signed.


Finally we’d just like to highlight a comment made by one of the listeners “Audit and assessment of the provider should be seen as the Insurance Certificate!” This is an excellent point and really emphasizes the key issue about Cloud Computing – you need to dig below the surface, get behind all of the hype and really understand the what, who, where and how.

There’s no reason why Cloud Computing shouldn’t be used for regulatory purposes as long as Regulated Companies exercise their responsibilities and work with Service Providers who are willing to be open about what they are doing. As far as the Users are concerned, the Cloud is still a Cloud (on-demand, rapid elasticity etc), but the Regulated Companies IT department and IT Quality group need to be in the Cloud with the Service Providers, understanding what’s going on and making sure that things are controlled.

Thank you again to everyone for their interest. The recording is still available online for anyone who didn’t catch the entire session and you can still register for the final webcast in the series via the Business & Decision Life Sciences website.