Maintaining the Qualified State

There were a couple of unanswered questions in yesterday's "Maintaining the Qualified State: A Day In The Life of a Data Center" webcast yesterday.

As usual, we've taken the time to provide the questions and answers here in our blog.

Q. Where can I get a copy of that ISPE article on (the qualification of) Virtualization?


A. There is a page on the Business & Decision Life Sciences website which provides access to the webcast on qualifying virtual environments and which also has a link to the relevant page on the ISPE website (ISPE members can download the Pharmaceutical Engineering article free of charge).
The page can be accessed at http://www.businessdecision-lifesciences.com/2426-webcast-qualification-of-virtualized-environments.htm


Our next answer responds to two similar questions:
Q. Do your IT Quality People review every change control?
Q. Do you view the change control as an auditable item, and as such require them to be written so that an auditor can understand it – requiring clarity beyond the level of a technical SME?

A. Our Quality Manager (or Quality designee) reviews and approves every Planned or Emergency Change Control. Pre-Planned (like-for-like) changes are not reviewed and approved by the Quality Manager.

However, all of the Change Control processes (Pre-Approved, Planned and Emergency) are subject to Periodic Review by the Quality Group, so if there was any issue with the Pre-Approved change process this would be picked up then.

All changes are also peer reviewed by a technical subject matter expert so we do not expect them to be written in such a way as to allow a technical ‘newbie’ to be able to understand them. Change are also reviewed by the System Owner and/or Client, to ensure that all impacts of the change are assessed (e.g. clients ability to access an application during a service outage).

The role of the Quality Manager is to ensure that the change control process is followed, not to ensure that the change is technically correct. Having said that, our Quality Team and independent Internal Auditor are technically competent and do understand what they’re reviewing, at least at a high enough level to understand what the change is about and why it’s required.

We find that some external auditors have a technical background and are able to understand the content of most change controls. However, some do not have a specific IT Quality role and can not understand the technicalities of all of the changes. If this is ever an issue during a client audit we get the originator of the change control to explain it.

Thanks again to everyone who tuned in to the live webcast. If you missed the live event the recording is still available at "Maintaining the Qualified State: A Day In The Life of a Data Center"

Qualifying the Cloud: Fact or Fiction?

There was a great deal of interest in last Wednesday’s webcast “Qualifying the Cloud: Fact or Fiction?”. Cloud Computing is certainly an issue with a number of people and your responses during the session clearly indicate that there are some regulatory concerns.

Despite adding 15 minutes to the originally scheduled session there were still more questions than we could fully answer in the time allowed and as promised we have provided written answers to your questions below.

Q. In your audit and/or customer experience, have you found that a SLA or service level agreements indicating demonstrative control over the infrastructure in the cloud is sufficient to meet GxP regulatory compliance, or are auditors still looking for IQ/OQ "installation operational qualification" checklists against a specific list of requirements

Different auditors look for different things and let’s start by saying that it’s pretty rare for regulatory inspectors to be spending any time in data centers unless there is due cause. Nowadays this is usually because of issues with an uncontrolled system that are encountered during a broader inspection.

When I am auditing on behalf of Life Sciences clients I will always look for evidence that IQ/OQ (or combined  IOQ) is performed properly. By this I mean not just that the as-built/installed infrastructure matches the configuration management records, but that the as-built/installed infrastructure complies with the design specifications and client requirements.

I once audited a major managed services and hosting provider and their processes for building and installing infrastructure platforms were very good and highly automated – which is good for the rapid elasticity required in Cloud Computing. They literally selected the options off a pick list – how much memory, how many CPUs, what disk capacity etc – and the system was built and installed accordingly in their data center accordingly.
However, there was no independent review of the specifications against the client requirements and no independent review of the as built/installed server platform against the specification. Configuration management records were generated directly from the as built/installed server and never compared against the specification.

As Neill described in the webcast, if someone had accidentally selected the wrong build option from the pick list (e.g. 20GB of storage instead of 40GB) no-one would have noticed until the Service Level Agreement requirements were unfulfilled. That’s why I will always check that there is some element of design review and build/install verification.

However, I’ll usually review the specification, design, build and verification procedures as part of the initial audit to check that these reviews are part of the defined process. I’ll also spot check some of the IOQ records to check that the verification has been done. During subsequent surveillance audits I’ll also check the IOQ records as part of whatever sampling approach I’m taking (sometimes I’ll follow the end-to-end specification, design, build/installation and verification for a particular platform or sometimes I’ll focus on the IOQ process). I'm not looking to verify the build/installation of the infrastructure myself, but I am looking for evidence that there is a process to do this and that someone has done it.

IOQ needn’t be a particularly onerous process – the use of checklists and standard templates can help accelerate this process and as long as people are appropriately trained I’m usually prepared to accept a signature of someone to say that the review activity was done i.e. a signed design specification signed by the reviewer.
As we've found in our own data center, if it's an integral part of the process (especially a semi-automated process) it doesn't have a significant impact on timescales and doesn't detract from the 'rapid elasticity' which as an essential characterristic of Cloud Computing. While issues of capacity are less of a problem in a extensible Cloud the process of IOQ does help catch other types of error (patches not being applied, two or three steps in an automated install having failed etc).

Q. Your early descriptions were good but how would you explain the concept of a the cloud to a traditional Quality person with only a very basic knowledge of Network Architecture? 

I don’t think I would!

Trying to explain to a non-IT specialist in the Quality Unit what the Cloud is always going to be difficult if you take the approach of saying that the Cloud is undefined and the Users don’t need to know what’s going on.
The way to explain it is to say that although the Users in the Regulated Company don’t need to know what the Cloud is, the Regulated Companies IT Department and their IT Quality Group do know what is going on in the Cloud, and that they have checked that it is appropriately controlled

You then need to demonstrate to your Quality Unit that you do know what’s going on in the Cloud. If it’s a Private Cloud you do this by showing them diagrams and specifications, qualification documents and so on. If it’s a Public Cloud (or an externally hosted Private Cloud) you do this by showing that you have audited the Cloud Provider to check that they have the diagrams and specifications, qualification documents and so on.
It’s all about perception. It’s okay for the Users not to know what’s going on in the Cloud, but someone clearly has to be in control. This needs to be the appropriate subject matter experts (either your own IT people or the Cloud Service Providers) and your own IT Quality Unit.

If you’re a small company without the resources or technical knowledge to assess your Cloud Providers you can rely on independent consultants for this support, but you have to select the right consultants and demonstrate due diligence in their selection.

Q. In the event of a regulatory audit, when you are using cloud resources (non-private), how does the Cloud Service Providers responsibility factor in?

Basically, you need your Cloud Service Providers to be on the hook with you and this means clearly defining what support they will provide both in terms of day to day service level requirements and in the event of a regulatory inspection.

Again, let’s emphasize that regulatory authorities rarely look in the data center without due cause and although we are prepared for them to come to our data center in Wayne, we’re not aware of any regulators actually having visited a true third party hosting facility. (However, with the concerns the industry is demonstrating around this issue we think that it’s only a matter of time before they visit someone’s third party data center, somewhere).

The worst case scenario is when, during a regulatory inspection and Inspector asks the question “Is the System Validated” and you have to say “We don’t know…” That’s when further questions will be asked, the answers to which will eventually lead to your Cloud Service Provider. A failure to have properly assessed your Provider will clearly demonstrate to the regulatory authorities a lack of control.

We know of a LOT of Life Sciences Regulated Companies who have outsourced based solely on cost, with the process driven by IT management and the accountants. They usually accept the Providers standard service levels and any involvement from quality/regulatory is often late and sometimes ignored. The result is that ‘compliance’ then becomes an added activity with added costs, the promised cost savings disappear and there is often no right to adequately audit or provide support for regulatory inspections including in the Service Level Agreement.

• Always conduct a full audit well before signing a contract (at least two days on-site, at least a month before the contract is due for signing).
• Agree in the contract how and when any quality/compliance/control ‘gaps’ from the audit (and any surveillance audits) will be addressed.
• Identify the penalties for not addressing any quality/compliance/control ‘gaps’ in the contract (this might include reducing service charges to cover the cost of the Regulated Companies additional quality/compliance/control activities or even cancellation of the contract – which we know one pharmaceutical company actually did).
• Include the right for surveillance audits in the contract.
• Include the need to support any regulatory inspections in the contract (this may never happen so can be a justifiable additional cost).

If the Cloud Service Provider won’t agree to these things we would recommend looking elsewhere or building your own Private Cloud in-house. Regulated Companies should remember that they are accountable for the control of systems they are using and that although they are the customer, the most power you’ll have in the relationship is immediately before the contract is signed.


Finally we’d just like to highlight a comment made by one of the listeners “Audit and assessment of the provider should be seen as the Insurance Certificate!” This is an excellent point and really emphasizes the key issue about Cloud Computing – you need to dig below the surface, get behind all of the hype and really understand the what, who, where and how.

There’s no reason why Cloud Computing shouldn’t be used for regulatory purposes as long as Regulated Companies exercise their responsibilities and work with Service Providers who are willing to be open about what they are doing. As far as the Users are concerned, the Cloud is still a Cloud (on-demand, rapid elasticity etc), but the Regulated Companies IT department and IT Quality group need to be in the Cloud with the Service Providers, understanding what’s going on and making sure that things are controlled.

Thank you again to everyone for their interest. The recording is still available online for anyone who didn’t catch the entire session and you can still register for the final webcast in the series via the Business & Decision Life Sciences website.

FDA to join PIC/S

On 8^th and 9^th November the Pharmaceutical Inspection Cooperation scheme (PIC/S) met in Kuala Lumpur Malaysia. The main news to come out of that was that the US FDA will become full members of PIC/S as of 1^st January 2011.  Some commentators may be saying “about time too” but what does it actually mean to the industry?

The FDA has rightly or wrongly been thought of (at least in the USA) as having the highest standards in the world for pharmaceutical products and medical devices. However the FDA applied for membership of PIC/s as far back as 2005.  So why have they not joined?

It was rumoured that under the previous US Administration there was a reluctance to allow overseas Agencies  review the FDA's own internal quality management system, which is required for new members to join PIC/S.

Another reason, if recent press reports are anything to go by, is that the PIC/s organization looked at the FDA regulations and concluded that they were insufficiently rigorous for admission to PIC/S.  Given that the FDA has now been admitted, it is an open question as to how far the FDA will go to comply with the full intentions of the scheme. Will this mean a tightening of regulations or just reciprocal recognition of inspections?

At the same time as the FDA joins PIC/S the Ukrainian regulatory agency (SIQCM) is being admitted too.  Does this mean that inspections of pharmaceutical plants in the Ukraine by the SIQCM will be recognized by the FDA as being of the same rigor as their own inspections?  This is as much a political as a regulatory question and it remains to be seen how far the FDA is prepared to go to comply with the PIC/s principles, and whether we can expect any change to regulatory law or guidance in the USA as a result.

[Posted on behalf of David Hawley]

Supplier Involvement - Don't Sign the Contract!

GAMP 5 tells us that Regulated Companies should be leveraging Supplier Involvement in order to efficiently take a risk-based approach to validation - but surely that's no more than common sense? Anyone contracting services from a Supplier should be looking to get as much out of their Suppliers as possible.

We constantly hear stories from clients, complaining about how poor some of their Suppliers are and in some cases the complaints are justified - software full of bugs, known problems not being acknowledged (or fixed), failure to provide evidence of compliance during audits, switching less skilled resources for the consultants you expected and so on.

The software and IT industry is no better or worse than any other - there are good Suppliers and less good Suppliers, but in a market such as Life Sciences the use of a less good Supplier can significantly increase the cost of compliance and in some rare circumstances place the safety of patients at risk.

Two years after the publication of GAMP 5 and five years after the publication of the GAMP "Testing of GxP Systems" Good Practice Guide (which leveraged the draft ASTM E2500) the Life Sciences industry is still:

• Struggling to understand how to get the best out of Suppliers,
• Complaining about compliance issues associated with outsourcing.

GAMP has Special Interest Groups looking at both of these issues, but it's not exactly rocket science. The real problem is that too many companies don't really understand what they're buying and focus solely on cost as a differentiator. At a time when IT industry standards provide excellent frameworks for defining requirements and service levels there are still a lot of people in the industry who go out to tender without a good definition of what they want.

This is especially true when it comes to defining quality and compliance requirements, so it's no wonder that Life Sciences companies struggle to leverage their Suppliers when they've failed to define what it is that they really expect.

In many cases quality and compliance people are involved in the selection of suppliers too late in the process to add any real value. In some circumstances there is no viable option than going with a 'less good' supplier (for instance, when a new Supplier has a really novel application or service that adds real competitive advantage) but in most cases it is possible to identify any gaps and agree how they should be rectified prior to signing a contract.

However, once a contract is signed it's too late to define quality and compliance requirements without Suppliers claiming that these are 'extras' which are outside the contract. While I've heard Regulated Companies make statements like "as a supplier to the Life Sciences industry you must have known that we'd need copies of your test results" (or whatever it is) you can't rely upon those unstated expectations in a court of law.

The result is that achieving the required quality and compliance standards often costs more that anticipated, either because the Supplier charges extra or the Regulated Company picks up the cost of the additional quality oversight. Very few Life Science's companies have actually achieved the promised cost savings with respect to the outsourcing of IT services, usually because the people driving the contract (purchasing, finance and IT) don't really understand what is required with respect to quality and compliance.

When Business & Decision are engaged in a supplier selection process we tell clients "don't sign the contract until you're happy - that's the best leverage you'll ever have over a Supplier" and it's advice worth repeating here.

At its best, the IT sector is a mature and responsible industry with standards and best practices that can be leveraged to assure that clients requirements are met. It's just a pity that the Life Sciences industry - which prides itself on being in control of most things it does - can't find a way to effectively leverage good practices like GAMP and standards like ISO 9001 and ISO 20000 to select and leverage the best IT Suppliers.

IT Infrastructure Qualification - Your Questions Answered

There we're a couple of questions relating to last week's "Pragmatic Best Practices for IT Infrastructure Qualification" webcast that we didn't get around to answering... so here are the questions and the answers.

Q.  What are the qualification strategies to be followed for infrastructure for which the applications it will support are unknown?

There are three basic approaches here:

The first is to qualify infrastructure platforms and components on the assumption of high risk severity of supported applications. This will mean that all infrastructure platforms and components are qualified such that they can support any application with no additional qualification activities being required at a later date. This is the approach taken by Business & Decision for shared platforms and components in our own data center and this provides us with the flexibility needed to meet changing customer requirements.

While this would appear ‘over kill’ to some, because qualification is really based on well documented good engineering practice (as per ASTM E2500) there is relatively little additional overhead over and above what any Class A data center should be doing to specify, build/install and test their infrastructure (this was covered in more detail in our webcast "A Lean Approach to Infrastructure Qualification").

The second approach is to qualify specific platforms and components for the risk associated with those applications that are known. This is possible for infrastructure that is dedicated to defined applications e.g. specific servers, storage devices etc. This can reduce the level of documentation in some cases, but this means that whenever a change is made at the applications layer, the risk associated with the infrastructure may need to be revisited. While additional IQ activities would not be required, it may be necessary to conduct additional OQ activities (functional or performance testing) of the infrastructure components prior to re (validating) the applications. This requires an on-going commitment to more rigorous change control impact assessment and can slow down the time taken to make changes. While Business & Decision might consider this approach for some client specific platforms and components our clients generally prefer the responsiveness the first approach provides.

The third approach (taken by some very large Regulated Companies in their own data centers) is to qualify different platforms according to different levels of risk e.g. there could be a cluster of servers, virtual machines and network attached storage dedicated to high risk applications, with the same (or a similar architecture) being dedicated to medium and low risk applications. This is probably the best solution because it balances flexibility and responsiveness with scalable risk-based qualification, but can tend to lead to over capacity and is only really a good solution in large data centers.

Q. Each component of our network software is individually validated.  What is the best strategy for qualifying the network itself?

The network isn’t really qualified in its entirety, but is qualified by way of qualifying all of the network platforms and components. This may include some functional testing of platforms or components, but the correct functioning of the network is really verified by validating applications.

The network can essentially be considered to be made up of the non-functional cables, fiber etc, the hardware (which may include firmware) and the software components that are necessary to make it work.

The software components (e.g. software based firewalls, server monitoring software, time synchronization software etc) should be designed, specified (including all configuration parameters), installed, configured and verified. Verification will include installation qualification, verification of configuration parameters and may also include some functional testing (OQ) which will be based on meeting the functional requirements of the software.

Hardware components such as bridges, switches, firewalls etc will be designed, specified, built/installed and verified. Verification will include IQ and if the ‘hardware’ component includes software (which is often the case) there will again be an element of configuration parameter verification and some functional testing. Business & Decision usually combine the IQ and OQ into a single a single verification test, simply for efficiency.

For the basic network (backbone cables, fiber, fiber switches and really ‘dumb’ network components with no configurable software element such as hubs), these will again be designed, specified, built/installed and verified, but verification will be limited to a simple IQ (recording installation details such a cable numbers, serial and model numbers etc). This can of course be done retrospectively.

All of the above can be scaled, based upon risk as discussed in the answer above.

Remmeber, if you have a question that you'd like us to answer, you can contact us on validation@businessdecision.com or you can submit your questions via the 'Ask An Expert' page on our Life Sciences website.

Computer Systems in China - to Validate or Not?

At today's Computer System Compliance session at the ISPE-CCPIE conference in Beijing we considered the new draft SFDA GMP regulations and the requirements (or not) to validate computer systems.

As blogged yesterday, the Chinese State FDA (SFDA) are committed to improving compliance, but the current draft of the new Chinese GMP regulations are ambiguous with respect to computer systems validation.

While certain Articles and Annexes infer a need to validate, calibrate etc certain control and monitoring systems there is no equivalent to US 21 CFR211 subpart 68 or PIC/S / EU Annex 11, which states a clear requirement to validate computer systems.

Some would claim that this is deliberate ambiguity on the part of SFDA and that it suits them to provide some 'wriggle room' for Chinese manufactures, but more likely is that computerized systems have been of relatively low priority. Re-writing and updating the GMP regulations is a significant undertaking and it is only reasonable that focus is given to basic GMP, especially when China is only just catching up with more developed countries with respect to the use of computer systems and there is no history and significant risk to patients as a result of failure to validate a computer system.

However, this ambiguity does not meet the stated aim of SFDA, which is to generally align national GMP regulations with World Health Organizations (WHO) guidelines. WHO guidelines (WHO Technical Report Series 937, 2006: WHO Expert Committee on Specifications for Pharmaceutical Preparations, Appendix 5) clearly have a requirement to validate computer systems, and this does appear to be missing from the current draft of the SFDA regulations.

There is also a view that SFDA want to limit the costs they are imposing on their local manufacturers at a time when the Chinese government is looking to reduce the cost of drugs and devices while at the same time as it is looking to provide healthcare to 900 million people.

However, if SFDA is serious about building a risk-based approach into the new GMP regulations it is perfectly feasible to include a clear requirement to validate computer systems while leveraging a cost effective risk-based approach which will limit the costs involved in computer systems validation according to risk.

Unless this is included in the new regulations Chinese companies face the prospect of a two tier approach to computer system validation depending on whether products are intended for the domestic or export market. This would be confusing, limit flexible operations and potentially cause problems where Enterprise Systems support domestic and export operations.

Let's hope that SFDA take on-board the need to align with international regulations in this regard and revise the current draft regulations to provide the clarity that the Chinese Life Sciences industry is looking for with respect to computer system validation.

Regulatory Changes In - and For - China

There were some interesting sessions at this morning's keynote sessions at the ISPE-CCPIE conference in Beijing.

The Chinese State FDA presented a brief history of the Chinese GMP regulations, comparing these to other international regulations (e.g. WHO) and although they provided an outline of the new Chinese GMP regulations there was no commitment in terms of a date by which these will be made effective.

Cynics at the conference suggested after the session that this is because 90% of local Chinese companies would not comply with the new Chinese GMP regulations, but while the SFDA do appear to be keeping their options open regarding timing (and appear to be moving away from introducing a target date with a period of grace during which companies could move to compliance) a session from the US FDA gave a different picture.

Its 12 months since the US FDA set up shop (a field office) in China and although there are still only seven full time FDA staff in country - with only half of these conducting for-cause and high priority inspections - there appears to have been good progress in working with the Chinese State FDA (SFDA) as well as some of the Provincial FDA offices.

What we appear to be seeing is the Chinese authorities committing to address the regulatory/quality concerns that threatened to impact their export markets last year while also starting to address the reform of regulations in their home market, recognizing that the latter will take a while to address in a market consisting of literally thousands of (rapidly consolidating) manufacturers and distributors.

The US FDA and SFDA now meet on a monthly basis, with the SFDA acting as observers to some US FDA inspections. The US FDA is helping to fund training for the SFDA, reviewed the pending Chinese GMP regulations and have provided 10 GCP regulations for the SFDA to translate. This is all part of the US FDA strategy of helping to educate other regulatory agencies on the requirements of the US market and to help build inspection capacity (through the education of a cadre of SFDA inspectors trained in international regulatory expectations).

At the same time, multinational Life Sciences companies are sharing concerns about Chinese products with the US FDA, who are in turn discussing issues with the SFDA and there is also agreement between the US FDA and SFDA to focus on ten high risk products (mainly pharmaceutical, but some medical devices).
This co-operation provides evidence of the US FDA's desire to work more effectively with other regulatory agencies and will certainly start to address concerns about Chinese product.

At the same time it will also help the Chinese authorities to better regulate their own market, which is forecast to be world's largest by 2013 (behind the US and Japan). While 'rogue traders' operating out of China will undoubtedly be of continuing concern with respect to product quality and counterfeiting at least problems with the legitimate market are starting to be addressed.

Whatever people may think about the Chinese governments method of implementing change, there is no doubt that effective reforms can be implemented and probably more quickly than in many other markets. Although this is just the beginning there is no doubt that regulatory change is happening - which will be good for patients in China as well as the rest of the world.