Collaborating on Collaboration - Validating Open Source Software

I'm delighted to hear that "Open Source Software in Life Science Research: Practical Solutions to Common Challenges in the Pharmaceutical Industry and Beyond" has just been published (available from Woodhead Publishing and also on Amazon).

 

The book has an interesting history, having started life as a discussion on LinkedIn on the use of open source software in the Life Sciences industry. After a very illuminating discussion it was suggested that we write a book on the subject and so Lee Harland of Pfizer and Mark Forster of Syngenta U.K. agreed to take on the editing of the book, which would look at the how open source software is used in our industry.

The result is a comprehensive look at the the current state of the market, with chapters looking at the use of various open source tools and packages looking at predictive toxicology, mass spectrometry, image processing, data analytics, high throughput genomic data analysis, web based collaboration and many, many more applications. As well as addressing general issues the book looks at specific tools and applications and is a useful reference for anyone looking for a guide as to the kind of software that is out there (many of these applications are quite well 'hidden' on the Internet!)

Without doubt, open source software is widely used in pharmaceutical research and development and is transforming the way the industry works in many ways. Open source software has many advantages - it's free to acquire and rather than wait for a software vendor to understand and respond to market requirements, the developer community just goes ahead and extends functionality in the direction that research and development needs.

My contribution to the book ("Validation and Regulatory Compliance of Free/Open Source Software") sounds a slightly more cautious note, highlighting when open source software may require validation, the challenges of validating open source software and sharing some ideas on how to go about it - including collaboratively!

This can be challenging, time consuming and does of course have costs associated with it, which is why we see less open source software in GMP areas. However, I have no doubt that the trend to collaboratively develop niche applications will continue to expand, especially with the prevalence of mature software development tools and Platform-as-a-Service development environments.

The process of collaborating to write a book is quite interesting - just like collaborating to develop software you're not quite sure what you're going to get, sometimes some contributors may head off at a tangent and you're never quite sure when everything is going to be delivered. Well done to Lee and Mark for sticking with the process. I think that the end result is well worth the wait.

Retrospective Validation of Enterprise Systems

Yesterday was the fourth and final stop on our virtual book tour, looking at important topics from "Validating Enterprise Systems: A Practical Guide" (published by the Parenteral Drug Association and available via their bookstore)

In yesterday's session (recording and slides available here) we looked at the topic of retrospectively validating enterprise systems including why it is important to retrospectively validate systems that have not previously been validated and various reasons which lead to retrospective validation.

As we discussed yesterday, some of these reasons are understandable and acceptable to regulatory authorities while other reasons (such as ignorance or cost avoidance) are not.

We got through most of your questions yesterday but there were a couple of questions we didn't quite get around to answering so here they are.

Q. How is retrospective validation different to normal validation?

A. Many of the actual activities are identical but when retrospectively validating the system it is possible to leverage existing documentation (if any) and actual experience with the system.
Where good documentation exists and has been kept up-to-date the task of retrospective validation can be relatively quick. However, when little or no documentation exists it can take almost as long to retrospectively validate a system as it did implement it.

If the system has been maintained and supported using well documented processes such as help desk, incident management and problem management it will also be possible to leverage this information and use it to inform detailed risk assessments. With actual empirical data it is possible to make a more accurate assessment of risk likelihood and probability of detection.

Where a system was well implemented and has been well maintained this additional information can be justifiably used to limit the extent of the verification activities required as part of the retrospective validation.

Where this empirical data highlights problems or issues with the system it can also be used to determine which areas of the system require greatest focus as part of the risk-based validation effort.

This can mean that in some cases retrospective validation can be more successful than prospective validation in terms of appropriately addressing real risks in a cost-effective manner. However, as we stated in the webcast yesterday, because retrospective validation is not conducted as part of the implementation activities it is generally significantly more expensive than prospective validation which is well integrated with the original implementation effort. For this reason retrospective should be avoided.

Q. How common is retrospective validation? Do many companies have this problem?

A. There is a good deal of variation in the industry. At one end of the scale there are well-managed companies who are well aware of their regulatory responsibilities and who prospectively validate all appropriate enterprise systems at the time of implementation. At the other end of the scale there are companies who are either ignorant of the need to validate certain enterprise systems, or who choose to ignore the requirement in order to save money.

To some extent this depends on the maturity of the market and of the organisation. The most mature organisations have learned that risk-based prospective validation adds relatively little to the cost of implementation and provides good return on investment in terms of more reliable and robust solutions.

Less mature organisations still do not understand the cost benefit of risk-based validation or are not aware of the need to validate certain enterprise systems. While to some extent the US and Europe are more mature markets in this regard this is not universally the case. There are still new start-ups and companies where profit margins are slim who still do not prospectively validated their systems.

In markets which have historically been seen as less mature (e.g. Asia, Africa, Latin and South America) there is a growing realisation of both the need and attractiveness of validation with respect to implementing reliable and robust enterprise systems which help to streamline business operations. While retrospective validation is currently quite common in these markets (as they move to meet the growing expectations of changing local regulations and loom to export products to markets where the need for validation is already well established) this will change over time - and quite rapidly if other indicators of change are reflected.

This means that while retrospective validation will continue to be required in many markets for a number of years to come (and hence the need for a chapter on the subject in "Validating Enterprise Systems: A Practical Guide") we predict that this will be the exception within the next 5-10 years, with retrospective validation becoming rarer in all markets.


Thanks to all of you who have supported the virtual book tour we do hope you will join us for our forthcoming series of IS compliance and computer system validation webcasts over the next few months (details and registration available here)

Using non-ERES Compliant Business Intelligence Tools in Support of GxP Decision Making

Another interesting question here - this time posted in the LinkedIn 21 CFR Part 11 discussion group on the use of business intelligence tools to support GxP decision making,. However, the business intelligence tools are not 21 CFR Part 11 compliant!

Q. We are implementing a 'system' which comprises of replicating software from Oracle and a destination replicated Oracle database. The purpose of this 'system' will be for running Oracle and Cognos reports from this replicated database instead of the high transactional source database. This system will be accessed only by the DBA.

The replicating software is pretty much command line based which is basically to say it does not have a GUI. From the command prompt user interface of this software, we can define (enter commands) for the source and target database; define the number of processes to run; set filters for what data are to be replicated and the frequencey for replication.

We did an assessment and found that part 11 is applicable. The problem is that we cannot satisfy all the part 11 requirements.

Although we deal with e-records (store the data from the source system including setup/ configuration of replication process) how do we still justify that e-signatures, audit trail requirements or password aging/restrictions are not applicable and not supported by the replicating software?

A. We've just finished doing exactly this for another project where we set up a reporting database to reduce the load on the transactional system. Some of the reports from the data warehouse were GxP significant.

 

We also had similar issues consolidating medical device traceability data for the device history record from signed records in SAP to the SAP Business Warehouse (BW) where we lost the secure relationship between the signed records in the transactional database and the copy in BW.

 

It's all about clearly identifying what records and data are used for what purposes, documenting the rationale and design, developing or updating appropriate SOPs and training your users and DBAs accordingly.

 

The first thing to do is to clearly differentiate between the records that you will ultimately rely on for regulatory purposes versus the data that you use for reporting. We maintained (and clearly documented) that the signed records in the transactional system would always be the ultimate source of data for regulatory decision-making (i.e. product recall, CAPA investigations etc). Where applicable these were signed and could be verified and were still Part 11 compliant.

 

Our rationale was that the data warehouse (and associated GxP reports) were acting as a grand 'indexing system' which allowed us to find the key regulatory records much faster (which has to be good for patient safety). We would not use these reports for making regulatory critical decisions, but we did use these reports to be able to more quickly find the key records in the transactional database which we could then rely upon for regulatory decision-making. Under that rationale the records and signatures which was subject to Part 11 remained in the transactional database.

 

We made sure that SOPs for key processes such as product recall, CAPA investigation etc were updated to reflect the appropriate use of the reports and the signed transactional records. We were able to demonstrate that using the data warehouse reports was between 2 to 3 times faster than running queries and filters in the transactional system and we could find the records much faster. In our testing we were also able to demonstrate that we always found the correct record (more than 99.9% of the time) unless something had subsequently been changed in the transactional database (less than 0.1% of the time) and even where this was the case the discrepancy was obvious and would not lead to erroneous decision-making.
 

However, that's not to say that the GxP significant reports were not important. We still validated the software which replicated the databases, we qualified the underlying IT infrastructure and we validated the GxP significant reports.

 

We essentially had three categories of report:

• Non-GxP reports, which were not validated (this was the majority of reports)
•  GxP significant reports, which were not based upon signed copies of Part 11 records, but which were validated.
• GxP significant reports, which were based upon signed copies of Part 11 records help in the transactional system.These were validated and were clearly annotated to the effect that they should not be relied upon for regulatory decision-making and they also gave a reference to the signed, transactional record in the transactional database.

 

On both of these projects we had much better tools for the replication of the data. Since you're using Oracle databases we would recommend that you create (and maintained under change control/configuration management) PL SQL programs to replicate the data bases. This will significantly reduce the likelihood of human error, allow you to replicate the databases on a scheduled basis and make it much easier to validate replication process.

 

For more background (discussion paper and webcast) on the use of validated Business Intelligence tools for GxP decision making on our website at http://www.businessdecision-lifesciences.com/2297-operational-analytics-in-life-sciences.htm.

Validating Clouded Enterprise Systems - Your Questions Answered

Thank you once again to those of you who attended the latest stage on our virtual book tour, with the latest stop looking at the validation of enterprise systems in the Cloud. This is in relation to chapter 17 of "Validating Enterprise Systems: A Practical Guide".

Unfortunate we had a few technical gremlins last Wednesday (both David Hawley and myself independently lost Internet access at our end just before the webcast was due to start) and so the event was postponed until Friday. Our apologies again for that, but we nevertheless received quite a number of registration questions which were answered during the event (you can find a recording of the webcast and copies of the slides here).

We did manage to get through the questions that were asked live during the webcast but we received one by e-mail just after the event which we thought we would answer here in the blog.

Q. "What elements should go into a Master VP for Clouded application / platforms?

A. It depends on the context that the phrase Master Validation Plan is being used. In some organisations a Master Validation Plan is used to define the approach to validating computerised systems on an individual site, in an individual business unit or, as will be the case here, for applications in the Cloud.

In other organisations a Master Validation Plan is used to define the common validation approach which is applied to an enterprise system which is being rolled out in multiple phases to multiple sites (each phase of the roll-out would typically have a separate Validation Plan defining what is different about the specific phase in the roll-out)

Logically, if we are implementing a Clouded enterprise application it could (and often would) be made available to all locations at virtually the same time. This is because there is limited configuration flexibility with a Software-as-a-Service solution and different sites have limited opportunities for significant functional differentiation. In this context is it is unlikely that the second use of a Master Validation Plan would be particularly useful so we'll answer the question in the first context.

Where a Master Validation Plan is being used to define the approach to validating Clouded enterprise systems it need to define the minimum requirements for validating clouded applications and provide a framework which:

• Recognises the various cloud computing models (i.e. Infrastructure-as-a-Service, Platform-As-a-Service, Software-as-a-Service; Private Cloud, Community Cloud, Public Cloud and Hybrid Cloud; On-Premise and Off-Premise
• Categorises platforms and applications by relative risk and identifies which cloud models are acceptable for each category of platform/application, which models are unacceptable and which ones may be acceptable with futher risk controls being put in place
• Identifies opportunities for leveraging provider (supplier) activities in support of the regulated company's validation (per GAMP 5/ASTM E2500)
• Stresses the importance of rigourous provider (supplier) assessments, including thorough pre-contract and surveillance audits
• Highlights the need to include additional risk scenarios as part of a defined risk management process (this should include risks which are specific to the Essential Characteristics of Cloud Computing as well as general risks with the outsourcing of IT services)
• Lists additional risk scenarios which may need to be considered, depending upon the Cloud Computing model being looked at (these are discussed in our various webcasts)
• Identifies alternative approaches to validating clouded enterprise systems. This would most usefully identify how the use of Cloud computing often prevents traditional approaches to computer systems validation from being followed and identifies alternative approaches to verifying that the Software-as-a-Service application fulfils the regulated companies requirements

With respect to the last point our webcast "Compliant Cloud Computing - Applications and Software as a Service" discusses issues with the validation of Software-as-a-Service applications using traditional approaches and outlines alternative verification techniques that can be used.

Whether it is in a Master Validation Plan or some form of Cloud strategy document, it is important for all regulated companies to start to think about how they will validate Clouded applications. This is clearly a topic that is not going to go away and is something that all life sciences companies will need to address.

You may also be interested to know that on 15th November 2012 we're going to be looking more closely at the current state of the Cloud computing market specifically with respect to meeting the need of regulated companies in the life sciences industry .  We'll be talking about where the market has matured and where appropriate providers can be leveraged - and where it hasn't yet matured. Registration is, as ever, free of charge and you can register for the event at the Business & Decision Life Sciences website.

We look forward to hearing from you on the last stage of our virtual book tour when we'll be looking at the retrospective validation of enterprise systems, which we know is a topic of great interest to many of our clients in Asia, Eastern Europe, the Middle East and Africa and in Latin and South America.

PLCs and GAMP Categorization

Here's another interesting question and answer that we thought we'd share on-line with a wider audience:

Q. We are developing a standard template to help in identifying and classifying PLCs into GAMP 5 categories. Could you please guide us to develop the right tool for the same.

A. The 'questions' to include are relatively simple and relate to the GAMP definitions of the categories, specifically on the context of PLCs - for example:
- Is the PLC (or parts of the software) used 'as is' with no modification or with simple changes to parameters e.g. run time, setpoint etc (typically with a machine or piece of equipment)? [Cat 3]
- Is the PLC (or parts of the software) reconfigured using standard graphical programming languages e.g. ladder logic, function blocks? [Cat 4]
- Is the PLC (or parts of the software) programmed e.g. are you writing code to achieve a function or operation that is not a standard feature of the PLC? [Cat 5]

You should also remember that it is quite likely that PLCs will contain a combination of Cat 3, 4 and 5 software and that your validation approach should reflect this.

However, the on-going  problem you will have is two-fold:

1. PLC systems have a great deal of variation in the way they they are parametrized/configured/coded, and sometimes the lines are blurred eg. some 'configuration' can be as complex and error prone (risk-likelihood) as traditional coding. You can't possibly expect to come up with questions and guidance on every type of PLC out there.

2. You need to educate people on what all of these terms mean, in the context of the PLC you are looking at. This takes time and experience.

The only time I have seen this done successfully is when a company standardises on just one or two manufacturers PLCs and then they can provide a checklist / guidance based on the manufacturers specific software development techniques.

When developing the validation approach you will also have to take into account the work that may already have been done by the supplier if the PLC is part of an embedded system.

If you are implementing a lot of PLC systems it will be worth developing an SOP or guidance document on the topic. You will however need to develop some subject matter expertise to be able to guide and support your engineers, because this is a specialist area and needs experience to do cost effectively.

Practical Risk Management - Webcast Follow Up

Thanks once again to those of you who joined us for yesterday's webcast - the second stop on our "virtual book tour" which looked at practical risk management. We had a good number of questions asked as part of the registration process which we handled in yesterday's webcast (you can watch a recording of the webcast and download the slides here) but unfortunately we didn't have time to answer all of your questions that were asked during the session.

As usual, we taken the time to answer the outstanding questions here on a life sciences blog.

Q. Can you say more about regulators who are worried about misused risk assessments?

A. During the webcast we mentioned that the number of inspectors from European and US regulatory agencies had commented that they have concerns about the quality of risk assessments and the resulting validation. This comments have been made during informal discussions and in one case at a conference.

Their concern is that the resulting validation is either not broad enough in terms of scope or not rigorous enough in terms of depth and that this has been uncovered during inspection of what they believe to be relatively critical systems. In a couple of cases inspectors have commented that they believe that this is a case of the companies involved using risk assessment as an excuse to reduce the level of effort and resources applied in validating such systems.

We know from their comments that in a number of cases this has led to inspection observations and enforcement actions and it appears that a number of regulatory inspectors are in their words "wise to the trick". As we said in a webcast yesterday is important that the scope and rigour of any validation is appropriate to the system and the risk assessment is used to determine which areas and functions in the system require greater focus. The objective of risk-based validation is not to simply produce a level of effort and expenditure but ensure that the efforts and resources are applied most appropriately.

Q. How much time and effort can be saved by using the right risk assessment approach?

A. Our experience is that by using a relative risk assessment process rather than a quantitative risk assessment process it is possible to reduce the time and effort spent on assessing risks by between 50 to 75%. We have also studied the outputs of both types of risk assessment process on very similar systems and it is encouraging to note that in many cases both processes have provided very similar outputs in terms of the distribution of high, medium and low risk priorities both in terms of the relative number of each risk priority grouping and the functions allocated to each group.

This means that for enterprise systems with lower risk it is possible to reduce the time spent assessing risks by half or three quarters and still come up with results which are sufficiently accurate to support appropriate risk-based validation. This is why it is so important that regulated companies have a variety of risk management processes and tools available to them so they can use the most appropriate and cost-effective approach.

Q. When would you use a quantitative risk assessment approach? For what type of systems?

A. You would typically use a quantitative risk assessment approach where it is necessary to distinguish low, medium and high risk impact amongst a variety of requirements or functions that are all or are mostly of high GxP significance. In this case a quantitative (numeric) approach allows you to take a more granular view and again focus your verification activities on the requirements or functions which are of the highest risk impact.

Typically these will be systems which are safety critical and while this approach could be very useful in terms of manufacturing systems, in terms of enterprise systems we see this approach being used to the most critical systems such as adverse event systems (AES), LIMS systems used for product release, MES etc. Even with these systems quantitative risk assessment can be used on a selective basis for those modules which the initial risk assessment determines to be most critical.

Q. Who should conduct the risk assessment of EDMS system supporting the whole Enterprise?

A. Risk assessments cannot be conducted alone. This was a key points bought out in this week's GAMP UK meeting where we ran a risk assessment exercise and it was clearly valuable to have a variety of opinions and experience feeding into the process. You need people who understand the requirements, the business processes and the resulting risks to give their expertise with respect to risk impact.

You also need technical subject matter experts from the engineering or IT group who are much more likely to understand the risk likelihood. Both groups can contribute to thinking about risk detectability, either in terms of detecting risks within the system or as part of the normal business process checks.

It is therefore very important to invite the right people with the right breadth and depth of knowledge to any risk assessment exercise and to allow sufficient time for the relevant risk scenarios to be identified and assessed.


Thank you as ever for your interesting questions - we hope we find the answers above useful. Remember that you can join us on 17th October when will be looking at the very thorny issue of validating enterprise systems in the Cloud  as Software-as-a-Service (registration is free and is open here)

Validating Enterprise Systems - Questions Answered

Thanks to those of you who attended the first session on our 'virtual book tour' - in which we looked at the some of the basics of validating corporate applications, based on the the book "Validating Enterprise Systems: A Practical Guide"

A number of people submitted questions before the webcast which we answered during the session and we also had time to answer a couple of questions submitted at the end of the session. You can see a recording of the webcast and download a copy of the slides at the Business & Decision Life Sciences website.

There were however a couple of questions that we didn't get around to and as usual we've addressed them here:

Q. What challenges to you see for Enterprise Systems in the next five years?

A. Over the past five years we've seen the functional footrpint of systems such as ERP, CRM, LIMS, PLM etc being extending beyond their traditional boundaries (as discussed in the webcast). However, even where business processes are extending beyond traditional 'silos' in the business, the extended business process is still very much within the enterprise.

We are already seeing Life Sciences companies struggle to extend their business processes beyond their own boundaries and this is becoming even more important as companies collaborate and outsource more. (I'm talking about this very issue at the Xavier / FDA Global Outsourcing Conference next week).

Enterprise systems will therefore need to be capable or orchestrating business process and consolidating data from across the extended enterprise - meaning across business partner, supplier and even customer processes and systems. While the tools and technical standards exist to start to do this today, very few companies are actually taking this step and it is an area which promises true business advantage.

Q. How do you see the use of Service Oriented Architecture changing the game with respect to validation?

A. This is really linked to the previous question. SOA - and Business Process Orchestration- is one of the enabling technologies that allows end-to-end business processes to be defined, monitored and optimized both within the traditional enterprise boundary and across the extended enterprise.
These changes will bring compliance and validation challenges.

Key regulatory business processes - such as adverse events management and product recall - will need to be coordinated across the extended enterprise. This will require validation of the end-to-end business process as well as the computer systems validation of the individual applications and Middleware / SOA components.

Like the technology, the techniques for this already exist but relatively few organizations know how to achieve such compliance efficiently in complex organizations and with such integrated architectures. This will be a challenge both to business process owners, to IT and especially to Quality and IT Compliance and CSV professionals, who sometimes tend to lag behind in applying well established principles to new technologies. It's therefore important that we keep up not only with the changes that have taken place over the past five years, but with the changes that are still to come.


In the next stage of the 'virtual book tour' we'll be looking at how the approach to risk management has changed in the last five years - we hope you'll be able to join us for that session, which is on 3rd October (registration is available here)

Measuring the Value of Validation?

An interesting question today on measuring the value of validation and assigning resources accordingly - something we see very few Life Sciences organizations doing well.

Q. How do others measure the value that validation creates within an organisation?

Does anyone have any experience of assigning value to validation activities?

I'm interested in how others may allocate resource and how within the validation planning process limits are/can be defined in terms of the cost of man hours against the economic benefits created from validation?

 

A. We have a process as part of our 'Lean IS Compliance' programme to put KPIs in place and measure cost and compliance levels (http://www.businessdecision-lifesciences.com/1170-lean-is-compliance.htm if anyone is interested). The challenge with measuring the value of validation is that it's difficult to compare projects with and without validation.

 

Most projects are different and many companies only see the cost of validation and not the value.

However, it can be done when you are e.g. rolling our an ERP system within a wider organisation and some business units need the system validating (APIs) and other business units do not (bulk chemicals).

 

Even in these cases most organisations only measure the cost and see validation as a 'negative', but if you are clever (which is what Lean IS Compliance is all about) you can also measure the value in terms of hard metrics (time and cost to fix software defects that make it into production) as well as soft metrics (user satisfaction when the system does - or does not - work right first time).

 

However, these are general benefits which accrue to any software quality process.

 

Although the principle of risk-based validation is to assign greater resources to systems and functions of greater risk, most companies again see this as an opportunity to reduces the level of resources assigned to lower risk systems/functions and the focus is again not on benefits.

 

It is possible to look at the implementation of systems of comparable size/complexity, where one system is high risk (and has more validation resources/focus) and another system has low/no risk (and few/no validation resources). Our work clearly shows that the higher risk systems do indeed go into production with fewer software issues and that this does have operational benefit (hard and soft benefits).

 

However, few companies track and recognise this and cannot correlate the investment in validation (quality) with the operational benefits. This is often an accounting issue, because the costs are usually capital expenditure and the benefits are associated with (lower) operational expenditure.

 

To really pull together a programme like Lean IS Compliance needs:

• Quality/Regulatory to truly accept a risk based approach (and that enough is enough)
• IT to understand the value of software quality activities (to which formal validation adds a veneer of documentation)
• Accountants to be willing to look at ROI over longer timescales than is often the case.

Computer System Validation Policy on Software-as-a-Service (SaaS)

In a recent LinkedIn Group discussion (Computerized Systems Validation Group: Discussion "Validation of Cloud), the topic of Software-as-a-Service (SaaS) was widely discussed and the need to identify appropriate controls in Computer System Validation (CSV) policies was discussed.

The reality is that relatively few compliant, validated SaaS solutions are out there, and relatively few Life Sciences companies have CSV policies that address this. 

However, there are a few CSV policies that I’ve worked on that address this and although client confidentiality means that I can’t share the documents, I did volunteer to publish some content on what could be included in a CSV policy to address SaaS.

Based on the assumption that any CSV policy leveraging a risk-based approach needs to provide a flexible framework which is instantiated on a project specific basis in the Validation (Master) Plan, I've provided some notes below (in italics) which may be useful in providing policy guidance. These would need to be incorporated in a CSV Policy using appropriate language (some Regulated Company's CSV Policy's are more prescriptive that others and the language should reflect this).

"When the use of Software-as-a-Service (SaaS) is considered, additional risks should be identified and accounted for in the risk assessment and in the development of the Validation Plan computer system validation approach. These are in addition to the issues that need to be considered with any third party service provider (e.g. general hosting and managed services). These include:

• How much control the Regulated Company has over the configuration of the application, to meet their specific regulatory or business needs (by definition, SaaS applications provide the Regulated Company (Consumer) with little or no control over the application configuration)

o   How does the Provider communicate application changes to the Regulated Company, where the Regulated Company has no direct control of the application?

o   What if Provider controlled changes mean that the application no longer complies with regulatory requirements?

• The ability/willingness (or otherwise) of the Provider to support compliance audits

• As part of the validation process, whether or not the Regulated Company can effectively test or otherwise verify that their regulatory requirements have been fulfilled

o   Does the Provider provide a separate Test/QA/Validation Instance?

o   Whether it is practical to test in the Production instance prior to Production use (can such test records be clearly differentiated from production records, by time or unique identification)

o   Can the functioning of the SaaS application be verified against User Requirements as part of the vendor/package selection process? (prior to contract - applicable to higher risk applications)

o   Can the functioning of the SaaS application be verified against User Requirements once in production use? (after the control - may be acceptable for lower risk applications)

• Whether or not the Provider applies applications changes directly to the Production instance, or whether they are tested in a separate Test/QA Instance

• Security and data integrity risks associated with the use of a multi-tenanted SaaS application (i.e. one that is also used by other users of the system), including

o   Whether or not different companies data is contained in the same database, or the same database tables

o   The security controls that are implemented within the SaaS application and/or database, to ensure that companies cannot read/write delete other companies data

• Where appropriate, whether or not copies of only the Regulated Companies data can be provided to regulatory authorities, in accordance with regulatory requirements (e.g. 21CFR Part 11)

• Where appropriate, whether or not the Regulated Companies data can be archived

• If it is likely that the SaaS application is de-clouded (brought in-house or moved to another Provider)

o   Can the Regulated Companies data be extracted from the SaaS application?

o   Can the Regulated Companies data be deleted in the original SaaS application?

If these issues cannot be adequately addressed (and risks mitigated), alternative options may be considered. These may include:

• Acquiring similar software from an acceptable SaaS Provider,
• Provisioning the same software as a Private Cloud, single tenancy application (if allowed by the Provider)
• Managing a similar application (under the direct control of the Regulated Company), deployed on a Platform-as-a-Service (PaaS)"

Hopefully these ideas will help people to develop their approach to SaaS, but CSV Policies should also address the use of PaaS and IaaS within the broader context of outsourcing.

Successful and Compliant ERP Projects

Unfortunately we ran out of time in yesterday's webcast “Secrets to Success - Plan and Implement Compliant ERP Projects”.

That was partly my fault because I was late dialing in. (Apparently, Microsoft Exchange/Outlook still doesn't automatically recognize that the US and Europe change to daylight savings on different weekends - doesn't anybody validate this software?). My apologies for that, and for the fact that we ran out of time to answer all of your questions as fully as we would have liked.

During the webcast we discussed how small to medium-sized life sciences companies can plan for the successful implementation of their ERP systems. We looked at how to align project planning and validation planning activities, we reviewed the typical project activities that are the responsibility of the regulated company and we looked at the importance of assigning the right people to the project. Below are the questions we didn't have time to answer fully and our answers - we hope that you find them useful.

Q. How does implementing ERP in pharmaceuticals vary from other industries?

A. The main differences are that in many cases the system requirements represent mandatory regulatory requirements which have to be fulfilled. There is no option to defer these to a later release and so many of the software vendor's ‘accelerated’ implementations using out-of-the-box software configurations cannot be used. There is also the fact that requirements, design specifications and testing all need to be formally documented and there may also be the issue of electronic records and electronic signatures to consider.

Although it is possible to implement an ERP system in a small to medium business within 8-12 weeks, the above factors make this virtually impossible in the life sciences industry. The fastest that we have ever been able to implement an ERP system in life sciences has been 16 weeks and for small to medium business 6 to 8 months is more typical.

Q. Does the increased focus on formal project management have any benefits?

A. The focus on formal project management controls means that time and cost overruns are usually better controlled. The focus on the formal definition and documentation of requirements also means that systems are much more likely to meet the real requirements of the real users. While the time and cost of implementing in life sciences is greater than some other industries, the fact that the system more completely fulfills the user requirements generally provides a better return on investment.

As an industry we must do a better job in demonstrating return on investment in order to justify the increased time and cost when compared to other industries. Where case studies are available they clearly show that a formally defined project management process, documented requirements specifications and tests and the need to demonstrably confirm that user requirements have been fulfilled delivers an ERP system that is fit for purpose, better meet the needs of users and provides better return on investment over the life of the system.

Q. How realistic are regulated companies in their expectations when looking to implement ERP or CRM?

A. Clients can certainly be very demanding and their expectations can be difficult to manage, especially when those expectations are informed by software vendors and system integrators who don’t really understand the life sciences industry.

As a company constantly engaged in implementing ERP and CRM systems but also competing to win such projects we often see small to medium life sciences companies with unrealistic expectations with respect to the real project budget, how long it will take to implement the system and the level of commitment their people will need to devote to the project. This is natural where the procurement process doesn't really understand the need for regulatory compliance, under values the benefits of formal validation and focuses mainly on comparing costs and implementation timescales.

On the validation side of business we have worked with a number of system integrators who are inexperienced in the life sciences industry and as a result we’ve had to help a lot of regulated companies bridge the gap between their initial expectations and what is really required for a successful and compliant project.

The reality is that it takes a minimum amount of time and effort to successfully implement a compliant ERP or CRM system. Small to medium life sciences companies would be better served by starting projects with realistic expectations and thereby avoiding having to go back to stakeholders to ask for additional funding and to explain why the project is “late”.

Q. Where do most ERP implementations fail?

A. Failure is a relative term. Most projects go live and deliver acceptable return on investment but are often seen as challenging projects or having failed because of initial unrealistic expectations with respect to the level of effort required of the regulated company. As discussed during the webcast, it is important that regulated companies really understand the activities that they will be responsible for and the deliverables that they will have to produce.

These need to be resourced appropriately; funding needs to be available and realistic timescales need to be set. If realistic timescales were put in front of stakeholders at the beginning of a project far fewer projects would be considered to have ‘failed’. Key to this is involving experienced resources in the concept phase of the system life cycle and during the early stages of the project planning.

Such resources need to have experience of implementing and validating ERP (or CRM) systems in the life sciences industry and the experience and knowledge that they bring to the table is invaluable.

As ever, if anybody has any follow-up questions from the webcast they can comment on the blog will get in touch through a usual e-mail address life.sciences@businessdecision.com. If you missed the webcast and would still like to view it the recording is available here.

Risk-Based Validation - How Are We Doing?

Yesterday, I posted a poll on LinkedIn asking the question

"How is Life Sciences industry embracing risk-based computer systems validation? Do we have the balance between risk assessment and risk mitigation right? Are we just rushing to reduce costs...?" 

People's responses are starting to come in and we're already starting to get some interesting comments on the question. If you're interesting in the poll and want to take part you can find it here (once you've answered the question you'll be able to see how everybody else has been voting).

Since I asked the question I guess it's only reasonable that I try and answer it and provide my own opinion. There is of course no single answer and even within a single Regulated Company there are different individuals and departments taking different approaches to risk-based validation. In many cases you have the business, and IT tried to reduce costs against a background of life sciences companies struggling to maintain profitability. At the other extreme you have the quality unit who are used to doing things the old-fashioned way and like to see every 'i' dotted and 't' crossed.

Hopefully, somewhere in the middle, you'll find some pragmatic validation practitioners genuinely trying to do their best to take a risk-based approach to validation and apply the appropriate resources to the highest areas of risk to patient safety, product quality and data integrity.

However, this is often thwarted by two obstacles:

1. The pragmatic practitioners in the middle are seldom the people with control of the budgets or with the power to effect real change. This often means that they'll pull from pole to pole at the behest of the business owners, the IT group and the quality unit who have different perceptions of how things should be done. In many cases these practical practitioners of the validation art are not recognised as true subject matter experts and they guidance, advice - and even wisdom - is ignored, often for the sake of point scoring and political expediency in large organisations.

2. Where practical and experienced practitioners are not available (often in small to medium sized organisations but also in organisations in emerging economies where computerised systems validation is a relatively new discipline) there is a genuine lack of understanding of how to take a practical approach to risk-based validation. in these cases and inability to invest in training and education is an obstacle to adopting risk-based validation.

I think therefore the what we will see is an industry which continues to adopt risk-based validation at a relatively slow pace. While some companies are using risk-based validation as an excuse to do less (and in some cases, too little) I think the majority of companies want to do the right thing but are hampered because of their own organisational structures or lack of experience, training and education.


From my experience as a consultant, and with my experience working with the small number of companies who are doing a good job with risk-based validation I genuinely do believe that taking a risk-based approach to validation has advantages both in terms of cost effectiveness but also in mitigating risk to patients. Regrettably, it's one of those things that most regulated companies will have to see for themselves before they believe, and they were set for themselves until they have gained the necessary experience.


As an industry I believe that we need to do a better job in publishing case studies demonstrating the effectiveness of risk-based validation, that we need to recognise the value of our internal and external subject matter expertsand he must be prepared to invest in training and education.

Cost Effective Validation of ERP and CRM - Your Questions Asnwered

In yesterday's webcast "Dissecting ERP and CRM Vendors for Cost Effective Validation" we discussed what regulated companies should be looking for when selecting ERP or CRM software and system integrators in the Life Sciences industry.

Specifically, we looked at how the choice of software vendor and system integrator affects the validation of the system, in terms of:

• Cost effective and efficient validation,
• Level of compliance achieved, both in terms of compliant business processes and compliant validation,
• The quality of the final solution delivered.

We also reviewed what regulated companies can do to ensure that projects obtain the right balance between project cost, timescales and quality.

Unfortunately we ran out of time to answers all of the questions submitted during the webcast and our apologies to the person whose question we didn't have time to get around to. As promised, here is our answers to your question.

Q. How current is your data on cost, time, and quality?

A. In the webcast we showed a slide that inferred that time and cost considerations are usually completely divorced from quality/validation, where of course they should be balanced. Our experience is that a small number of Life Sciences companies (mainly larger pharmaceuticals) have invested significantly in adopting GAMP 5 risk-based validation and have developed flexible SDLCs which can accommodate ERP/CRM implementation in a cost effective and efficient manner.

However, this is still a small minority and as the later vote showed, most companies are still struggling to achieve the proper balance between time, cost and quality.

We also mentioned that it is possible to reduce the cost of validation to around 2-3% of the overall project budget, but most projects are still around 10-15%. The 2-3% figure assumes a well-defined SDLC that is specific to the ERP/CRM system being validated and also a great deal of process repeatability. In the case of Business & Decision, we can achieve those figures because (a) with due modesty, we are experts in implementing ERP/CRM and are also experts in risk-based validation and (b) we have done this dozens and dozens of times before in Life Sciences.

Some large pharmaceutical companies are also quoting similar single digit figures for the cost of validation, but the reality is that they are being selective in the figures they are quoting i.e. they represent mature processes for validating the roll-out of new phases of an existing system, using a system specific, mature and well-understood SDLC.

For most companies implementing ERP or CRM for the first time, 10-15% is more realistic. 5-10% can be achieved if you engage a specialist who really knows ERP and CRM validation and you engage them early in the project planning. 10-15% is more likely if your own validation staff work it out with a less experienced system integrator.

These figure are based on experience over the last 3-5 years, since the publication of GAMP 5.

You can view the webcast and hear all the other questions - and answers - by viewing the recording of the webcast. We hope that people found the webcast useful and that you’ll be able to join us for the remaining webcasts in the ERP/CRM series – details of which can be found in the ‘webcasts’ page on the Business & Decision Life Sciences website.

Computer Systems in China - to Validate or Not?

At today's Computer System Compliance session at the ISPE-CCPIE conference in Beijing we considered the new draft SFDA GMP regulations and the requirements (or not) to validate computer systems.

As blogged yesterday, the Chinese State FDA (SFDA) are committed to improving compliance, but the current draft of the new Chinese GMP regulations are ambiguous with respect to computer systems validation.

While certain Articles and Annexes infer a need to validate, calibrate etc certain control and monitoring systems there is no equivalent to US 21 CFR211 subpart 68 or PIC/S / EU Annex 11, which states a clear requirement to validate computer systems.

Some would claim that this is deliberate ambiguity on the part of SFDA and that it suits them to provide some 'wriggle room' for Chinese manufactures, but more likely is that computerized systems have been of relatively low priority. Re-writing and updating the GMP regulations is a significant undertaking and it is only reasonable that focus is given to basic GMP, especially when China is only just catching up with more developed countries with respect to the use of computer systems and there is no history and significant risk to patients as a result of failure to validate a computer system.

However, this ambiguity does not meet the stated aim of SFDA, which is to generally align national GMP regulations with World Health Organizations (WHO) guidelines. WHO guidelines (WHO Technical Report Series 937, 2006: WHO Expert Committee on Specifications for Pharmaceutical Preparations, Appendix 5) clearly have a requirement to validate computer systems, and this does appear to be missing from the current draft of the SFDA regulations.

There is also a view that SFDA want to limit the costs they are imposing on their local manufacturers at a time when the Chinese government is looking to reduce the cost of drugs and devices while at the same time as it is looking to provide healthcare to 900 million people.

However, if SFDA is serious about building a risk-based approach into the new GMP regulations it is perfectly feasible to include a clear requirement to validate computer systems while leveraging a cost effective risk-based approach which will limit the costs involved in computer systems validation according to risk.

Unless this is included in the new regulations Chinese companies face the prospect of a two tier approach to computer system validation depending on whether products are intended for the domestic or export market. This would be confusing, limit flexible operations and potentially cause problems where Enterprise Systems support domestic and export operations.

Let's hope that SFDA take on-board the need to align with international regulations in this regard and revise the current draft regulations to provide the clarity that the Chinese Life Sciences industry is looking for with respect to computer system validation.

Answers to Webcast Questions - Testing Best Practices: 5 Years of the GAMP Good Practice Guide

The following answers are provided to questions submitted during the "Testing Best Practices: 5 Years of the GAMP Good Practice Guide" and which we did not have time to answer while we were live.


Can we thank you all for taking the time to submit such interesting questions.

Q. Retesting: What is your opinion on retesting requirements when infrastructure components are upgraded? i.e. O/S patches, database upgrades, web server upgrades
A. The GAMP "IT Infrastructure Control and Compliance" Good Practice Guide specifically addresses this question. In summary, this recommends a risk-based approach to the testing of infrastructure patches, upgrades etc. Based on risk severity, likelihood and detectability this may require little or no testing, will sometime require testing in a Test/QA instance or in some cases they may or should be rolled out to the Production environment (e.g. anti-virus updates). Remember - with a risk-based approach there is no 'one-size-fits-all' approach.
 

Q. No value add for independent review and oversight? Why not staff SQE's?
A. Assuming that 'SQE' is Software Quality Expert, we would agree that independent review by such SQE's does add value, specifically because they are experts in software and should understand software testing best practices. Where we do question the value of quality reviews (based on current gidance) is where the Quality Unit has no such expertise to draw upon. In these cases the independent Quality Unit still has a useful value add role to play, but this is an oversight role, ensuring that test processes and procedures are followed (by review of Test Strategies/Plans/Reports and/or periodic review or internal audit)

Q. What FDA guidance was being referred to re: QA review of test scripts etc not being necessary?
A. The FDA Final Guidance document “General Principles of Software Validation” doesn’t specifically state that QA review of test scripts is not necessary, but like the GAMP “Testing of GxP Systems“ Good Practice Guide, GAMP 5 and ASTM E2500, it places the emphasis on independent PEER review. i.e. by suitably qualified, trained or experienced peers (e.g. software developers, testers etc) who are able to independently review test cases. Although QA IT people may well have the necessary technical background to play a useful part in this process (guiding, supporting etc) this is not always the case for the independent Quality Unit who are primarily responsible for product (drug, medical device etc) quality.
 

Q. Do the regulators accept the concept of risk-based testing?
A. As we stated in response to a similar question in the webcast, regulatory authorities generally accept risk-based testing when it is done well. There is a concern amongst some regulators (US FDA and some European inspectors) that in some cases risk-assessments are being used to justify decisions that are actually taken based on timescale or cost constraints.
In the case of testing, the scope and rigor of testing is sometimes determined in advance and the risk assessment (risk criteria, weightings etc) are 'adjusted' to give the desired answer e.g. "Look - we don't need to do any negative case testing after al!"
The better informed regulators are aware of this issue, but where testing is generally risk-based our experience is that this is viewed positively by most inspectors.
 

Q. Do you think that there a difference in testing good practices in different sectors e.g pharma vs. medical device vs. biomedical?
A. There shouldn't be, but in reality the history of individual Divisions in the FDA (and European Agencies) means that there are certain hot topics in some sectors e.g.

• Because of well understood failures to perform regressions analysis and testing the CBER are very hot on this topic in blood banking.
• Because of the relatively high risk of software embedded in medical devices, some inspectors place a lot of focus on structural testing.

Although this shouldn't change the scope or rigor of the planned testing it is necessary that the testing is appropriate to the nature of the software and the risk, and that project documentation shows that valid regulatory concerns are addressed. It is therefore useful to be aware of sector specific issues, hot topics and terminology.

Q. Leaving GMP systems aside and referring to GxP for IT, Clinical and Regulatory applications. How do you handle a vendors minimum hardware spec for an application in a virtual environment?
We have found that vendors overstate the minimums (# of CPUs, CPU spec, minimum RAM, disk space usage, etc.) by a huge margin when comparing actual usage after a system is in place.
A large pharma I used to work for put a standard VM build of 512k RAM and to increase it if needed.  This was waived for additional  servers of the same application.   In the newest version of VMware (vSphere 4) all of these items can be changed while the guest server is running.
A. Software vendors do tend to cover themselves for 'worst case' (peak loading of simultaneous resource intensive tasks, maximum concurrent users etc - and then add a margin), to ensure that the performance of their software isn't a problem. The basic answer is to use your own experience based on a good Capacity Planning and Performance Management process (see the GAMP "IT Infrastructure Control and Compliance" Good Practice Guide again). This shoud tell you whether your hardware is over-rated or not and you can use historic data to size your hardware. It can also be useful to seek out the opinion of other users via user groups, discussion boards and forums etc.
Modern virtualization (which we also covered in a previous webcast "Qualification of Virtualized Environments") does allow the flexibility to modify capacity on the fly, but this isn't an option for Regulated Companies running in a traditional hardware environment. Some hardware vendors will allow you to install additional capacity and only pay for it when it is 'turned on' , but these tend to be large servers with mutliple processors etc.
At the end of the day it comes down to risk assessment - do you take the risk of not going with the software vendors recommendation for the sake of reducing the cost of the hardware? This is the usual issue of balancing project capex' budget against the cost to the business of poor performance.