Thanks once again to those of you who joined us for yesterday's webcast - the second stop on our "virtual book tour" which looked at practical risk management. We had a good number of questions asked as part of the registration process which we handled in yesterday's webcast (you can watch a recording of the webcast and download the slides here) but unfortunately we didn't have time to answer all of your questions that were asked during the session.
As usual, we taken the time to answer the outstanding questions here on a life sciences blog.
Q. Can you say more about regulators who are worried about misused risk assessments?
A. During the webcast we mentioned that the number of inspectors from European and US regulatory agencies had commented that they have concerns about the quality of risk assessments and the resulting validation. This comments have been made during informal discussions and in one case at a conference.
Their concern is that the resulting validation is either not broad enough in terms of scope or not rigorous enough in terms of depth and that this has been uncovered during inspection of what they believe to be relatively critical systems. In a couple of cases inspectors have commented that they believe that this is a case of the companies involved using risk assessment as an excuse to reduce the level of effort and resources applied in validating such systems.
We know from their comments that in a number of cases this has led to inspection observations and enforcement actions and it appears that a number of regulatory inspectors are in their words "wise to the trick". As we said in a webcast yesterday is important that the scope and rigour of any validation is appropriate to the system and the risk assessment is used to determine which areas and functions in the system require greater focus. The objective of risk-based validation is not to simply produce a level of effort and expenditure but ensure that the efforts and resources are applied most appropriately.
Q. How much time and effort can be saved by using the right risk assessment approach?
A. Our experience is that by using a relative risk assessment process rather than a quantitative risk assessment process it is possible to reduce the time and effort spent on assessing risks by between 50 to 75%. We have also studied the outputs of both types of risk assessment process on very similar systems and it is encouraging to note that in many cases both processes have provided very similar outputs in terms of the distribution of high, medium and low risk priorities both in terms of the relative number of each risk priority grouping and the functions allocated to each group.
This means that for enterprise systems with lower risk it is possible to reduce the time spent assessing risks by half or three quarters and still come up with results which are sufficiently accurate to support appropriate risk-based validation. This is why it is so important that regulated companies have a variety of risk management processes and tools available to them so they can use the most appropriate and cost-effective approach.
Q. When would you use a quantitative risk assessment approach? For what type of systems?
A. You would typically use a quantitative risk assessment approach where it is necessary to distinguish low, medium and high risk impact amongst a variety of requirements or functions that are all or are mostly of high GxP significance. In this case a quantitative (numeric) approach allows you to take a more granular view and again focus your verification activities on the requirements or functions which are of the highest risk impact.
Typically these will be systems which are safety critical and while this approach could be very useful in terms of manufacturing systems, in terms of enterprise systems we see this approach being used to the most critical systems such as adverse event systems (AES), LIMS systems used for product release, MES etc. Even with these systems quantitative risk assessment can be used on a selective basis for those modules which the initial risk assessment determines to be most critical.
Q. Who should conduct the risk assessment of EDMS system supporting the whole Enterprise?
A. Risk assessments cannot be conducted alone. This was a key points bought out in this week's GAMP UK meeting where we ran a risk assessment exercise and it was clearly valuable to have a variety of opinions and experience feeding into the process. You need people who understand the requirements, the business processes and the resulting risks to give their expertise with respect to risk impact.
You also need technical subject matter experts from the engineering or IT group who are much more likely to understand the risk likelihood. Both groups can contribute to thinking about risk detectability, either in terms of detecting risks within the system or as part of the normal business process checks.
It is therefore very important to invite the right people with the right breadth and depth of knowledge to any risk assessment exercise and to allow sufficient time for the relevant risk scenarios to be identified and assessed.
Thank you as ever for your interesting questions - we hope we find the answers above useful. Remember that you can join us on 17th October when will be looking at the very thorny issue of validating enterprise systems in the Cloud as Software-as-a-Service (registration is free and is open here)
Showing posts with label Risk Management. Show all posts
Showing posts with label Risk Management. Show all posts
Thursday, October 4, 2012
Friday, March 30, 2012
Computer System Validation Policy on Software-as-a-Service (SaaS)
In a recent LinkedIn Group discussion (Computerized Systems Validation Group: Discussion "Validation of Cloud), the topic of
Software-as-a-Service (SaaS) was widely discussed and the need to identify appropriate
controls in Computer System Validation (CSV) policies was discussed.
The reality is that relatively few compliant, validated SaaS
solutions are out there, and relatively few Life Sciences companies have CSV
policies that address this.
However, there are a few CSV policies that I’ve worked on that address this and
although client confidentiality means that I can’t share the documents, I did
volunteer to publish some content on what could be included in a CSV policy
to address SaaS.
Based on the assumption that any CSV policy leveraging a
risk-based approach needs to provide a flexible framework which is instantiated
on a project specific basis in the Validation (Master) Plan, I've provided some notes below (in italics) which may be useful in providing policy guidance. These would need to be incorporated in a CSV Policy using appropriate language (some Regulated Company's CSV Policy's are more prescriptive that others and the language should reflect this).
"When the use of Software-as-a-Service (SaaS) is considered,
additional risks should be identified and accounted for in the risk assessment and in the development of the Validation Plan computer system validation
approach. These are in addition to the issues that need to be considered with
any third party service provider (e.g. general hosting and managed services).
These include:
- How much control the Regulated Company has over the configuration of the application, to meet their specific regulatory or business needs (by definition, SaaS applications provide the Regulated Company (Consumer) with little or no control over the application configuration)
o
How does the Provider communicate application
changes to the Regulated Company, where the Regulated Company has no direct
control of the application?
o
What if Provider controlled changes mean that
the application no longer complies with regulatory requirements?
- The ability/willingness (or otherwise) of the Provider to support compliance audits
- As part of the validation process, whether or not the Regulated Company can effectively test or otherwise verify that their regulatory requirements have been fulfilled
o
Does the Provider provide a separate
Test/QA/Validation Instance?
o
Whether it is practical to test in the
Production instance prior to Production use (can such test records be clearly
differentiated from production records, by time or unique identification)
o
Can the functioning of the SaaS application be
verified against User Requirements as part of the vendor/package selection
process? (prior to contract - applicable to higher risk applications)
o
Can the functioning of the SaaS application be
verified against User Requirements once in production use? (after the control -
may be acceptable for lower risk applications)
- Whether or not the Provider applies applications changes directly to the Production instance, or whether they are tested in a separate Test/QA Instance
- Security and data integrity risks associated with the use of a multi-tenanted SaaS application (i.e. one that is also used by other users of the system), including
o
Whether or not different companies data is
contained in the same database, or the same database tables
o
The security controls that are implemented
within the SaaS application and/or database, to ensure that companies cannot
read/write delete other companies data
- Where appropriate, whether or not copies of only the Regulated Companies data can be provided to regulatory authorities, in accordance with regulatory requirements (e.g. 21CFR Part 11)
- Where appropriate, whether or not the Regulated Companies data can be archived
- If it is likely that the SaaS application is de-clouded (brought in-house or moved to another Provider)
o
Can the Regulated Companies data be extracted
from the SaaS application?
o
Can the Regulated Companies data be deleted in
the original SaaS application?
If these issues cannot be adequately addressed (and risks
mitigated), alternative options may be considered. These may include:
- Acquiring similar software from an acceptable SaaS Provider,
- Provisioning the same software as a Private Cloud, single tenancy application (if allowed by the Provider)
- Managing a similar application (under the direct control of the Regulated Company), deployed on a Platform-as-a-Service (PaaS)"
Tuesday, September 20, 2011
GAMP® Conference: Cost-Effective Compliance – Practical Solutions for Computerised Systems
A very interesting and useful conference held here in Brussels over the past two days, with a focus on achieving IS compliance in a cost effective and pragmatic way. It's good to see ISPE / GAMP® moving past the basics and getting into some more advanced explorarations of how to apply risk-based approaches to projects and also the operational phase of the system life cycle.
There was understandably a lot of discussion and highlighting of the new Annex 11 (Computerised Systems), with many of the presenters tying their topics back to the new guidance document, which has now been in effect for just two and a half months.
One of the most interesting sessions was when Audny Stenbråten, a Pharmaceutical Inspector of the Norwegian Regulator (Statens Legemiddelverk) provided a perspective of Annex 11 from the point of view of the regulator. It was good to see an open approach to the use of pragmatic risk-based solutions, but as was highlighted throughout the conference, risk-based approaches require a well-documented rationale.
Chris Reid of Integrity Solutions presented a very good session on Managing Suppliers and Service Providers and Tim Goossens of MSD outlined how his company is currently approaching Annex 11.
Siôn Wyn, of Conformity, provided an update on 21 CFR Part 11, which was really ‘no change’. The FDA are continuing with their add-on Part 11 inspections for the foreseeable future, with no planned end date and no defined plans on how to address updates or any changes to Part 11.
On the second day, after yours truly presented some case studies on practical risk management in the Business & Decision Life Sciences CRO and our qualified data center, Jürgen Schmitz of Novartis Vaccines and Diagnostics presented an interesting session on how IT is embedded into their major projects.
Mick Symonds of Atos Origin presented on Business Continuity in what I thought was an informative and highly entertaining presentation, but which was non-industry specific and was just a little too commercial for my liking.
Yves Samson (Kereon AG) and Chris Reid led some useful workshops looking at the broader impacts of IT Change Control and the scope, and scalability of Periodic Evaluations. These were good, interactive sessions and I’m sure that everyone benefitted from the interaction and discussion.
In the final afternoon René Van Opstal, (Van Opstal Consulting) gave an interesting presentation on aligning project management and validation and Rob Stephenson (Rob Stephenson Consultancy) presented a case study on Decommissioning which, although it had previously been presented at a GAMP UK meeting, was well worth airing to a wider audience.
All in all it was a good couple of days with some useful sessions, living up to its billing as suitable for intermediate to advanced attendees. On the basis of this session I’d certainly recommend similar sessions to those responsible for IS Compliance in either a QA or IT role and I’m looking forward to the next GAMP UK meeting, and to presenting at the ISPE UK AGM meeting and also the ISPE Global AGM meeting later in the year.
There was understandably a lot of discussion and highlighting of the new Annex 11 (Computerised Systems), with many of the presenters tying their topics back to the new guidance document, which has now been in effect for just two and a half months.
One of the most interesting sessions was when Audny Stenbråten, a Pharmaceutical Inspector of the Norwegian Regulator (Statens Legemiddelverk) provided a perspective of Annex 11 from the point of view of the regulator. It was good to see an open approach to the use of pragmatic risk-based solutions, but as was highlighted throughout the conference, risk-based approaches require a well-documented rationale.
Chris Reid of Integrity Solutions presented a very good session on Managing Suppliers and Service Providers and Tim Goossens of MSD outlined how his company is currently approaching Annex 11.
Siôn Wyn, of Conformity, provided an update on 21 CFR Part 11, which was really ‘no change’. The FDA are continuing with their add-on Part 11 inspections for the foreseeable future, with no planned end date and no defined plans on how to address updates or any changes to Part 11.
On the second day, after yours truly presented some case studies on practical risk management in the Business & Decision Life Sciences CRO and our qualified data center, Jürgen Schmitz of Novartis Vaccines and Diagnostics presented an interesting session on how IT is embedded into their major projects.
Mick Symonds of Atos Origin presented on Business Continuity in what I thought was an informative and highly entertaining presentation, but which was non-industry specific and was just a little too commercial for my liking.
Yves Samson (Kereon AG) and Chris Reid led some useful workshops looking at the broader impacts of IT Change Control and the scope, and scalability of Periodic Evaluations. These were good, interactive sessions and I’m sure that everyone benefitted from the interaction and discussion.
In the final afternoon René Van Opstal, (Van Opstal Consulting) gave an interesting presentation on aligning project management and validation and Rob Stephenson (Rob Stephenson Consultancy) presented a case study on Decommissioning which, although it had previously been presented at a GAMP UK meeting, was well worth airing to a wider audience.
All in all it was a good couple of days with some useful sessions, living up to its billing as suitable for intermediate to advanced attendees. On the basis of this session I’d certainly recommend similar sessions to those responsible for IS Compliance in either a QA or IT role and I’m looking forward to the next GAMP UK meeting, and to presenting at the ISPE UK AGM meeting and also the ISPE Global AGM meeting later in the year.
Thursday, April 22, 2010
Computer System Validation – Business as Usual?
A colleague asked me earlier today what were the big issues at the moment in computer system validation – and I couldn’t really think of any.
After more than twenty years introducing computer system validation to a lot of companies, consulting on Part 11, getting ready for Y2K, responding to Part 11, addressing infrastructure qualification and adopting a risk-based approach to validation the question is very much ‘where next?’.
To some extent it depends on what happens with risk-based validation. As the results from our webcast polls show, many Life Sciences organisations are still struggling to adopt a justifiable risk-based and cost effective approach to computer system validation.
At the moment it does appear to be business as usual – we still see computer system validation issues cited in FDA Warning Letters (and anecdotally reported by other regulatory agencies) but its justified and at a reasonable level in comparison to other more pressing topics – very much what we were used to around a decade ago.
However, if companies continue to use taking a risk-based approach as an excuse for simply doing less – rather than providing a real risk-based rationale for shifting resources to areas of the greatest risk – things may change. Some regulatory agencies have already commented that they are getting wise to ‘risk-based’ equating to ‘simply doing less’ and companies simply adopting GAMP® 5 as a flag of convenience for reducing spending on computer system validation without any clear rationale for doing less. Some inspectors have warned that they will take enforcement actions unless there is a clear and sound risk-based rationale for reducing the level of validation. Efficiency savings are fine, but only when the same goals are met.
There is then a possibility that we could see an increase in enforcement actions in response to Life Sciences companies taking the cost savings too far, but hopefully common sense will prevail as more individuals and organisations really start understand how to achieve the same objectives with less time and effort.
That leaves us with the other ‘big issue’ – which is how the industry is looking to changes in IT - such as cloud computing, virtualization, outsourcing and the like – and wondering how to apply risk-based principles to new technology and different business models.
While many Life Sciences companies are still relatively slow to change others are quietly moving ahead and the immediate future is probably one of evolution and not revolution. That’s not to say however that such evolution isn’t exciting – there is great potential to leverage newer technologies and models to deliver enhanced business performance, reduce costs and help restore the bottom line. If we can seize these opportunities and also address the compliance and validation issues in a cost effective manner then we’re in for a very interesting time – even if it’s not quite as exciting as when the regulators were giving everyone a hard time.
After more than twenty years introducing computer system validation to a lot of companies, consulting on Part 11, getting ready for Y2K, responding to Part 11, addressing infrastructure qualification and adopting a risk-based approach to validation the question is very much ‘where next?’.
To some extent it depends on what happens with risk-based validation. As the results from our webcast polls show, many Life Sciences organisations are still struggling to adopt a justifiable risk-based and cost effective approach to computer system validation.
At the moment it does appear to be business as usual – we still see computer system validation issues cited in FDA Warning Letters (and anecdotally reported by other regulatory agencies) but its justified and at a reasonable level in comparison to other more pressing topics – very much what we were used to around a decade ago.
However, if companies continue to use taking a risk-based approach as an excuse for simply doing less – rather than providing a real risk-based rationale for shifting resources to areas of the greatest risk – things may change. Some regulatory agencies have already commented that they are getting wise to ‘risk-based’ equating to ‘simply doing less’ and companies simply adopting GAMP® 5 as a flag of convenience for reducing spending on computer system validation without any clear rationale for doing less. Some inspectors have warned that they will take enforcement actions unless there is a clear and sound risk-based rationale for reducing the level of validation. Efficiency savings are fine, but only when the same goals are met.
There is then a possibility that we could see an increase in enforcement actions in response to Life Sciences companies taking the cost savings too far, but hopefully common sense will prevail as more individuals and organisations really start understand how to achieve the same objectives with less time and effort.
That leaves us with the other ‘big issue’ – which is how the industry is looking to changes in IT - such as cloud computing, virtualization, outsourcing and the like – and wondering how to apply risk-based principles to new technology and different business models.
While many Life Sciences companies are still relatively slow to change others are quietly moving ahead and the immediate future is probably one of evolution and not revolution. That’s not to say however that such evolution isn’t exciting – there is great potential to leverage newer technologies and models to deliver enhanced business performance, reduce costs and help restore the bottom line. If we can seize these opportunities and also address the compliance and validation issues in a cost effective manner then we’re in for a very interesting time – even if it’s not quite as exciting as when the regulators were giving everyone a hard time.
Wednesday, February 24, 2010
Answers to Webcast Questions - Leveraging ICH Q9 / ISO 14971 in Support of IS Compliance
Thanks to everyone who attended the webcast "Leveraging ICH Q9 / ISO 14971 in Support of IS Compliance" and who submitted questions. The recording is now on-line and subscribers can download the slides from the Business & Decision website as usual.
Listed below are the questions that we didn't have time for in the live webcast, along with the answers we promised to provide.
Q. Do you find that IT teams want to take the time to conduct proper risk assessments?
A. It all depends on the risk assessment process and model, whether it is scaled appropriately to the project / system and how well trained the IT team is. Assessing the risk severity is best left to the quality / regulatory and business subject matter experts, leaving the IT staff to think about technical risk scenarios and the risk likelihood and detectability.
Most professional IT staff evaluate and mitigate risk on an automatic basis, at least as far as the technology is concerned. For example, if it’s a critical business system the IT team will usually suggest redundant discs or mirroring to a DR site as a matter of course. In many cases you need them to reverse engineer their logic and document the rationale for their decisions using appropriately scaled tools and templates.
If you can make it clear to the IT staff that their expertise is valued and respected, that we just want them to rationalize and document their decisions with a process that isn’t too onerous we usually find that there is good buy-in
Q. Why do all your risk diagrams or maps make a low impact/high probability event equivalent to a high impact/low probability event....surely this is both misleading and dangerous.
A. They’re not our diagrams and maps – they are from the GAMP® Guide or GAMP® Good Practice Guides. Using the GAMP® risk assessment model gives Risk Class 2 for both high severity/low likelihood and low severity/high likelihood.
Equating severity and likelihood in the way wouldn’t be wise and could possibly increase the possibility of an unacceptable risk being seen as acceptable when considering the hazards associated with a medical device or risk to a patient through the use of a new drug. However, GAMP® attempts to provide a relatively simple risk assessment model which is cost effective when used in the implementation of computerized systems.
What wasn’t shown in the project example included in this webcast were the specific criteria used to qualitatively assess risk severity and risk likelihood, and which erred in the side of caution for this relatively high risk project/system.
Q. Can you comment on how pressure testing a system can provide data on probability of failure?
A. Assuming that ‘pressure testing’ relates to the stress testing of software rather than the pressure testing of a process vessel, it can only provide a limited set of data on the probability of failure. Because software does not change over time (assuming effective change control and configuration management processes) stress testing has little value in terms of the software functionality. Boundary, structural (path & branch) and negative case testing has more value here and should provide data on the failure modes of the software rather than the probability of failure.
Where stress testing can be useful is in looking at the probability of failure of the infrastructure i.e. network constraints, CPU capacity, storage speed and capacity. Stress testing can provide not only a useful idea of the probability of failure, but should allow users to identify the circumstances (loading) that lead to a particular failure mode and then define sensible limits which should not be exceeded.
Q. Do you think that proper selection of risk analysis technique (like DFMEA, FTA) greatly improves risk management of medical device companies?
A. Yes, absolutely. Both ICH Q9 and ISO 14971 talk about the appropriate selection of appropriate risk assessment models and tools and ICH Q9 Annex I provides a useful discussion on this topic.
Thanks again to everyone who joined us for the webcast and we look forward to catching up for the next webcasts.
Listed below are the questions that we didn't have time for in the live webcast, along with the answers we promised to provide.
Q. Do you find that IT teams want to take the time to conduct proper risk assessments?
A. It all depends on the risk assessment process and model, whether it is scaled appropriately to the project / system and how well trained the IT team is. Assessing the risk severity is best left to the quality / regulatory and business subject matter experts, leaving the IT staff to think about technical risk scenarios and the risk likelihood and detectability.
Most professional IT staff evaluate and mitigate risk on an automatic basis, at least as far as the technology is concerned. For example, if it’s a critical business system the IT team will usually suggest redundant discs or mirroring to a DR site as a matter of course. In many cases you need them to reverse engineer their logic and document the rationale for their decisions using appropriately scaled tools and templates.
If you can make it clear to the IT staff that their expertise is valued and respected, that we just want them to rationalize and document their decisions with a process that isn’t too onerous we usually find that there is good buy-in
Q. Why do all your risk diagrams or maps make a low impact/high probability event equivalent to a high impact/low probability event....surely this is both misleading and dangerous.
A. They’re not our diagrams and maps – they are from the GAMP® Guide or GAMP® Good Practice Guides. Using the GAMP® risk assessment model gives Risk Class 2 for both high severity/low likelihood and low severity/high likelihood.
Equating severity and likelihood in the way wouldn’t be wise and could possibly increase the possibility of an unacceptable risk being seen as acceptable when considering the hazards associated with a medical device or risk to a patient through the use of a new drug. However, GAMP® attempts to provide a relatively simple risk assessment model which is cost effective when used in the implementation of computerized systems.
What wasn’t shown in the project example included in this webcast were the specific criteria used to qualitatively assess risk severity and risk likelihood, and which erred in the side of caution for this relatively high risk project/system.
Q. Can you comment on how pressure testing a system can provide data on probability of failure?
A. Assuming that ‘pressure testing’ relates to the stress testing of software rather than the pressure testing of a process vessel, it can only provide a limited set of data on the probability of failure. Because software does not change over time (assuming effective change control and configuration management processes) stress testing has little value in terms of the software functionality. Boundary, structural (path & branch) and negative case testing has more value here and should provide data on the failure modes of the software rather than the probability of failure.
Where stress testing can be useful is in looking at the probability of failure of the infrastructure i.e. network constraints, CPU capacity, storage speed and capacity. Stress testing can provide not only a useful idea of the probability of failure, but should allow users to identify the circumstances (loading) that lead to a particular failure mode and then define sensible limits which should not be exceeded.
Q. Do you think that proper selection of risk analysis technique (like DFMEA, FTA) greatly improves risk management of medical device companies?
A. Yes, absolutely. Both ICH Q9 and ISO 14971 talk about the appropriate selection of appropriate risk assessment models and tools and ICH Q9 Annex I provides a useful discussion on this topic.
Thanks again to everyone who joined us for the webcast and we look forward to catching up for the next webcasts.
Subscribe to:
Posts (Atom)
Posts
