Using non-ERES Compliant Business Intelligence Tools in Support of GxP Decision Making

Another interesting question here - this time posted in the LinkedIn 21 CFR Part 11 discussion group on the use of business intelligence tools to support GxP decision making,. However, the business intelligence tools are not 21 CFR Part 11 compliant!

Q. We are implementing a 'system' which comprises of replicating software from Oracle and a destination replicated Oracle database. The purpose of this 'system' will be for running Oracle and Cognos reports from this replicated database instead of the high transactional source database. This system will be accessed only by the DBA.

The replicating software is pretty much command line based which is basically to say it does not have a GUI. From the command prompt user interface of this software, we can define (enter commands) for the source and target database; define the number of processes to run; set filters for what data are to be replicated and the frequencey for replication.

We did an assessment and found that part 11 is applicable. The problem is that we cannot satisfy all the part 11 requirements.

Although we deal with e-records (store the data from the source system including setup/ configuration of replication process) how do we still justify that e-signatures, audit trail requirements or password aging/restrictions are not applicable and not supported by the replicating software?

A. We've just finished doing exactly this for another project where we set up a reporting database to reduce the load on the transactional system. Some of the reports from the data warehouse were GxP significant.

 

We also had similar issues consolidating medical device traceability data for the device history record from signed records in SAP to the SAP Business Warehouse (BW) where we lost the secure relationship between the signed records in the transactional database and the copy in BW.

 

It's all about clearly identifying what records and data are used for what purposes, documenting the rationale and design, developing or updating appropriate SOPs and training your users and DBAs accordingly.

 

The first thing to do is to clearly differentiate between the records that you will ultimately rely on for regulatory purposes versus the data that you use for reporting. We maintained (and clearly documented) that the signed records in the transactional system would always be the ultimate source of data for regulatory decision-making (i.e. product recall, CAPA investigations etc). Where applicable these were signed and could be verified and were still Part 11 compliant.

 

Our rationale was that the data warehouse (and associated GxP reports) were acting as a grand 'indexing system' which allowed us to find the key regulatory records much faster (which has to be good for patient safety). We would not use these reports for making regulatory critical decisions, but we did use these reports to be able to more quickly find the key records in the transactional database which we could then rely upon for regulatory decision-making. Under that rationale the records and signatures which was subject to Part 11 remained in the transactional database.

 

We made sure that SOPs for key processes such as product recall, CAPA investigation etc were updated to reflect the appropriate use of the reports and the signed transactional records. We were able to demonstrate that using the data warehouse reports was between 2 to 3 times faster than running queries and filters in the transactional system and we could find the records much faster. In our testing we were also able to demonstrate that we always found the correct record (more than 99.9% of the time) unless something had subsequently been changed in the transactional database (less than 0.1% of the time) and even where this was the case the discrepancy was obvious and would not lead to erroneous decision-making.
 

However, that's not to say that the GxP significant reports were not important. We still validated the software which replicated the databases, we qualified the underlying IT infrastructure and we validated the GxP significant reports.

 

We essentially had three categories of report:

• Non-GxP reports, which were not validated (this was the majority of reports)
•  GxP significant reports, which were not based upon signed copies of Part 11 records, but which were validated.
• GxP significant reports, which were based upon signed copies of Part 11 records help in the transactional system.These were validated and were clearly annotated to the effect that they should not be relied upon for regulatory decision-making and they also gave a reference to the signed, transactional record in the transactional database.

 

On both of these projects we had much better tools for the replication of the data. Since you're using Oracle databases we would recommend that you create (and maintained under change control/configuration management) PL SQL programs to replicate the data bases. This will significantly reduce the likelihood of human error, allow you to replicate the databases on a scheduled basis and make it much easier to validate replication process.

 

For more background (discussion paper and webcast) on the use of validated Business Intelligence tools for GxP decision making on our website at http://www.businessdecision-lifesciences.com/2297-operational-analytics-in-life-sciences.htm.

Scaleable Database Controls for Part 11 Compliance

A good question posted on-line in the 21CFRPart 11 Yahoo Group, going to the heart of security controls around Electronic Records

Q Consider an electronic record system utlizing a typical database where a User ID and Password combination are used as an electronic signature to sign the records.  Assume the front end application has sufficient controls to prevent manipulation of the signature by the users.

Without getting too technical, what types of controls and solutions have you folks seen in terms of ensuring compliance with the requirements of 11.70 in regards to access to the database on the back end, typically by a DBA? 

How do you ensure that the linkage from record to signature is secure enough in the database to prevent manipulation by the DBA via ordinary means (i.e. with standard functionality and tools)?

Is there a risk justification for allowing the DBA some ability to manipulate the signature linkage, or should it be prevented by all but extraordinary means (i.e. using non-sanctioned tools, hacking, etc.)?

A. Without getting too technical there are a variety of controls that can be used. Which is implemented depends on risk and should be decided based upon the outcome of a documented risk assessment.

The first step is to use digital rather than electronic signatures. At its simplest, this means calculating some sort of 'hash' (fancy checksum) which is calculated using the contents of both the record and signature components. If either the record or signature components are changed, any attempt to recalculate the hash will come up with a different answer and you will the know that something has changed.

It won't however tell you what was changed - for that you need to rely on audit trails which can be set up at either the applications level (looking at changes to the defined record) or at the database level, to identify any inadvertent or accidental changes at the database level.

There is still the chance that the DBA can turn off the audit trails, so with some databases (e.g. Oracle) you have a couple of other options. The first is to write the audit trails into a separate database schema to which the DBA does not have access. This can use something like Audit Vault technology, which means the audit trails are written to a separate database server. The second is to make the database tables 'virtual private' database tables, which means the DBA can't even see them, let alone access them.

There may be circumstances when the DBA needs to access the database to make changes at the data level e.g. data corruption, or an accidental deletion by a user. It is permissible to make corrections under these circumstances, but you need a strict change control procedure to record what is being fixed, why and when (i.e. to preserve the audit trail). In such circumstances you would typically issue the DBA with a 'Super DBA' user account and password to make the changes, and then change the password afterwards (obviously someone needs to be trusted at some point).

One important principle is the idea of 'motivation' - why would a DBA want to make such changes? Organisational controls should be in place (including training and sanctions) to prevent e.g. a DBA and a system user from working in collusion to falsify records. Clearly system users should not also be DBAs.

It is therefore possible to make your Part 11 records and signatures very, very secure, but the more secure they are the more complex and expensive the solution. That's where the risk assessment is important - to identfy what is appropriate for any given set of records.

Cloud Computing Comes of Age in Life Sciences

For a while now we have been saying the Cloud was coming of age in the Life Sciences industry.

Business & Decision,  along with a small number of other Providers have been providing Infrastructure-as-a-Service and Platform-as-a-Service for some time.

We also said that as far as Software-as-a-Service was concerned, we would see Life Sciences specialist vendors (e.g. LIMS, Quality Management, Learning Management etc) providing compliant Software-as-a-Service solutions - simply because they understand our industry both at the functional level and also at the regulatory level.

We are working with a number of such vendors to deploy their software on our Platform-as-a-Service solutions, leveraging virtualization to provision solutions that are inherently flexible, scalable and - perhaps just as importantly - compliant.

At the same time, we have just started to engineer our first compliant 'Cloud Anywhere' solutions - which allow us to deploy pre-engineered and pre-qualified Platforms (hardware, power, HVAC, storage, virtualization, operating systems, database servers and applications servers) anywhere in the world. This was an idea first developed with Oracle with their Exadata and Exalogic machines (for which Business & Decision developed standard Qualification Packs).

Based upon a wider and more affordable technology base ‘Cloud Anywhere’ allows Business & Decision to leverage our investment in our Quality Management System to provision compliant Private or Community Cloud solutions with the minimum of additional qualification activities. These can be installed on client sites, in third party data centres of in the data centres of our software partners.

As well as deploying the solution, these 'Cloud Anywhere' solutions also come complete with Managed Services from Business & Decision - meaning that clients, partners etc no longer need to worry about the management of the Platform. All of this is taken care of remotely by our own staff (with the exception of local power and network connections of course) and the solutions can also be engineered to automatically failover to a remote Disaster Recovery site.

In the last couple of years we have seen people asking "How long will it be before everything is in the Cloud?", but the reality is that this will never be case in Life Sciences. There will always be Life Sciences companies who need or want some infrastructure on their own sites (because of network latency issues or data integrity issues) and the reality is that we are moving towards a mixed-Model Cloud Environment.

We will see a mixture of non-clouded Infrastructure, Platforms and Software, and various Cloud models, including On-Premise & Off-Premise and Public & Private Clouds.

The coming of age of safe, secure multi-tenanted Software-as-a-Service and the availability of solutions such as 'Cloud Anywhere' means that Life Sciences companies now have the ability to mix'n'match their Cloud environments to meet their specific business needs - and address their regulatory compliance requirements.

It may not seem like it now, but in the next few years we will see these solutions move from leading-edge to mainstream and we will wonder what all the fuss about Cloud was for.

Answers to Webcast Questions - Using Compliant ERP E-Records in Support of Regulatory Compliance

In yesterday's webcast Using Compliant ERP E-Records in Support of Regulatory Compliance, there were a couple of technical questions around the use of E-Records in Oracle E-Business Suite that we didn't get time to answer.

Thanks to our colleagues at Oracle for supporting the webcast and their help in answering these questions.

Q. Are new Oracle E-Business E-Record enabled events being added to the 11.5.10 release or just Release 12?
A. New developments are focused on Oracle E-Business Suite Release 12 and most of the recent E-Record enabled events are part of the Release 12 functionality e.g. Manufacturing Execution System. Release 11.5.10 is entering the maintenance mode of its life cycle so although some Release 12 functionality was previously ported back to 11.5.10, do not expect much, if any, new functional development on 11.5.10 moving forward


Q. In a earlier Business & Decision webcast (Testing Best Practices: 5 Years of the GAMP Good Practice Guide), it was suggested to get testing documentation from the vendor. What can Oracle provide to help minimize our internal testing?
A. As we discussed on the E-Records webcast, Oracle E-Business Suite customers can access automated test scripts that will run against the E-Business Suite Vision data set from the Oracle support site (formerly MetaLink). Just log in and search on "Test Starter Kit".
For clients implementing Oracle E-Business Suite using Oracle Accelerators test scripts are also generated by the Oracle Accelerator tool and these are specific to the client's configured instance generated by the Oracle Accelerator tool (see webcast "Compliant ERP Implementation in the Regulated Life Sciences Industry" for more information).

Thanks to all of you for your questions and remember that you can submit questions at any time on validation@businessdecision.com or erp@businessdecision.com, or by following the 'Ask an Expert' links on the website

OpenWorld 2009 in Retrospect

Oracle OpenWorld is over for another year and the streets of San Francisco will be getting back to normal (even after Tuesday's record rainfall). After attending a number of sessions, demos and meetings during the week its worth asking what the 'take away' is for Life Sciences companies.

The first is the growing move of Oracle into the hardware space and Larry Ellison's clear challenge to IBM. IBM has worked closely with Oracle in the past but Oracle's acquisitions in the hardware space has now created a new competitive dynamic. For those at OpenWorld for the technology the promise of Sun and Oracle getting together and this years announcements around Exadata were creating considerable excitement for companies with large data warehouses. For those Life Sciences companies looking to build clinical data warehouses, sales and marketing analytics data marts etc the promise of improved performance at lower cost appeared to be quite compelling.

In the short term there will be considerable opportunities to leverage this in R&D and the promise of extending this to applications databases is also quite compelling. Once the Sun deal receives final regulatory approval it looks like we'll see the new Oracle changing the landscape here.

The other thing on the technology side was the breadth of technology and middleware now available from one vendor (Oracle's demo grounds seem to get bigger every year). There is still a way to go in terms of integrating all of this but Oracle appears to have made a promising start and a number of clients I spoke to were welcoming the move that Oracle is making in at least some (larger) accounts, to help Enterprise Architects navigate their way through this maze of technology.

From my perspective part of that challenge is to put that technology into an appropriate business context and Oracle will almost certainly continue to struggle to field specialists in both the technology and industry expertise. I've certainly known cases where Oracle's own pre-sales people have been aware of Life Sciences functionality that is embedded in some of the technology products (e.g. DICOM support and BLAST searches in the database) and I suspect that's where the focus on Partner Specialization will come into play and partners like Business & Decision will continue to help client's to understand how to plan and deploy appropriate technologies in support of business requirements (see Sunday's Blog).

On the application side it was a chance to see a number of new applications in the Life Sciences industry. This included a number of acquisitions (e.g. Argus from Relsys for pharmacovigilance, the latest PLM acquisitions for pharmaceuticals) as well as new products from Oracle such as Oracle Pedigree and Serialization Manager and industry specific flavors of broader applications e.g. Siebel CRM for Life Sciences.

Industry focus is clearly one key dimension for Oracle and although there will always be more to do (including development work with Partners) Oracle appears to be extending its lead over traditional rival SAP and relative newcomer Microsoft. While there will continue to be best of breed solutions, Oracles ability to integrate via Fusion Middleware and to continue to acquire will see Oracle expand the functional footprint in Life Sciences and other key industries.

On the application front it appears to be evolution rather than revolution and systems integration via middleware was getting a lot of attention. Life Sciences companies with any sort of Oracle Applications footprint will really need to get up to speed with Fusion Middleware if they are looking to integrate and automate business processes.

In summary, OpenWorld 2009 was perhaps a little quieter than last year but there was even more to see and discuss. Despite financial uncertainty over the past year the I world certainly hasn't stood still and Life Sciences companies looking to move forward with projects would certainly have found it an interesting and useful week.

Product Lifecycle Managament Integration in Life Sciences

Sitting in on some of the conference session yesterday, looking at interfacing Product Lifecycle Management (PLM) solutions, it was quite obvious that not all PLM solutions are created equal.

PLM is one area where Oracle has grown strong through acquisition, initially by acquiring Agile. This brings arguably the best PLM solution for medical devices companies - what many still know as the Agile A9 product. Oracle also had a process version but is mainly used in food and beverage and although there is some good process functionality it was not specialized enough to manage the drug development lifecycle.

Then in June this year Oracle announced its intention to purchase the IP of Conformia, bringing a strong product to manage the drug development lifecycle into their portfolio, with the intention to integrate this into the Agile product line.

This will provide large Life Sciences companies first rate products for either medical device development and design management or for managing the drug discovery and development process. This will also provide some flexibility to manage the lifecycle of products that are manufactured using either discrete or process manufacturing operations.

Although many take a simplistic view, the world isn't as simple as medical devices companies use discrete manufacturing and pharmaceuticals uses process. In some cases medical devices use batch manufacturing and pharmaceuticals need discrete. At the PLM level, the medical device Quality System Regulations have specific requirements for device development which Agile A9 meets very well, but not so Conformia.

Back to the topic of interfaces: While Oracle is doing a good job in developing standard process integrations between A9 and their core ERP systems (E-Business Suite and JDEdwards, using AIA PIPs), there is a growing realization that the combination of regulatory requirements for managing the various lifecycles of devices or drugs and discrete or process manufacturing means that there will need to be complete flexibility to interface either of the two 'flavors' of PLM to either discrete or process manufacturing solutions in Oracle's own (and competitors such as SAP) ERP systems.

Oracle's AIA framework certainly provides that opportunity and is no doubt the easiest way to develop such integrations. Indeed, Oracle has a roadmap to develop such Process Integration Packs, but it will be a while yet until the Conformia product is integrated into Agile and the AIA PIPs developed.

From a business perspective this means that Life Sciences companies looking to acquire PLM solutions (not only from Oracle) really need to understand:

• The nature of their product development lifecycle
• The regulatory requirements which govern their PLM processes (depending on device class or drug profile)
• The nature of their manufacturing process - discrete, process or possibly mixed mode for some product lines
• Exactly how they want to integrate their PLM systems to their manufacturing systems from a business process perspective

It is this that will drive the functional requirements of their interface and determine whether an out-of-the-box integration will deliver what is needed, or whether integration tools will be needed to develop some customer elements of the interface.

On a positive note, for Life Sciences customers who have taken the step to integrate their PLM and ERP systems and processes, there is a good return on investment both in terms of reducing costs and helping maintain compliance through the design, development and introduction of new products and new versions of products. With respect to this latter point, with regulatory authorities now looking at risks introducing during new product introduction (or scale up) it is good news that these concerns can be addressed at the same time these processes are made more efficient.

Labeling and Identification in Life Sciences – A Perfect Storm?

At least a couple of times today I heard the various regulations currently being drafted around the area of product labeling and identification as a ‘Perfect Storm’, driving Life Sciences companies to address these issues.

While Pedigree requirements have been deferred in the US, the US FDA continues to make progress with their UDI initiative that would require unique product identification for medical devices, at least to the level currently required for ethical drugs (requiring the National Drug Code, expiry date and batch number to be included in human and machine readable format) and blood products (where the international ISBT 128 standard is required).

At the same time, the European Union is slowly making progress on the issue of serialization, which is of course already a requirement in many countries for some classes of medical device.

While individual software solutions can help in these areas (see below), the implications of these requirements on Life Sciences companies may be enormous. While many Regulated Companies are seeing this as solely as an IT issue, the reality it is that in some cases complying with these regulations will require significant changes to supply chain models and significant investments in terms of upgrading facilities or even setting up complete new packaging, labeling and distribution centers.

While the business issues are enormous at least some of the software vendors are taking the issue seriously. Here at Oracle OpenWorld there are numerous vendors with labeling and printing solutions, but the most significant news from Oracle is the pending availability of their Oracle Pedigree and Serialization Manager (OPSM), which is set to be at the heart of many serialization and labeling solutions.

OPSM provides the ability to generate and manage serial numbers, interface to ERP and labeling/packaging systems and the ability to submit secure serial numbers to the proposed Government databases . Even better, OPSM is significantly ‘software service’ enabled which means that it is relatively easy to integrate with external systems (it is one of Oracle’s first ‘Fusion Applications’, having been developed on Oracle’s open standards based Fusion Middleware), with the majority of transactions running in the background (generating, tracking and submitting serial numbers, maintaining a database of packaging hierarchies etc).

The use of integrated Fusion Middleware based Business Intelligence means that users in the Life Sciences industry will be able to focus on exceptions (suspect serial numbers, potential counterfeited product, matching and authorizing returns etc) without having to manually interchange data between systems. While other serialization products exist, OPSM has the potential to be at the center of serialization business processes in pharmaceuticals and medical devices, uniquely integrating ERP systems, shop floor packaging and labeling systems and submission to the coming Government databases.

Based on the traffic at the OPSM stand in the OpenWorld exhibition hall there appears to be a good deal of interest in the product - there is certainly a need given current and pending regulations in this area. It will be interesting to see how many Regulated Companies really understand the nature of the coming “Perfect Storm” and translate their interest in OPSM in to real world solutions.

Oracle OpenWorld – Partner Specialization

I'm here in San Francisco at Oracle's OpenWorld event (Sunday 1th to Thursday 1th October).

The big message at the today’s Partner sessions was 'Specialized. Recognized by Oracle. Preferred by Customers.'
This means that under the new Partner program the value of more specialized partners will be more highly valued by Oracle, specialized both in terms of technology and industry.

This is reflecting a broader industry trend in IT as customers and software vendors alike start to really understand the value of specialists in terms of delivering business value, but in the Life Sciences industry the value of suppliers who understand the industry and the specific sector - pharmaceutical, medical device, biotech etc - has largely always been valued.

Oracle, like other major software vendors, is realizing that they can no longer maintain the breadth of technology and industry skills themselves and that they need more specialized Partners to get the best out of their software and deliver successful solutions.

For niche or 'boutique' players like Business & Decision this is good news - with just under 3,000 consultants we cannot compete with the biggest System Integrators - but with our focus on specific solutions within the Life Sciences sector a broader recognition of the value of Specialization is good news.

It should also be good news for prospective clients looking for confirmation that they are selecting the right supplier. When Oracle and other software vendors are really ready to recommend and stand behind specialized Partners (recommending the best and not just the biggest) it should make the job of supplier assessment so much easier.