Measuring the Value of Validation?

An interesting question today on measuring the value of validation and assigning resources accordingly - something we see very few Life Sciences organizations doing well.

Q. How do others measure the value that validation creates within an organisation?

Does anyone have any experience of assigning value to validation activities?

I'm interested in how others may allocate resource and how within the validation planning process limits are/can be defined in terms of the cost of man hours against the economic benefits created from validation?

 

A. We have a process as part of our 'Lean IS Compliance' programme to put KPIs in place and measure cost and compliance levels (http://www.businessdecision-lifesciences.com/1170-lean-is-compliance.htm if anyone is interested). The challenge with measuring the value of validation is that it's difficult to compare projects with and without validation.

 

Most projects are different and many companies only see the cost of validation and not the value.

However, it can be done when you are e.g. rolling our an ERP system within a wider organisation and some business units need the system validating (APIs) and other business units do not (bulk chemicals).

 

Even in these cases most organisations only measure the cost and see validation as a 'negative', but if you are clever (which is what Lean IS Compliance is all about) you can also measure the value in terms of hard metrics (time and cost to fix software defects that make it into production) as well as soft metrics (user satisfaction when the system does - or does not - work right first time).

 

However, these are general benefits which accrue to any software quality process.

 

Although the principle of risk-based validation is to assign greater resources to systems and functions of greater risk, most companies again see this as an opportunity to reduces the level of resources assigned to lower risk systems/functions and the focus is again not on benefits.

 

It is possible to look at the implementation of systems of comparable size/complexity, where one system is high risk (and has more validation resources/focus) and another system has low/no risk (and few/no validation resources). Our work clearly shows that the higher risk systems do indeed go into production with fewer software issues and that this does have operational benefit (hard and soft benefits).

 

However, few companies track and recognise this and cannot correlate the investment in validation (quality) with the operational benefits. This is often an accounting issue, because the costs are usually capital expenditure and the benefits are associated with (lower) operational expenditure.

 

To really pull together a programme like Lean IS Compliance needs:

• Quality/Regulatory to truly accept a risk based approach (and that enough is enough)
• IT to understand the value of software quality activities (to which formal validation adds a veneer of documentation)
• Accountants to be willing to look at ROI over longer timescales than is often the case.