#KPITuneUp

Is it time your Vendor/CRO KPIs had a tune up?

As the late, great Michael Hammer once said in The Seven Deadly Sins of Measurement, “…there is a widespread consensus that [companies] measure too much or too little, or the wrong things, and that in any event they don’t use their metrics effectively.” Hammer wrote this in 2007 and I suspect many would think it still rings true today. What are Hammer’s deadly sins?

Vanity – measuring something to make you look good. In a culture of fear, you want to make sure your KPIs are not going to cause a problem. So best to make sure they can’t! If you use KPIs to reward/punish then you’re likely to have some of these. The KPIs that are always green such as percent of key team member handovers with handover meetings. Maybe the annualized percent of key staff turnover might not be so green.
Provincialism – sub-optimising by focusing on what matters to you but not the overall goal. The classic example in clinical trials (which was in the draft of E8 R1 but was removed in the final version) is the race to First Participant In. Race to get the first one but then have a protocol amendment because the protocol was poorly designed in the rush. We should not encourage people into rushing to fail.
Narcissism – not measuring from the customer’s perspective. This is why it is important to consider the purpose of the KPI, what is the question you are trying to answer? If you want investigators to be paid on time, then measure the proportion of payments that are made accurately and on time. Don’t measure the average time from payment approved to payment made as a KPI.
Laziness – not giving it enough thought or effort. To select the right metrics, define them well, verify them, and empowering those using them to get most value from them needs critical thinking. And critical thinking needs time. It also needs people who know what they are doing. A KPI that is a simple count at month end of overdue actions is an example of this. What is it for? How important are the overdue actions? Maybe they are a tiny fraction of all actions or maybe they are most of them. Better to measure the proportion of actions being closed on time. This focuses on whether the process is performing as expected.
Pettiness – measuring only a small part of what matters. OK, so there was an average of only 2 findings per site audit in the last quarter. But how many site audits were there? How many of the findings were critical or major? Maybe one of the sites audited had 5 major findings and is the largest recruiting site for the study.
Inanity – measuring things that have a negative impact on behaviour. I have come across examples of trying to drive CRAs to submit Monitoring Visit Reports within 5 days of a monitoring visit leading to CRAs submitting blank reports so that they meet the timeline. It gets even worse if KPIs are used for reward or punishment – people will go out of their way to make sure they meet the KPI by any means possible. Rather than focus effort on improving the process and being innovative, they will put their effort into making sure the target is met at all costs.
Frivolity – not being serious about measurement. I have seen many organizations do this. They want KPIs because numbers gives an illusion of control. Any KPIs will do, as long as they look vaguely reasonable. And people guess at targets. But no time is spent on why KPIs are needed and how they are to be used. Let alone training people on the skills needed. Without this, KPIs are a waste of resource and effort.

I think Hammer’s list is a pretty good one and covers many of the problems I’ve seen with KPIs over the years.

How well do your KPIs work between you and your CRO/vendor? Does it take all the effort to gather them ready for the governance meeting only to have a cursory review before the next topic? Do you really use your KPIs to help achieve the overall goals of a relationship? Have you got the right ones? Do you and your staff know what they mean and how to use them?

Perhaps it’s time to tune up your KPIs and make sure they’re fit for 2023. Contact me and I’d be happy to discuss the approach you have now and whether it meets leading practice in the industry. I can even give your current KPI list a review and provide feedback. #KPITuneUp

Image – Robert Couse-Baker, PxHere (CC BY 2.0)