Process Improvement

#KPITuneUp

Is it time your Vendor/CRO KPIs had a tune up?

As the late, great Michael Hammer once said in The Seven Deadly Sins of Measurement, “…there is a widespread consensus that [companies] measure too much or too little, or the wrong things, and that in any event they don’t use their metrics effectively.” Hammer wrote this in 2007 and I suspect many would think it still rings true today. What are Hammer’s deadly sins?

Vanity – measuring something to make you look good. In a culture of fear, you want to make sure your KPIs are not going to cause a problem. So best to make sure they can’t! If you use KPIs to reward/punish then you’re likely to have some of these. The KPIs that are always green such as percent of key team member handovers with handover meetings. Maybe the annualized percent of key staff turnover might not be so green.
Provincialism – sub-optimising by focusing on what matters to you but not the overall goal. The classic example in clinical trials (which was in the draft of E8 R1 but was removed in the final version) is the race to First Participant In. Race to get the first one but then have a protocol amendment because the protocol was poorly designed in the rush. We should not encourage people into rushing to fail.
Narcissism – not measuring from the customer’s perspective. This is why it is important to consider the purpose of the KPI, what is the question you are trying to answer? If you want investigators to be paid on time, then measure the proportion of payments that are made accurately and on time. Don’t measure the average time from payment approved to payment made as a KPI.
Laziness – not giving it enough thought or effort. To select the right metrics, define them well, verify them, and empowering those using them to get most value from them needs critical thinking. And critical thinking needs time. It also needs people who know what they are doing. A KPI that is a simple count at month end of overdue actions is an example of this. What is it for? How important are the overdue actions? Maybe they are a tiny fraction of all actions or maybe they are most of them. Better to measure the proportion of actions being closed on time. This focuses on whether the process is performing as expected.
Pettiness – measuring only a small part of what matters. OK, so there was an average of only 2 findings per site audit in the last quarter. But how many site audits were there? How many of the findings were critical or major? Maybe one of the sites audited had 5 major findings and is the largest recruiting site for the study.
Inanity – measuring things that have a negative impact on behaviour. I have come across examples of trying to drive CRAs to submit Monitoring Visit Reports within 5 days of a monitoring visit leading to CRAs submitting blank reports so that they meet the timeline. It gets even worse if KPIs are used for reward or punishment – people will go out of their way to make sure they meet the KPI by any means possible. Rather than focus effort on improving the process and being innovative, they will put their effort into making sure the target is met at all costs.
Frivolity – not being serious about measurement. I have seen many organizations do this. They want KPIs because numbers gives an illusion of control. Any KPIs will do, as long as they look vaguely reasonable. And people guess at targets. But no time is spent on why KPIs are needed and how they are to be used. Let alone training people on the skills needed. Without this, KPIs are a waste of resource and effort.

I think Hammer’s list is a pretty good one and covers many of the problems I’ve seen with KPIs over the years.

How well do your KPIs work between you and your CRO/vendor? Does it take all the effort to gather them ready for the governance meeting only to have a cursory review before the next topic? Do you really use your KPIs to help achieve the overall goals of a relationship? Have you got the right ones? Do you and your staff know what they mean and how to use them?

Perhaps it’s time to tune up your KPIs and make sure they’re fit for 2023. Contact me and I’d be happy to discuss the approach you have now and whether it meets leading practice in the industry. I can even give your current KPI list a review and provide feedback. #KPITuneUp

Image – Robert Couse-Baker, PxHere (CC BY 2.0)

Knock, Knock! Who’s There?

Recently, in my street, a gas company was relining all the gas pipes under the road, and to each house. There was a safety problem. And it needed them to turn off gas supply to each house one by one as they worked on each of the delivery pipes. Unfortunately, the method of communication of when your gas needed to be turned off was by knocking on your door and talking to you. Perhaps during Covid lockdowns, this worked OK. But not now – they kept missing me, either because I was out or because I was on a work video conference. This led to frustrations on both sides and when they eventually got hold of me, they complained at how difficult I was to contact. I asked whether anyone had considered using a phone or perhaps dropping a message through the letterbox. “That’s a good suggestion!” the workman said. I pondered this for a while. This is a national company and they do this all the time. This “problem” must have come up before, surely? Why didn’t they have a standard process for contacting householders? They know all the addresses after all.

A really challenging area in process improvement is how to make changes stick? Processes invariably rely somewhere on people. And people get used to doing things the way they have always done them. So, to change that takes effort – change management. But after the improvement project is finished, what if people go back to doing things the way they always did? When thinking about the “Control” part of process improvement, you have to think carefully about how you can ensure the process changes stay in place. And how those responsible can get an early signal if they do not. If you don’t do this, the improvement may gradually be lost.

As I was leaving the house later that day, I bumped into the same workman. He told me to give them a call as soon as I was back so I could get the gas turned on again. He gave me a card with the phone number to call. The phone number was on a card which was designed to be dropped through someone’s letterbox if they were not at home. It had a space to enter the start and end date of the work and a phone number to contact. He had not thought to use it for its actual purpose! Presumably, at some point, someone had made a process improvement and introduced these cards, but the change had not stuck.

You might be wondering, in this example, how you could make sure the changes are permanent. Well, you could ask residents about the service and monitor the responses for the original issue repeating. You could audit the process. And you could monitor how many cards are reordered to give a signal as to whether they are being used at the expected rate.

Any changes you introduce to a process need to be effective. But if they work, you also want to make them permanent. Thinking about how you can make those changes stick is an important part of any process improvement project.

Image – Pavel Danilyuk, pexels.com

Don’t let metrics distract you from the end goal!

We all know the fable of the tortoise and the hare. The tortoise won the race by taking things at a steady pace and planning for the end rather than rushing and taking their eye off the end goal. Metrics and how they are used can drive the behaviours we want but also behaviours that mean people take their eye off the end goal. As is often said, what gets measured gets managed – and we all know metrics can influence behaviour. When metrics are well-designed and are focused on answering important questions, and there are targets making it clear to a team what is important, they can really help focus efforts. If the rejection rate for documents being submitted to the TMF is set to be no greater than 5% but is tracking well above, then there can be a focus of effort to try and understand why. Maybe there are particular errors such as missing signatures, or there is a particular document type that is regularly rejected. If a team can get to the root causes then they can implement solutions to improve the process and see the metric improve. That is good news – metrics can be used as a great tool to empower teams. Empowering them to understand how the process is performing and where to focus their effort for improvement. With an improved, more efficient process with fewer errors, the end goal of a contemporaneous, high quality, complete TMF is more likely to be achieved.

But what if metrics and their associated targets are used for reward or punishment? We see this happen with metrics when used for personal performance goals. People will focus on those metrics to make sure they meet the targets at almost any cost! If individuals are told they must meet a target of less than 5% for documents rejected when submitted to the TMF, they will meet it. But they may bend the process and add inefficiency in doing so. For example, they may decide only to submit the documents they know are going to be accepted and leave the others to be sorted out when they have more time. Or they may avoid submitting documents at all. Or perhaps they might ask a friend to review the documents first. Whatever the approach, it is likely it will impact the process of a smooth flow of documents into the TMF by causing bottlenecks. And they are being done ‘outside’ the documented process – sometimes termed the ‘hidden factory’. Now the measurement is measuring a process of which we no longer know all the details – it is different to the SOP. The process has not been improved, but rather made worse. And the more complex process is liable to lead to a TMF that is no longer contemporaneous and may be incomplete. But the metric has met its target. The rush to focus on the metric in exclusion to the end goal has made things worse.

And so, whilst it is good news that in the adopted ICH E8 R1, there is a section (3.3.1) encouraging “the establishment of a culture that supports open dialogue” and critical thinking, it is a shame that the following section in the draft did not make it into the final version:

“Choose quality measures and performance indicators that are aligned with a proactive approach to design. For example, an overemphasis on minimising the time to first patient enrolled may result in devoting too little time to identifying and preventing errors that matter through careful design.”

There is no mention of performance indicators in the final version or the rather good example of a metric that is likely to drive the wrong behaviour – time to first patient enrolled. What is the value in racing to get the first patient enrolled if the next patient isn’t enrolled for months? Or a protocol amendment ends up being delayed leading to an overall delay in completing the trial? More haste, less speed.

It can be true that what gets measured gets managed – but it will only be managed well when a team is truly empowered to own the metrics, the targets, and the understanding and improvement of the process. We have to move away from command and control to supporting and trusting teams to own their processes and associated metrics, and to make improvements where needed. We have to be brave enough to allow proper planning and risk assessment and control to take place before rushing to get to first patient. Let’s use metrics thoughtfully to help us on the journey and make sure we keep our focus on the end goal.

Image – openclipart.org

Is this a secret way to engage employees?

In circles of people discussing continuous improvement methodologies and approaches, there is often talk about the need to engage employees in these activities. That is of course true, after all, people doing the day-to-day work are most likely to know how to delight customers, cut unnecessary expenses, make processes faster, and so on. They understand the details and when given the right support (time, tools, expert help, encouragement etc), they can make huge improvements. So, we need to engage them in the activities, right?

Well yes. But I think that’s the wrong way round of looking at it. Being involved in continuous improvement and making processes better for you, your colleagues, and your customers can be enormously satisfying. I see this again, and again in improvement teams. Once a team is truly empowered, they love to take time to think more deeply about the processes they work in, and how they can modify them to work better. And to actually make a difference. I was reminded of this at the recent MCC summit. There was real excitement in the live discussions on how to understand and use metrics to improve processes. Indeed, I was contacted afterwards by someone who told me it had reignited his passion in this area and brought a real bright spot into what had become a dull job. Involving people in this way will also improve process ownership, buy-in for change and can improve the measurement of processes too.

According to Gallup’s State of the Global Workplace report 2021, only 20% of employees worldwide are engaged in their jobs – meaning that they are emotionally invested in committing their time, talent and energy in adding value to their team and advancing the organization’s initiatives. In Western Europe, engagement is at 11%!

Rather than thinking about how we can get employees engaged in continuous improvement, maybe we could get employees more engaged in their roles by supporting them to be involved in continuous improvement efforts? How can you free up your employees to improve processes and, at the same time, engage them more in their daily work?

Picture: RobinHiggins, Pixabay

You’re Solving the Wrong Problem!

The basic idea behind continuous process improvement is not difficult. It’s the idea of a cycle – defining the problem, investigating, determining actions to improve, implementing those actions, and then looking again to see if there has been improvement. It’s the Plan-Do-Check-Act cycle of Shewhart and Deming. Or the DMAIC cycle of Six Sigma. It’s a proven approach to continually improving. But it takes time and effort. It takes determination. And it can easily be derailed by those who say “Just get on with it!” Much better to be rushing into implementation to show how you are someone of action rather than someone who suffers from “paralysis by analysis.” But a greater danger is to move into actions without taking time to analyse properly – or even to define the problem. It looks great because you’re taking action. But what if your actions make things worse?

Let’s take the example of HS2 in the UK. This is the UK’s second high-speed railway line. The cost is enormous and keeps going up. Building is underway and billions have been spent already. The debate continues as to whether it is worth all the money. During one of the many consultations, in 2011, I wrote to give my perspective. I had read the proposal and was shocked to see there was no problem defined. Here was an expensive solution without a clear definition of the problem it was designed to resolve. It talked about trains being overcrowded currently. If that was the problem, then was this the best solution? I suggested they take that problem and drill down some more – when are the trains crowded? Where? Why? And so on. Then see if they could come up with solutions. Preferably ones that don’t cost tens of billions. If they are overcrowded during commuting times, I suggested that perhaps people could be given a tax incentive to work from home. Which would have the added advantage of being better for the environment.

Of course, since then, we’ve had the pandemic. And many have been working from home. Trains have not been overcrowded. And many have found they rather like working from home. So while the case for HS2 was flimsy 10 years ago, it’s become transparently thin since then. And because they didn’t spend time defining the problem or analysing it, there is no obvious route to go back and re-evaluate the decision. Given the change of circumstances, is it still the right thing to do? We can’t answer because we don’t know the problem it is trying to solve.

I do find it odd that so many organisations (governments included) rush into implementing changes without taking time to define the problem and analyse it. I suspect motives such as vanity – “let’s implement this new, shiny thing because it’ll make me look good”, and wanting to be seen as someone of action. Interesting that Taiichi Ohno, creator of the Toyota Production System, and Lean, used to get graduates to spend time just watching production. Afterwards, he would ask them what they saw and if he didn’t think they had observed enough he would get them to watch some more. Better to pause, observe, reflect, analyse than to go straight into actions that might actually make things worse.

For process improvement, make sure you understand the problem you’re trying to solve. Solving the wrong problem can be costly and wasteful!

Picture: pxhere.com

Why won’t they accept the change?

In process improvement, this question comes up again and again. When Management is convinced of the change, they sometimes view employees who doubt as being ‘change resistant’. These employees are seen as roadblocks to be overcome. As people who just want to keep doing things the way they’ve done them. Of course, there are people like this – but fewer than you’d think in my experience.

Why are people resistant to change after all? Maybe they have good reason. Maybe they can see flaws in the approach – they know the detail of the work after all. Maybe they think there is a better way to do it but no-one’s asked them. I’m simply not convinced that most people are resistant to change. Why is it that so many make New Year’s resolutions to get fit, lose weight, join a dating agency and so on? Because they want change. But they want change when they are in the driving seat. Not to be told what changes are going to happen to them. I spoke with someone once about the introduction of computers to the workplace. She was an experienced secretary and shorthand typist. She arrived at work one day to find her typewriter replaced by a computer. And her boss thought she’d be delighted. She was not. Was she ‘change resistant’? No – but she wanted to be involved in the change rather than have it done to her. So it is when the decision is made to automate a process without involving the people who do the day-to-day work in improving the process first.

One of the trickiest parts of process improvement efforts is not the complex techniques, or the statistics, it is implementing change that sticks. The secret is to involve the people doing the work. Something I have seen time and time again is the enthusiasm and ingenuity of those doing the work to actually improve what they do. People love to take time to understand their work better, with measurements if possible. And to come up with new ways of working. When those ways of working are implemented, they have a much better chance of sticking than the top-down ones from Management. These small teams are a delight to facilitate. With some guidance in process improvement, and the time and support, they can move mountains.

To get effective change, Management should set the direction and then support the employees to work out the best way to get there. This is true empowerment. And change resistance will melt away.

Picture: pxfuel.com

KPIs: What’s not to like?

Many organizations set Key Performance Indicators (KPIs) to monitor their performance against an overall goal or target. Makes sense, surely, to monitor progress with something tangible. And they can be very effective. But there are a lot of pitfalls. And I’m not convinced they work for all circumstances.

A major pitfall in implementing KPIs and targets is an overly top-down approach. Every department / function is told it must have a set of KPIs with targets. After all, this will ensure everyone is accountable. And there will be lots of data showing KPIs against targets Management to review. When these requests come through, most people just shrug their shoulders and mouth “here we go again,” or something less polite. They put together some KPIs with targets that will be easy to achieve and hope that will keep the Management quiet. After a bit of horse-trading, they agree slightly tougher targets and hope for the best.

Or even worse, Management wants to “hold their feet to the fire” and they impose KPIs and targets on each department. They require the cycle time of site activation to be reduced by 20%, or 20% more documents to be processed with the same resource, for example. This leads to much time spent on the definitions – what can be excluded, what should we include. How can we be ingenious and make sure the KPI meets the goal, regardless of the impact on anything else. We can, after all, work people much harder to do more in less time. But the longer-term consequences can be detrimental as burnout leads to sicknesses and resignations and loss of in-depth knowledge about the work.

This is an exercise in futility. It is disrespectful to the people working in the organization. It is wasting the time, ingenuity, and talent of those doing the work – those creating value in the organization. “The whole notion of targets is flawed. Their use in a hierarchical system engages people’s ingenuity in managing the numbers instead of improving their methods,” according to John Seddon in Freedom from Command & Control. Rather than understanding the work as a process and trying to improve it, they spend their time being ingenious about KPIs that will keep Management off their backs and making sure they meet the targets at whatever cost. There are plenty of examples of this and I’ve described two in past posts – COVID testing & Windrush.

Much better is for the team that owns the work to use metrics to understand that work. To set their own KPIs and goals based on their deep understanding. And to be supported by Management all the way in putting in the hard graft of process improvement. As W. Edwards Deming said, “There is no instant pudding!” Management should be there to support those doing the work, those adding value. They should set the framework and direction but truly empower their workforce to use metrics and KPIs to understand and improve performance. Longer term, that’s better for everyone.

Picture: KPI Board by Anna Sophie from the Noun Project

Pareto: Focus Your Efforts

For some of my work with the Metrics Champion Consortium, I was looking at MHRA inspection finding categories. MHRA publish reports on their findings – the most recent is for the year 2017-2018. For major findings, 86% are within just 21% of the categories. If this is representative of the industry, then focusing our improvement efforts on the processes associated with those 21% of categories could have a disproportionate impact on findings in the future. This fits the pattern of the Pareto Principle.

The Pareto Principle was proposed by Joseph Juran, a 20^th century pioneer of quality improvement. He based it on the observation of the economist Vilfredo Pareto of Italy who noted that 80% of Italy’s land was owned by 20% of the people. The principle is that in any given situation, roughly 80% of the effect is due to 20% of the causes. It seems to work well in many fields, for example:

- 20% of the most reported software bugs cause 80% of software crashes
- It is often claimed in business that 80% of the sales comes from 20% of the clients
- 20% of people account for 80% of all healthcare spending
- Even in COVID-19, 80% of deaths have occurred among 20% of the population (65 and older)

The principle is sometimes called the 80:20 rule or the law of the vital few because it implies that if you can focus on the 20% and put effort into improving that, you can impact 80% of the results – having a disproportionate effect on the whole. It is regularly discussed in business and I once worked with a company which had the 80:20 rule as one of its guiding principles.

Davis Balestracci’s Data Sanity has a really interesting observation on the power of the Pareto Principle in process improvement. One mode of process improvement is taking the exceptional and trying to understand why it happened and to learn from it. So, if site contracts in one country take much longer than in others to finalise, you can focus on that country to understand why and to improve. Or, of course, you could take the country with the shortest cycle time and try to understand why so you can spread “best practice”. This is the world of root cause analysis (RCA) & CAPA and can be effective in improvement. But what if the approach is over-used – for example maybe there are regularly issues detected in site audits for clinical trials that relate to problems with the process of Informed Consent. If there are many issues, then perhaps it would be better to look at them all rather than take each one individually as its own self-contained issue. In other words, maybe there is a systemic cause that is not related to the individual sites or studies. If you took all the issues (findings) together, you could use the Pareto Principle. It’s likely that 80% of the effects seen are due to a small number of causes. Why not work to find out what they are and implement changes to the whole system that affect those? Then continue to measure over time to see if it’s worked. Isn’t that likely to get better results than lots of independent RCA & CAPA efforts that each only has a small part of the picture?

That does bring up the challenge of how you determine when one issue is similar (or the same) as another. If you categorized all the issues in a consistent way, you’d see that around 80% of the observed issues come from 20% of the categories – the Pareto Principle in action. Just as we see from the MHRA. It’d be a good idea to focus process improvement on those 20% of categories.

Next time you look to improve a process, make sure you use the Pareto Principle to help focus your efforts so you can have maximum effect.

Tip: Pronounce Pareto as “pah-ree-toh”

Bringing Processes into Focus

I have been leading a process integration from a merger recently. The teams provided their many long SOPs and I tried to make sense of them – but with only minimal success. So, at the first meeting (web-based of course), I said we should map the process at high level (one page) for just one of the organisations. People weren’t convinced there would be a benefit but were willing to humour me. In a two-hour meeting, we mapped the process and were also able to:

Mark where the existing SOPs fit in the high-level process – giving a perspective no-one had seen before
Highlight differences in processes between the two organisations – in actual process steps, equipment or materials
Discuss strengths, weaknesses and opportunities in the processes
Agree an action plan for the next steps to move towards harmonisation

Mapping was done using MS PowerPoint. They loved this simple approach that made sure the focus of the integration effort was on the process – after all, to quote W. Edwards Deming, “If you can’t describe what you are doing as a process, you don’t know what you’re doing.” At a subsequent meeting, reviewing another process, one of the participants had actually mapped their process beforehand – and we used that as the starting point.

Process maps are such a powerful tool in helping people focus on what matters – without getting into unnecessary detail. They help people to come to a common perspective and to highlight differences to discuss. We also use them this way at the Metrics Champion Consortium where one of the really important outcomes from mapping is the recognition of different terminology used by different organisations. We can then focus on harmonising the terminology and developing a glossary of terms that we all agree on. This reduces confusion in subsequent discussions.

Process maps are really a great tool. They are useful when complete, but so much more benefit comes from a team of people with different perspectives actually developing them. They help to bring processes into focus. And can even help with root cause analysis. If you don’t use them, perhaps you should!

For those that use process maps, what do you find as the benefits? And the challenges?

Picture – PublicDomainPictures from Pixabay

Are we seeing a breakthrough in clinical trial efficiency?

I joined my first CRO as an “International Black Belt” in 2005. Having come from a forward-thinking manufacturer who had been implementing six sigma and lean philosophy, I was dumb-founded by what I saw. After the first few weeks, I mentioned to a colleague that most of what seemed to happen in clinical trials was about checking because the process could not be relied on to be right the first time. Manufacturing learned in the 1980s and 1990s that checking (or “inspection” as they call it) is costly, inefficient, and ineffective. This colleague recently repeated this back to me. We’ve all seen examples in clinical trial – TMF documents being checked before sending, checked on receipt, then checked during regular QCs; reports going through endless rounds of review; data queries being raised for items that can have no impact on trial results or patients. When challenged, often the response is that we’ve always done it that way. Or that QA, or the regulators, tell us we have to do it that way. I’ve spent my career in clinical trials trying to get people to focus on the process:

- What is the purpose?
- What are the inputs and outputs?
- What is the most efficient way to get from one to the other?
- How can we measure the process use the measurement to continuously improve?
- What is the perspective of the “customers” of the process?
- What should we do when a process goes wrong?

And I’ve had a number of successes along the way – the most satisfying of which is when someone has an “Aha!” moment and takes the ideas and runs with them. Mapping a process themselves to highlight where there are opportunities to improve, for example. But I do often wonder why it is so difficult to get the industry to make the significant changes that we all know it needs. Process improvement should not be seen as an optional extra. It is a necessity to stay in business. It seems unfair to blame regulators who have been pushing us along to be process focused – for example with the introduction of Quality Tolerance Limits in GCP in 2016.

COVID-19 has caused so much loss of life and impacted everybody’s lives. It has been hugely to the detriment of the people of the world. And yet, there are some positives too. In clinical trials, suddenly, people are starting to ask “how can we make this change?” rather than “why can’t we make this change?” At meetings in the Metrics Champion Consortium we have heard stories of cycle times that were thought impossible for developing a protocol, for example, of a company that has switched from 100% Source Document Verification to 0% after reviewing evidence of the ineffectiveness of the process; and of companies implementing remote and centralized monitoring in record time. There are some great examples from the COVID-19 RECOVER study in the UK. And, at the same time, pharmaceuticals and the associated clinical trials are seen as critical to helping us turn the corner of the pandemic.

Let’s hope this new-found momentum to improve continues in our industry when this pandemic is finally declared over. And we can bring new therapies to patients much quicker in the future – with less cost and with quality and safety as high or even higher than in the past. We are showing what’s possible. Let’s continue to challenge each other on that assumption that because we’ve always done things one way, we have to continue.

Picture – Gerd Altmann, Needpix.com