Measurement – Page 2

What My Model of eTMF Processing Taught Me (Part II)

In a previous post, I described a model I built for 100% QC of documents as part of an eTMF process. We took a look at the impact of the rejection rate for documents jumping from 10% to 15%. It was not good! So, what happens when an audit is announced and suddenly the number of documents submitted doubles? In the graph below, weeks 5 and 6 had double the number of documents. Look what it does to the inventory and cycle time:

The cycle time has shot up to around 21 days after 20 weeks. The additional documents have simply added to the backlog and that increases the cycle time because we are using First In, First Out.

So what do we learn overall from the model? In a system like this, with 100% QC, it is very easy to turn a potential bottleneck into an actual bottleneck. And when that happens, the inventory and cycle time will quickly shoot upwards unless additional resource is added (e.g. overtime). But, you might ask, do we really care about cycle time? We definitely should: if the study team can’t access documents until they have gone through the QC, those documents are now not available for 21 days on average. That’s not going to encourage every day use of the TMF to review documents (as the regulators expect). And might members of the study team send in duplicates because they can’t see the documents that are awaiting processing? Adding further documents and impacting inventory and cycle time still further. And this is not a worst case scenario as I’m only modelling one TMF here – typically a Central Files group will be managing many TMFs and may be prioritizing one over another (i.e. not First In, First Out). This spreads out the distribution of cycle times and will lead to many more documents that are severely delayed through processing.

“But we need 100% QC of documents because the TMF is important!” I hear you shout. But do you really? As the great W Edwards Deming said, “Inspection is too late. The quality, good or bad, is already in the product.” Let’s get quality built in in the first place. You should start by looking at that 15% rejection rate. What on earth is going on to get a rejection rate like that? What are those rejections? Are those carrying out the QC doing so consistently? Do those uploading documents know the criteria? Is there anyone uploading documents who gets it right every time? If so, what is it that they do differently to others?

What if you could get the rejection rate down to less than 1%? At what point would you be comfortable taking a risk-based approach – that assumes those uploading documents do it right the first time? And carrying out a random QC to look for systemic issues that could then be tackled? How much more efficient this would be. See the diagram in this post. And you’d remove that self-imposed bottleneck. You’d get documents in much quicker, costing less and with improved quality. ICH E6 (R2) is asking us to consider quality as not being 100% but concerning ourselves with errors that matter. Are we brave enough as an industry to apply this to the TMF?

Picture: CC BY 2.0 Remko Van Dokkum

What My Model of eTMF Processing Taught Me

On a recent long-haul flight, I got to thinking about the processing of TMF documents. Many organisations and eTMF systems seem to approach TMF documents with the idea that every one must be checked by someone other than the document owner. Sometimes, the document owner doesn’t even upload their own documents but provides them, along with metadata, to someone else to upload and index. And then their work is checked. There are an awful lot of documents in the TMF and going through multiple steps of QC (or inspection as W Edwards Deming would call it) seems rather inefficient – see my previous posts. But we are a risk-averse industry – even having been given the guidance to used risk-based approaches in ICH E6 (R2) and so many organizations seem to use this approach.

So what is the implication of 100% QC? I decided I would model it via an Excel spreadsheet. My assumptions are that there are 1000 documents submitted per week. Each document requires one round of QC. The staff in Central Files can process up to 1100 documents per week. I’ve included a random +/-5% to these numbers for each week (real variation is much greater than this I realise). I assume 10% of documents are rejected at QC. And that when rejected, the updated documents are processed the next week. I’ve assumed First In, First Out for processing. My model looks at the inventory at the end of each week and the average cycle time for processing. It looks like this:

It’s looking reasonably well in control. The cycle time hovers around 3 days after 20 weeks which seems pretty good. If you had a process for TMF like this, you’d probably be feeling pretty pleased.

So what happens if the rejection rate is 15% rather than 10%?

Not so good! It’s interesting just how sensitive the system is to the rejection rate. This is clearly not a process in control any more and both inventory and cycle time are heading upwards. After 20 weeks, the average cycle time sits around 10 days.

Having every document go through a QC like this forms a real constraint on the system – a potential bottleneck in terms of the Theory of Constraints. And it’s really easy to turn this potential bottleneck into a real bottleneck. And a bottleneck in a process leads to regular urgent requests, frustration and burn-out. Sound familiar?

In my next post, I’ll take a look at what happens when an audit is announced and the volume of documents to be processed jumps for a couple of weeks.

Picture: CC BY 2.0 Remko Van Dokkum

Performance Appraisal – A Better Way

I wrote previously about the waste of the annual performance appraisal. Perhaps you’ve just gone through yours or you’re about to. As I wrote at the time, “With employees and managers hating the process of annual performance appraisals, isn’t it about time we ditched them in favour of a continuous assessment approach and an ongoing focus on goals – for both the employee and organization?” A reasonable criticism is – but how would that process work? And wouldn’t it suffer from the same problems?

My friend, Linda Sullivan, recommended a book for me to read recently – John Doerr’s “Measure What Matters”. It’s about a process called OKRs (Objectives, Key Results) and Part 2 of the book is about moving away from annual appraisals to continuous performance management. Worth a read if you want to see another way. As Doerr says “individuals cannot be reduced to numbers”. Something we all know really. Some ideas that I think could be revolutionary in work places that focus on the annual performance appraisal and goal-setting:

- Objectives should have a short cycle time – maybe only 3 months
- Objectives shouldn’t be between just the employee and manager. They should be shared broadly. This makes it clear what the priorities are – if your request isn’t within my priorities, don’t be surprised if I put it off for now. And if I’m struggling to meet objectives, please help me!
- Don’t stick to objectives just because they were agreed at the start. Things change. Objectives sometimes need to change too.
- Don’t link achieving of objectives directly to remuneration. This encourages “sand-bagging” and meeting objectives at all costs.
- Regular employee meetings should focus on learning, coaching, understanding barriers and development.

For next year, could you persuade your management and HR department to get rid of the hated annual performance appraisals and goal setting?

Want to learn more about using KPIs correctly? Drop me a line! Or take a look at the training opportunities.

Happy New Year!

Picture: Marco Verch (CC BY 2.0)

Windrush: Use and Abuse of Metrics

Here in the UK, a huge scandal has blown up in the government’s face recently. The so-called Windrush generation are members of the former British Empire in the Caribbean who came to the UK after the war at Britain’s request. They were wanted to help rebuild Britain. And they made the UK their home. They worked, had families, paid taxes and made a difference. They are British and recognised as such since an act in 1971. Approximately 500,000 came to the UK (including from other countries such as India and Pakistan). Now as they reach retirement, they have fallen foul of successful measures taken by the government to make a hostile environment for illegal immigrants. They are not illegal but are caught out by the rules – they have to prove they are not illegal and not all have been able to do so. There are tragic stories in the news of people being detained, deported, denied work, housing, healthcare even though they are British. The country has been horrified – how on earth is this possible?

As more and more leaks out, it is becoming clear that metrics are an important part of the story. The government has been desperate since 2010 to reduce immigration. They have been proud of creating a hostile environment for illegal immigrants and deporting them if they cannot prove their right to be here. And some of the Windrush generation have been caught up in this. It has emerged that the Home Office had a target to forcibly return around 12,000 illegal immigrants per year. These outrageous examples of deporting British people with the right to be here are included in that total and help the department meet the target. With huge pressure to meet a target, people will try their hardest to do so – whatever the means. If you had a target to deport illegal immigrants and your job (or bonus) depended on it then what would you do? How sympathetic would you be to someone who could not prove they were legally here?

Metrics can drive the wrong behaviour as well as the right behaviour. The country is appalled at the way these people have been treated including newspapers who are normally strongly supportive of the government. This was clearly the wrong behaviour – being driven to meet a metric target. And the evidence of these cases has been there for several years. But the target was always more important.

This is the wrong use of metrics. They should not be used at the exclusion of thinking and compassion. Whatever metrics you use, they can drive the wrong behaviour – always look behind them to understand what is happening. Ask the right questions. If the metric is improving, ask why. Ask how. Is it driving the right behaviour or are there negative consequences? Think.

Thanks to a free press, these stories have come into the open and the government has apologised and taken action to reverse the injustices. But these injustices should not have happened in the first place and some people’s lives have been turned upside-down. The lesson – use metrics carefully and thoughtfully and watch for them driving the wrong behaviour.

A great book that makes clear the use and abuse of metrics in the public sector is “Systems Thinking in the Public Sector” by John Seddon.

Picture: kmusser

Not everything that counts can be counted!

It’s coming to that time of year again – performance appraisals! Do you know anyone who likes them? When I first became a line manager, I was lucky, I had actually received training on the importance of performance appraisals and how to run them. The time came and I was ready. Except I wasn’t ready for the wave of negativity. Employees came in to my office and slumped on the chair ready for what they clearly considered was a pointless annual appraisal. I tried my best, I used all the techniques I had been taught. I got feedback from peers, I considered strengths, weaknesses, opportunities. I spent hours putting the text together for the appraisal and further hours working out the SMART goals for the next year. But to no avail. Perhaps I wasn’t doing it right. I tried other ways but somehow over many cycles of these and different employers and employees I have never managed to work out the right formula. It was the same with my own appraisals – I never felt they really added much to what I already knew. And my sense of disappointment and unfairness at getting a “Meets Expectations” one year where I thought I had achieved so much lasted a long time.

Could it be that actually the annual appraisal process itself is not fit for purpose in the modern world of work? In fact, perhaps it never was. So much of what we do is team work, it is actually quite tricky to separate out the individual contribution. And things change so quickly in organizations. What seems important when goals are set may be irrelevant even two months later. As for SMART goals, there are many critiques of these and I have really been left questioning their value. They encourage you to set targets you know you’re going to achieve rather than challenging ones you might fail at. The challenging ones will bring your performance rating down. I’ve been in organizations that spent months going round and round trying to agree SMART objectives for the year and only getting there by May (when the appraisal year started in January, five months earlier!) And there were way too many goals that employees only looked at when the manager reminded them.

As to the actual rating process, I have never come across one that worked well. Many seemed based on the 5 point scale (1=Unacceptable Performance, 2=Needs Improvement, 3=Meets Expectations, 4=Exceeds Expectations, 5=Exceptional Performance). As the resident data nerd in one organization, I was given all the data on ratings to crunch to see what the distribution looked like. You can probably guess. 0.2% got a 1 rating, 0.8% got a 2 rating, and 0.2% got a 5 rating. In other words, 98.8% of people got a 3 or 4 rating. Managers tend to choose the middle ratings because it is easier – explaining the extreme values to employees and superiors is hard work.

Of course, good managers do much better. They manage performance on an ongoing basis. The annual performance appraisal becomes more of an administrative burden. Isn’t it about time we got rid of this failed approach and got managers managing performance with their employees on an ongoing basis? They could talk about strengths, weaknesses, opportunities, ways to grow without worrying about comparing individuals using numbers.

With employees and managers hating the process of annual performance appraisals, isn’t it about time we ditched them in favour of a continuous assessment approach and an ongoing focus on goals – for both the employee and organization? If you’re wondering how that might work, take a look here.

A phrase often wrongly attributed to Einstein but actually thought to be from the sociologist, William Bruce Cameron should give us pause for thought when using ratings for annual performance appraisals: “not everything that can be counted counts, and not everything that counts can be counted”.

Want to learn more about using KPIs correctly? Drop me a line! Or take a look at the training opportunities.

Picture: Rizkyharis CC BY-SA 4.0