The Quality Measurement Journey - Sikker Patient

Milestones in 

The Quality Measurement Journey 

Prepared by 

Bob Lloyd, PhD 

Executive Director Performance Improvement 

Institute for Healthcare Improvement 

Learning Session 4 

20 March 2012 


AIM (How good By when) 

Concept 

Measures 

Operational Definitions 

Data Collection Plan 

Data Collection 

Analysis 

ACTION 

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004. 

©2012 Institute for Healthcare Improvement/R. Lloyd 

Institute for Healthcare Improvement


AIM (How good By when) 

Concept 

Measures 




Analysis 

ACTION 


©Copyright 2011 IHI/R. Lloyd 

From Concept to Measure 

“How will I know that a 

change is an improvement” 

How can we construct measures that 

will usefully and efficiently answer 

this question 



Every concept can have many measures 

Concept 

Hand Hygiene 

Medication Errors 

Ventilator Associated 

Pneumonias (VAPs) 

Potential Measures 

Ounces of hand gel used each day 

Ounces of gel used per staff 

Percent of staff washing their hands 

(before & after visiting a patient) 

Percent of inpatients with C.Diff 

Percent of errors 

Number of errors 

Medication error rate 

Percent of patients with a VAP 

Number of VAPs in a month 

The number of days without a VAP 

Compliance with the ventilator bundle 



Three Types of Measures 

(a Family of Measures) 

• Outcome Measures: Voice of the customer or patient. 

How is the system performing What is the result 

• Process Measures: Voice of the workings of the 

system. Are the parts/steps in the system performing as 

planned 

• Balancing Measures: Looking at a system from 

different directions/dimensions. What happened to the 

system as we improved the outcome and process 

measures (e.g. unanticipated consequences, other 

factors influencing outcome) 



Potential Set of Measures for Improvement 

in the Accident & Emergency (ER) 

Topic 

Outcome 

Measures 

Process 

Measures 

Balancing 

Measures 

Improve waiting 

time and patient 

satisfaction in 

the ER 

Total Length of 

Stay in the ER 

Patient 

Satisfaction 

Scores 

Time to 

registration 

Patient and staff 

comments on flow 

% patient receiving 

discharge 

materials 

Volumes 

% Leaving 

without being 

seen 

Staff satisfaction 

Financials 

Availability of 

antibiotics 


Balancing Measures: Looking at the 

System from Different Dimensions 

• Outcome (quality, time) 

• Transaction (volume, no. of patients) 

• Productivity (cycle time, efficiency, 

utilisation, flow, capacity, demand) 

• Financial (charges, staff hours, materials) 

• Appropriateness (validity, usefulness) 

• Patient satisfaction (surveys, customer 

complaints) 

• Staff satisfaction 




AIM (Why are you measuring) 

Concept 

Measures 




Analysis 

ACTION 




“Would you tell me, please, which way I 

ought to go from here,” asked Alice 

“That depends a good deal on where you 

want to get to,” said the Cat. 

“I don’t much care where” - said Alice. 

“Then it doesn’t matter which way you 

go,” said the Cat. 

From Alice in Wonderland, Brimax Books, London, 1990. 



An Operational Definition... 

… is a 

description, in 

quantifiable 

terms, of what 

to measure and 

the steps to 

follow to 

measure it 

consistently. 

• It gives communicable 

meaning to a concept 

• Is clear and 

unambiguous 

• Specifies measurement 

methods and equipment 

• Identifies criteria 

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004. 

11 


How do you define the following 

healthcare concepts 

• World Class Performance • Delayed discharges 

• Alcohol related admissions • End of life care 

• Teenage pregnancy 

• Falls (with/without injuries) 

• Cancer waiting times 

• Childhood immunizations 

• Health inequalities 

• Complete maternity service 

• Asthma admissions 

• Patient engagement 

• Childhood obesity 

• Moving services closer to home 

• Patient education 

• Successful breastfeeding 

• Health and wellbeing 

• Ambulatory care 

• Adding life to years and years to life • Access to health in deprived areas 

• Children's palliative care • Diagnostics in the community 

• Safe services 

• Productive community services 

• Smoking cessation 

• Vascular inequalities 

• Urgent care 

• Breakthrough priorities 



Exercise 


• Select a measure for your project (it can 

be an outcome, process or balancing 

measure). 

• Evaluate this measure against the 

criteria for a “good” operational 

definition. 

• How well does your measure stand up 

• Use the Operational Definition 

Worksheet to guide and record your 

work. 


Operational Definition Worksheet © 

Name of team:_______________________________ Date: _____________ 

Measure Name 

(Provide a specific name such 

as medication error rate) 

Operational Definition 

(Define the measure in very specific terms. 

Provide the numerator and the denominator 

if a percentage or rate. Indicate what is to be 

included and excluded. Be as clear and 

unambiguous as possible) 

Data Source(s) 

(Indicate the sources of 

the data. These could 

include medical records, 

logs, surveys, etc.) 

Data 

Collection: 

•Schedule (daily, weekly, 

monthly or quarterly) 

•Method (automated 

systems, manual, telephone, 

etc.) 

Baseline 

•Period 

•Value 

Goals 

•Short term 

•Long term 





NON-SPECIFIC CHEST PAIN PATHWAY MEASUREMENT PLAN 

Measure Name 

(Provide a specific name such 

as medication error rate) 

Operational Definition 

(Define the measure in very specific 

terms. 

Provide the numerator and the 

denominator if a percentage or rate. 

Indicate what is to be included and 

excluded. Be as clear and 

unambiguous as possible) 

Data Source(s) 

(Indicate the sources of the 

data. These could include 

medical records, logs, 

surveys, etc.) 

Data 

Collection: 

•Schedule (daily, weekly, 

monthly or quarterly) 

•Method (automated systems, 

manual, telephone, etc.) 

Baseline 

•Period 

•Value 

Goals 

•Short term 

•Long term 

Percent of patients who have 

MI or Unstable Angina as 

diagnosis 

Numerator = 

Patients entered into the NSCP 

pathway each month who have Acute 

MI or Unstable Angina as the 

discharge diagnosis 

Denominator = 

All patients entered into the NSCP 

pathway each month 

1.Medical Records 

2.Midas data system 

3.Variance Tracking Form 

1.Discharge diagnosis will be 

identified for all patients entered 

into the NSCP pathway 

2.QA-URwill retrospectively 

review charts of all patients 

entered into the NSCP pathway. 

Data will be entered into MIDAS 

system 

1.Currently 

collecting 

baseline data. 

2.Baseline 

will be 

completed by 

end of 1 st Q 

2012 

Once the baseline of 

missed diagnosis is 

calculated short and 

long term reductions 

will be established as 

the goals 

Number of patients who are 

admitted to the hospital or seen 

in an ED due to chest pain 

within one week of when we 

discharged them 

A patient that we saw in our ED who 

reports during the call-back interview 

that they have been admitted or seen in 

an ED (ours or some other ED) for 

chest pain during the past week 

All patients who have been 

managed within the NSCP 

protocol throughout their 

hospital stay 

1.Patients will be contacted by 

phone one week after discharge 

2.Call-back interview will be the 

method 

1.Currently 

collecting 

baseline data. 

2.Baseline 

will be 

completed by 

end of 1 st Q 

2012 

Ultimately the goal 

is to have no patients 

admitted or seen in 

the ED within a 

week after discharge. 

The baseline will be 

used to help 

establish initial 

goals. 

Costs per cardiac diagnosis (a 

rate) 

Numerator = 

Total costs each month for hospital 

care of NSCP pathway patients 

Denominator = 

Number of patients each month 

entered into the NSCP pathway with a 

discharge diagnosis of MI or Unstable 

Angina 

1.Finance 

2.Chart Review 

Can be calculated every month 

from financial and clinical data 

already being collected 

1.Months 

2011 

2.Will be 

computed 

within 10 

days of the 

close of the 

previous 

month 

The initial goal will 

be to reduce the 

baseline cost per 

case by 5%within 

the first six months 

of initiating the 

project. 




Concept 

Measures 




Analysis 

ACTION 




Now that you have selected and defined 

your measures, it is time to head out, cast 

your net and actually gather some data! 


Key Data Collection Strategies 

Stratification 

• Separation & classification 

of data according to 

predetermined categories 

• Designed to discover 

patterns in the data 

• For example, are there 

differences by shift, time of 

day, day of week, severity 

of patients, age, gender or 

type of procedure 

• Consider stratification 

BEFORE you collect the 

data 

Sampling Strategies 

• Systematic sampling 

• Simple random sampling 

• Stratified sampling 

• Proportional stratified 

random sampling 

• Quota sampling 

• Judgment sampling 

• Convenience sampling 

• Knowledge of sampling 

techniques will make data 

collection much easier and 

less time consuming 



Examples of Stratification Problems 


Sampling Methods 

Probability Sampling Methods 

• Simple random sampling 

• Stratified random sampling 

• Stratified proportional random sampling 

• Systematic sampling 

• Cluster sampling 

Non-probability Sampling Methods 

• Convenience sampling 

• Quota sampling 

• Judgment sampling 





Concept 

Measures 




Analysis 

ACTION 



You have 

performance data. 

Now what the heck 

do you do with it 

22 



“If I had to reduce 

my message for 

management to just 

a few words, I’d say 

it all had to do with 

reducing variation.” 

W. Edwards Deming 

23 


The Problem 

Aggregated data presented in 

tabular formats or with summary 

statistics, will not help you 

measure the impact of process 

improvement/redesign efforts. 

Aggregated data can only lead to 

judgment, not to improvement! 

24 



Average CABG Mortality 

Before and After the Implementation of a New Protocol 

25 

5.2 

Percent Mortality 

3.8 

5.0% 

WOW! 

A “significant drop” 

from 5% to 4% 

4.0% 

Time 1 Time 2 

Conclusion -The protocol was a success! 

A 20% drop in the average mortality! 


Average CABG Mortality 

Before and After the Implementation of a New Protocol A Second Look at the Data 

9.0 

Protocol implemented here 

Percent Mortality 

5.0 

1.0 

UCL= 6.0 

CL = 4.0 

LCL = 2.0 

24 Months 

Now what do you conclude about the impact of the protocol 

26 



The average of a set of numbers can be 

created by many different distributions 

Measure 

X 

(CL) 

Time 

27 


If you don’t understand the variation that 

lives in your data, you will be tempted to ... 

• Deny the data (It doesn’t fit my view of 

reality!) 

• See trends where there are no trends 

• Try to explain natural variation as special 

events 

• Blame and give credit to people for things 

over which they have no control 

• Distort the process that produced the data 

• Kill the messenger! 

28 



Dr. Walter A Shewhart 

Shewhart - Economic Control of Quality of 

Manufactured Product, 1931. 

“A phenomenon 

will be said to be 

controlled when, 

through the use of 

past experience, 

we can predict, at 

least within limits, 

how the 

phenomenon may 

be expected to vary 

in the future” 


“What is the variation in one system 

over time” Walter A. Shewhart - early 1920’s, Bell Laboratories 

Dynamic View 

UCL 

Static View 

time 

LCL 

Every process displays variation: 

• Controlled variation 

stable, consistent pattern of variation 

“chance”, constant causes 

30 

Static View 

• Special cause variation 

“assignable” 

pattern changes over time 



Types of Variation 

Common Cause Variation 

• Is inherent in the design of the 

process 

• Is due to regular, natural or 

ordinary causes 

• Affects all the outcomes of a 

process 

• Results in a “stable” process that 

is predictable 

• Also known as random or 

unassignable variation 

Special Cause Variation 

• Is due to irregular or unnatural 

causes that are not inherent in 

the design of the process 

• Affect some, but not necessarily 

all aspects of the process 

• Results in an “unstable” 

process that is not predictable 

• Also known as non-random or 

assignable variation 

31 


Point … 

Common Cause does not mean “Good Variation.” It only means 

that the process is stable and predictable. For example, if a 

patient’s systolic blood pressure averaged around 165 and was 

usually between 160 and 170 mmHg, this might be stable and 

predictable but completely unacceptable. 

Similarly Special Cause variation should not be viewed as “Bad 

Variation.” You could have a special cause that represents a very 

good result (e.g., a low turnaround time), which you would want to 

emulate. Special Cause merely means that the process is unstable 

and unpredictable. 

You have to decide if the output of the process is 

acceptable! 



Unplanned Returns to Ed w/in 72 Hours 

Month M A M J J A S O N D J F M A M J J A S 

ED/100 41.78 43.89 39.86 40.03 38.01 43.43 39.21 41 .90 41.78 43.00 39.66 40.03 48.21 43.89 39.86 36.21 41.78 43.89 31.45 

Retur ns 17 26 13 16 24 27 19 14 33 20 17 22 29 17 36 19 22 24 22 

u chart 

1.2 

chart 

1.0 

0.8 

0.6 

0.4 

0.2 

0.0 

1 

UCL = 0 .88 

Mean = 0.54 

LCL = 0.19 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

How can I depict variation 

Rate per 100 ED Patients 

33 

STATIC VIEW 

Descriptive Statistics 

Mean, Median & Mode 

Minimum/Maximum/Range 

Standard Deviation 

Bar graphs/Pie charts 

DYNAMIC VIEW 

Run Chart 

Control Chart 

(plot data over time) 

Statistical Process Control (SPC) 


The SPC Pioneers 

W. Edwards 

Deming 

(1900 - 1993) 

Walter 

Shewhart 

(1891 – 1967) 

Joseph Juran 

(1904 - 2008) 



How do we analyze variation for 

quality improvement 

Run and Control Charts 

are the best tools to 

determine if our 

improvement strategies 

have had the desired 

effect. 

35 


Elements of a Run Chart 

6.00 

5.75 

5.50 

The centerline (CL) on a 

Run Chart is the Median 

Measure 

Pounds of Red Bag Waste 

5.25 

5.00 

4.75 

4.50 

4.25 

4.00 

Median=4.610 

~ 

X (CL) 

3.75 

3.50 

3.25 

Time 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 

Point Number 

Four simple run rules are used to determine if 

non-random data patterns are present 



Elements of a Shewhart 

Control Chart 

50.0 

45.0 

40.0 

(Upper Control Limit) 

UCL=44.855 

A 

Measure 

Number of Complaints 

35.0 

30.0 

25.0 

20.0 

15.0 

10.0 

An indication of 

a special cause 

B 

C 

CL=29.250 

C 

B 

A 

X (Mean) 

LCL=13.645 

(Lower Control Limit) 

5.0 

Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02 

Time 

Month 


Run and Control Charts 

don’t tell you 

• The reasons(s) for a Special Cause 

• Whether or not a Common Cause process 

should be improved (Is the performance of 

the process acceptable) 

• How the process should actually be 

improved or redesigned 



The Sequence for Improvement 

Test under 

a variety of 

conditions 

Make part 

of routine 

operations 

Implementing a 

change 

Sustaining 

improvements and 

Spreading the changes 

to other locations 

Theory 

and 

Prediction 

Developing a 

change 

Testing a 

change 

Act 

Study 

Plan 

Do 


General References on Quality 

• The Improvement Guide: A Practical Approach to Enhancing 

Organizational Performance. G. Langley, K. Nolan, T. Nolan, C. Norman, 

L. Provost. Jossey-Bass Publishers., San Francisco, 1996. 

• Quality Improvement Through Planned Experimentation. 2nd edition. R. 

Moen, T. Nolan, L. Provost, McGraw-Hill, NY, 1998. 

• The Improvement Handbook. Associates in Process Improvement. 

Austin, TX, January, 2005. 

• A Primer on Leading the Improvement of Systems,” Don M. Berwick, 

BMJ, 312: pp 619-622, 1996. 

• “Accelerating the Pace of Improvement - An Interview with Thomas 

Nolan,” Journal of Quality Improvement, Volume 23, No. 4, The Joint 

Commission, April, 1997. 

40 



References on Measurement 

• Brook, R. et. al. “Health System Reform and Quality.” Journal of 

the American Medical Association 276, no. 6 (1996): 476-480. 

• Carey, R. and Lloyd, R. Measuring Quality Improvement in 

healthcare: A Guide to Statistical Process Control Applications. 

ASQ Press, Milwaukee, WI, 2001. 

• Lloyd, R. Quality Health Care: A Guide to Developing and Using 

Indicators. Jones and Bartlett Publishers, Sudbury, MA, 2004. 

• Nelson, E. et al, “Report Cards or Instrument Panels: Who Needs 

What Journal of Quality Improvement, Volume 21, Number 4, 

April, 1995. 

• Provost, L. and Murray, S. The Health Care Data Guide. Jossey- 

Bass, San Francisco, 2011. 

• Solberg. L. et. al. “The Three Faces of Performance Improvement: 

Improvement, Accountability and Research.” Journal of Quality 

Improvement 23, no.3 (1997): 135-147. 

41 


“Quality begins with 

intent, which is fixed 

by management.” 

W. E. Deming, Out of the Crisis, p.5 

42 



Robert Lloyd, Ph.D. 

Executive Director Performance Improvement 

can be reached at: 

43 

The Institute for Healthcare 

Improvement 

(630) 717-5383 Chicago office 

(630) 717-8564 fax 

rlloyd@ihi.org 

Dr. Lloyd’s books, Measuring Quality Improvement in Healthcare: A 

Guide to Statistical Process Control Applications and Quality 

HealthCare: A Guide to Developing and Using Indicators, can be 

obtained from ASQ Press (800) 248-1948 and Jones & Bartlett 

Publishing (800) 832-0034 respectively.

The Quality Measurement Journey - Sikker Patient

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?