Evaluating Patient-Based Outcome Measures - NIHR Health ...

32
Criteria for selecting a patient-based outcome measure
patients beyond a certain level (Ganiats
et al., 1992).
Distribution of baseline scores
The responsiveness of an instrument may also
be influenced by the relationship of items in the
instrument to the distribution of levels of difficulty
or severity in the underlying construct. As a hypothetical
example, it is possible to imagine an instrument
designed to measure mobility where items
mainly reflected ‘easy’ tasks; that is the majority
of respondents could be expected to report no
problem, for example, in walking a very short
distance. Because most items in the scale reflect
‘easy’ items, a large amount of change could be
produced (i.e. the patient reports change over the
majority of items) even when only a small amount
of real improvement had occurred. Stucki and
colleagues (1995) show that the problem of the
relationship of items to an underlying range of
degrees of difficulty or seriousness is not entirely
hypothetical. They provide evidence that many
items from the physical ability scale of the SF-36
reflect intermediate rather than extremes of level
of difficulty for patients undergoing total hip
arthroplasty. Thus patients experiencing improvements
at this intermediate level of physical difficulty
can be expected to experience high levels of
gain according to SF-36 at least in part because of
the range of items. As Stucki and colleagues argue,
this problem can arise from the ways in which
scales are often developed, as described in earlier
sections of this report, with emphasis upon high
levels of agreement between items on a scale
(internal reliability), rather than requiring items
that reflect a full range of difficulty or severity
of an underlying problem. We have already seen
arguments against excessive reliance on inter-item
agreement to develop instruments rehearsed by
Kessler and Mroczeck (1995) in the context of
reliability, above. Here it is possible to see
problems arising from excessive emphasis
upon internal reliability in the context
of responsiveness.
Summary
The need for an instrument to be responsive
to changes that are of importance to patients
should be of evident importance in the context
of clinical trials. Whilst there are no universally
agreed methods for assessing this property, at a
more general level all discussions require evidence
of statistically significant change of some form
from observations made at separate times and
when there is good reason to think that changes
have occurred that are of importance
to patients.
Precision
How precise are the scores of
the instrument?
This review is primarily concerned with the use
of patient-based outcome measures in the context
of clinical trials. Investigators will need to examine
the pattern of responses to health status measures
in a trial to determine whether there are clear and
important differences between the arms of a trial.
They therefore need to examine a number of
aspects of candidate instruments’ numerical
properties which have not been clearly delineated
in the literature, but which relate to the precision
of distinctions made by an instrument. Testa
and Simonson (1996) refer to this property
as ‘sensitivity’:
‘Although a measure may be responsive to changes
in Q (quality of life), gradations in the metric of Z
(the instrument) may not be adequate to reflect these
changes. Sensitivity refers to the ability of the measurement
to reflect true changes or differences in Q’
(1996: 836).
Stewart (1992) also refers to this property as
‘sensitivity’. In particular, she refers to the number
of distinctions an instrument makes; the fewer,
the more insensitive it is likely to be. Kessler
and Mroczek (1995) refer to this property as
‘precision’, which is probably less confusing since
sensitivity has a number of other uses and meanings
in this field. As Kessler and Mroczek argue,
an instrument may have high reliability but low
precision if it makes only a small number of crude
distinctions with regard to a dimension of health.
Thus at the extreme one instrument might
distinguish with high reliability only between
those who are healthy and those who are ill. For
the purposes of a trial, such an instrument would
not be useful because it is degrees of change
within the category of ‘unwell’ that are likely
to be needed to evaluate results of the arms
of the trial.
There are a number of ways in which the issue of
precision has been raised in relation to patientbased
outcome measures. This is fairly disparate
evidence and it is reviewed under a number of
more specific headings.
Precision of response categories
One of the main influences on the precision of
an instrument is the format of response categories;
i.e. the form in which respondents are able to give
their answers. At one extreme answers may be given
by respondents in terms of very basic distinctions,