Evaluating Patient-Based Outcome Measures - NIHR Health ...

28
Criteria for selecting a patient-based outcome measure
wide-ranging purposes, they require more
extensive validation.
There are potentially three ways in which such
measures need to be assessed for validity: as
measures of (i) health status, (ii) of personal
preferences and utilities, and (iii) of the social
value. It is important to recognise that these are
distinct constructs. Thus, Nord (1992) suggests that
if measures are intended to provide assessments of
the social value of interventions, as opposed to a
measure of the individual utilities of patients, then
one important component of validation would
need to be reflective equilibrium whereby respondents
are directly invited to consider and accept
the implications in terms of resource allocation
of judgements made about weightings. To the
extent that public opinion accepted decisions
about resource allocation of health services based
on evidence of costs and benefits of treatments
and where benefits are in part measured by
utilities, according to Nord, this would be an
indirect form of validation of utility measures
used. An Australian study found very little public
support for health policies that aimed to maximise
health benefit without egalitarian considerations
(Nord et al., 1995).
Nord and colleagues has examined the validity
of generic measures by more direct methods.
In a series of surveys of Norwegian and Australian
samples, Nord and colleagues (1993) examined
respondents’ ratings of specific health states for an
instrument in comparison with the same health
states judged by ‘Person Trade Off’. The latter
method involves respondents being asked to state
what numbers of patients receiving one treatment
are equivalent to a specific number receiving a
second different treatment and is intended more
directly to assess the social value of different
health states. By comparing respondents’ ratings
from the two methods, it emerged that the
instrument weighted by conventional rating
scales produced much lower values for health
states than did the Person Trade Off method, so
that it appeared to produce lower social value for
interventions. As Nord and colleagues argue,
measures need to be examined separately to
assess their validity as measures of individuals’
utilities and as measures of social value. The two
are distinct constructs and a measure may be
valid as a measure of one but not the other and
this is an area of continued methodological
debate as to the validity of instruments for these
different purposes (Carr-Hill, 1992; The EuroQol
Group, 1992; Gafni and Birch, 1993; Dolan and
Kind, 1996).
Summary
The apparently simple question as to whether an
instrument measures what it purports to measure
has to be considered by means of a range of different
kinds of evidence including how content was
determined, inspection of the content, and of
patterns of relationships to other variables. Because
no single set of observations is likely to determine
validity and different kinds of evidence are needed,
judgement of this property of an instrument in
relation to a specific trial is not straightforward.
Responsiveness
Does the instrument detect changes
over time that matter to patients?
For use in trials, it is essential that a health
status questionnaire can detect important changes
over time within individuals, that might reflect
therapeutic effects (Kirshner and Guyatt, 1985;
Kirshner, 1991). This section addresses sensitivity
to change, or responsiveness. The latter term is
preferable because sensitivity has a number of
more general uses in epidemiology. As it is conceivable
for an instrument to be both reliable
and valid but not responsive, this dimension
of a health status measure is increasingly essential
to evaluate. Potential confusion is caused by the
fact that in order to emphasise its importance,
some authors treat it as an aspect of validity
(Hays and Hadorn, 1992). Guyatt and colleagues
(1989a) define responsiveness as the ability of
an instrument to detect clinically important
change. They provide illustrative evidence of
the importance of this aspect of instruments with
data from a controlled trial of chemotherapy for
breast cancer. Four health status instruments
considered to be validated were completed by
women. However only one of the four instruments
showed expected differences over time as well
as providing valid evidence of women’s’ health
status. Guyatt and colleagues (1987, 1992a,b)
have emphasised an important distinction
between discriminative instruments intended to
be particularly valid in distinguishing between
respondents at a point in time and evaluative
instruments that need to be particularly sensitive
to changes within individuals over time in the
context of clinical trials. The question at the
beginning of this section emphasises that
patient-based outcome measures changes of
importance to patients. Whilst responsiveness
as such is not defined in terms of importance
to patients, this would seem an important
specification in relation to patient-based
outcome measures.