Evaluating Patient-Based Outcome Measures - NIHR Health ...

30
Criteria for selecting a patient-based outcome measure
considered small, 0.5 as medium and 0.8 or greater
as large (Cohen, 1977, Kazis et al., 1989).
Standardised response mean
An alternative measure is the SRM. It only differs
from an effect size in that the denominator is the
standard deviation of change scores in the group in
order to take account of variability in change rather
than baseline scores (Liang et al., 1990). Because
the denominator in the SRM examines response
variance in an instrument whereas the effect size
does not, Katz and colleagues (1992) consider
that the SRM approach is more informative.
Modified standardised response mean
A third method of providing a standardised
expression of responsiveness is that of Guyatt and
colleagues (1987b). As with effect sizes and SRMs,
the numerator of this statistic is the mean change
score for a group. In this case, the denominator is
the standard deviation of change scores for individuals
who are identified by other means as stable
(Tuley et al., 1991). This denominator provides an
expression of the inherent variability of changes in
an instrument, ‘an intuitive estimate of background
noise’ (Liang, 1995). Unlike the two other expressions,
this method requires independent evidence
that patients are indeed stable, in the form of a
transition question asked at follow-up. MacKenzie
and colleagues (1986b) used a transition index
and the modified SRM to test the responsiveness
of the SIP.
Relative efficiency
Another approach is to compare the responsiveness
of health status instruments when used in studies
of treatments widely considered to be effective, so
that it is very likely that significant changes actually
occur. As applied by Liang and colleagues (1985)
who developed this approach, the performance
of different health status instruments is compared
to a standard instrument amongst patients who
are considered to have experienced substantial
change. Thus they asked patients to complete a
number of health status questionnaires before
and after total joint replacement surgery. Health
status questionnaires were considered most
responsive that produced the largest paired t-test
score for pre and post surgical assessments. Liang
and colleagues (1985) produce a standardised
version of the use of t-statistics (termed ‘relative
efficiency’), the square of the ratio of t-statistic
for two instruments being compared. As noted
earlier, much of the data in this field is nonparametric,
so that t statistics are not appropriate
and non-parametric forms of relative efficiency
need to be used.
Sensitivity and specificity of
change scores
Another approach to assessing responsiveness
is to consider the change scores of a health status
instrument as if they were a screening test to
detect true change (Deyo and Inui, 1984). In
other words, data for a health status measure
are examined in terms of the sensitivity of change
scores produced by an instrument (proportion
of true changes detected) and specificity of
change scores (proportion of individuals who
are truly not changing, detected as stable). This
method requires that investigators identify
somewhat arbitrarily a specific change score
that will be taken as of interest to examine, say an
improvement of five points between observations.
The sample is then divided according to whether
or not they have reported five points improvement
or not. Some external standard is also needed to
determine ‘true’ change; commonly it is a
consensus of the patient’s and or clinician’s
retrospective judgement of change (Deyo and
Inui, 1984). Essentially, the extent of agreement
is then examined between change as defined in
this hypothetical case as more than five points
change and independent evidence of whether
individuals have changed. The same analysis is
then provided for specificity. Scores of less than
five points are counted as ‘unchanged’. Individuals
scores are then examined to determine how much
this classification of individuals as unchanged
agrees with independent evidence
that they have not changed.
This approach permits an assessment of the
sensitivity and specificity of different amounts of
change registered by an instrument. For example,
an instrument which, for the sake of simplicity,
has five possible scores (1–5), when applied on
two occasions over time, may either produce
identical scores on both occasions or register
up to four points of change (for example a
change from ‘5’ to ‘1’). If one has an external
standard of whether change truly occurred, such
as a consensus of patient’s and doctor’s opinion,
one can assess how sensitive and specific to the
external evidence of change are changes of one
point, two points etc. As sensitivity improves,
specificity may deteriorate.
Receiver-operating characteristics
Deyo and Centor (1986) extend the principle
of measuring the sensitivity and specificity of
an instrument against an external criterion by
suggesting that information be synthesised into
receiver-operating characteristics. Like the simpler
version just described in the previous section, this