Evaluating Patient-Based Outcome Measures - NIHR Health ...

34
Criteria for selecting a patient-based outcome measure
an instrument has a high level of precision because
scores are expressed as percentages, the range of
actual possible values may still be quite small and
scores are in no sense interval.
By contrast to such common-sense based methods
of weighting are efforts directly to assess the
relative severity or undesirability of different
states. The SIP is an example of an instrument
with a more sophisticated and more explicitly
based weighting system. Once the questionnaire
items for the instrument had been identified, a
panel of patients, health professionals and preprofessional
students used category scaling to
assign weights to items by making judgements
of the relative severity of dysfunction of items
(Bergner et al., 1976). To illustrate the impact of
this weighting approach to questionnaire items,
in the English version of the instrument, the most
severe items in the body care and movement scale
are ‘I am in a restricted position all the time’
(–124) and ‘I do not have control of my bowels’
(–124), whereas the least severe items are ‘I dress
myself but do so very slowly’ (–043) and ‘I am very
clumsy’ (–047). Separate weighting exercises on
American and English versions by separate panels
in the two language communities arrived at very
similar weightings for items for the SIP (Patrick
et al., 1985). Other instruments that include such
explicitly derived weighting systems include the
Nottingham Health Profile (NHP), QWB and
EQ-5D.
There are two particularly striking problems if the
numerical values used in different patient-based
outcomes are examined. On the one hand, many
instruments use methods of scoring items that are
deceptively simple. Although apparently simple,
such scoring nevertheless may require strong
assumptions; for example that the difference
between the first and second responses is regarded
as the same as the difference between the fourth
and fifth response in a five-point Likert scale, if
scores are analysed as interval scale scores.
On the other hand, the other most striking
problem is that scoring methods that attempt
directly to estimate the values of such response
categories such as in the SIP by weighting systems,
risk being deceptively precise. Their numerical
exactness might lend pseudo-precision to an
instrument. For investigators examining the
numerical values of instruments, it is sensible to
treat all scoring methods as weighted, differing
only in how transparent weights are, and to look
beyond superficial aspects of precision to examine
how weightings have been derived and validated.
More pragmatically, it is appropriate to ask
whether weighting systems make a difference
(Björk and Roos, 1994). Sensitivity analysis may
reveal that they make no significant difference to
results. For example, Jenkinson and colleagues
(1991) analysed patterns of change over time in
health status for patients with rheumatoid arthritis
by means of the FLP and NHP. Sensitivity to
change as indicated by a battery of other clinical
and laboratory measures was very similar, whether
weighted or unweighted (items valued as ‘1’ or
‘0’) versions of the instruments were used. Other
studies have similarly suggested that weighted
scales may not improve upon the sensitivity of
unweighted scales (O’Neill et al., 1996).
The response format of a patient-based outcome
measure to some extent determines the kinds of
statistical tests that may be used on it. This is here
considered an aspect of precision in the sense that
many instruments contain items that are at best
ordinal in form (i.e. questionnaire items where
there is an implied rank to responses: ‘very often’,
‘quite often’ etc.) but not interval (i.e. where the
interval between responses is of known value) or
ratio (where there is a meaningful zero point). It
might be argued that instruments that have only
ordinal level measurement properties are capable
of less precision (Haig et al., 1986). Certainly, a
review of the statistical properties of a series of
health status scales published in the literature
concluded that the majority of scales were presented
and analysed as if based on interval-level
when this property was not established (Coste et al.,
1995). Whilst it might be argued that an advantage
of visual analogue scale over Likert format answers
is that it would enable more extensive use of
parametric statistics, this needs to be balanced
against the lower acceptability of visual analogue
scale techniques and the risk of pseudo-precision
that this technique involves (Aaronson, 1989).
Mackenzie and Charlson (1986) reviewed trials
employing ordinal scales in three medical journals
over a 5-year period and found that many measures
purporting to be ordinal were not. For example,
values for the items of a scale were not truly
hierarchical, so it was not clear whether lower
numerical scores truly reflected worse
underlying states.
As Streiner and Norman (1995) point out, there is
a large and unresolved literature as to the propriety
of using interval level statistics when it is unclear
that there is a linear relationship of a measure to
the underlying phenomenon. In practice, there
may be many circumstances where cautious