Evaluating Patient-Based Outcome Measures - NIHR Health ...

supplementary questions, favoured by Gill and
Feinstein, or by use of previously validated instruments,
favoured by Guyatt and Cook. With the
former method, the patient is directly asked his or
her judgements in an open ended way, minimally
constrained by predetermined questionnaire items.
The latter method largely relies on predetermined
questions validated in previous research. To some
extent, the dispute reflects differences of philosophy
in how to assess patients’ experiences that
cannot be resolved by current evidence. Their
common ground is that appropriate measures
in a trial are those that particularly address
patients’ concerns.
If clinical trials do use instruments that have
been developed by psychometric criteria, they may
still be flawed. Psychometric principles (reliability,
validity and responsiveness) are further explored in
later sections of this chapter. Coste and colleagues
(1995) reviewed 46 studies published in six medical
journals over the period 1988 to 1992, in which
scales or indices were used to measure constructs
such as QoL and physical function. In less than
a fifth of studies did they regard construct and
content validity to have been adequately addressed.
In only a quarter of studies was adequate attention
given to reliability. They describe many of the
instruments as being ‘ad-hoc’. Another review, in
which independent assessment of studies was made,
found that in randomised clinical trials that used
patient-based outcome measures, only 10 out
of 55 trials used instruments with established
validity and responsiveness (Veldhuyzen van
Zanten, 1991).
Instruments do need to be clearly focused on
patients’ concerns and to be psychometrically
sound to be considered as appropriate based
measures of outcome in trials. However, these
properties do not exhaust the list of considerations
in determining whether an instrument is
appropriate for any particular trial.
Most obviously, an instrument needs to fit the
purpose of a trial. This purpose needs to be specified
as precisely as is reasonable and outcome
measures selected accordingly (Liang et al., 1982;
Fallowfield, 1996). A Lancet editorial reiterates
part of Gill and Feinstein’s critique and argues
that the rationale for selection of outcome
measures is often not clear (Editorial, 1995).
Investigators are uncritically inserting questionnaires
into their trials without careful consideration
of content and relevance to the purpose
of the trial. This will primarily mean that the
instrument selected must be particularly relevant
Health Technology Assessment 1998; Vol. 2: No. 14
to the health problem and proposed intervention
as possible.
As already stated, this judgement involves
simultaneous examination of, on the one hand,
the specific treatment and patient group being
investigated and on the other the content of
instruments in order optimally to match instrument
to objective (Guyatt et al., 1991). Investigators have
to determine how narrow or broad a measure of
health they require. An intervention may be evaluated
in which only very accurate assessment of,
say, mobility or pain is needed. More often, investigators
are uncertain of all the likely consequences
of their intervention and opt for a broader
measure or set of measures to capture more
unexpected consequences.
A useful distinction can also be made between
‘proximal’ and ‘distal’ outcome measures (Brenner
et al., 1995). Brenner and colleagues suggest that
it is helpful to think of a continuum of outcomes
in relation to any disease and its treatment. Outcomes
that are proximal most closely represent
manifestations of the disease itself, for example,
pain and stiffness in arthritis. Slightly less proximal
and removed from disease are aspects of physical
functioning. Distal outcomes are those most
removed from disease such as, for example, life
satisfaction. The value of the continuum is in
making explicit that as one incorporates more
distal outcome measures in a trial, the less likely
it is that the intervention will have greater effects
on those outcomes in the study group compared
to controls. On the other hand, they suggest that
the more effective an intervention, the greater
the likelihood will be that more distal outcome
measures will be relevant. Circumstances of a
trial will dictate whether distal as well as proximal
effects are of interest and therefore important to
monitor. It is a useful discipline to consider this
continuum in selecting outcome measures.
It is impossible to be clear simply from the titles
of instruments or of their constituent scales and
dimensions what precisely is being measured.
Titles of instruments and constituent scales of
instruments cannot be taken at face value, and
cannot therefore be assumed to be appropriate on
the basis of title alone (Ware, 1987). This is most
obviously the case for dimensions of instruments
which refer very broadly to, for example, ‘social
function’. Dimensions of instruments assessing this
aspect of patients’ experiences may refer to quite
disparate issues. Thus two patient-based outcome
measures for cancer provide ‘social scores’ but only
weakly agree with each other when patients
21