Evaluating Patient-Based Outcome Measures - NIHR Health ...

assumption of interval properties with ordinally
based data does not seriously mislead.
One quite practical illustration of the need for
caution is in the calculation of sample sizes for
trials using patient-based outcome measures as a
primary end-point. Using the SF-36 as example,
Julious and colleagues (1995) show that for those
dimensions of SF-36 where the distribution of
scores are highly skewed, sample size calculations
are very different if parametric and non-parametric
methods are used to estimate required sample size.
Distribution of items over true range
The items and scores of different instruments may
vary in how well they capture the full underlying
range of problems experienced by patients. It is
not easy to examine the relationship between the
distinctions made by a measuring instrument and
the true distribution of actual experiences, for the
obvious reason that one usually does not have access
to the true distribution other than through one’s
measuring instrument. Nevertheless a number of
arguments have been put forward that show that
this is a real issue. Kessler and Mroczek (1995) have
illustrated this problem by means of an instrument
to measure psychological distress. They showed that
it was possible to select short form versions of small
numbers of items taken from a full set of 32 items
measuring distress that, whilst all having the same
reliability as the full 32-item scale and agreeing
strongly with the total scale, differed markedly in
ability to discriminate between distressed and not
distressed individuals at different levels in an overall
continuum of severity of psychological distress. One
short form version was most discriminating at low
levels of distress, and so on. A comparison that
makes this point more intuitively understandable
would be a range of intelligence tests, with, at one
extreme a test that could distinguish the very
cleverest as a category from all others who would
be grouped together. At the opposite extreme,
tests would sensitively distinguish those with very
low intelligence from all others. The ideal test,
whether of health or intelligence would have
equal precision at every level.
Another illustration of the problematic relationship
between items and the ‘true’ distribution of
what is being measured is provided by Stucki and
colleagues’ (1996) analysis of SF-36 physical ability
scores in patients undergoing total hip replacement
surgery. They showed that many of the items
of this scale represent moderate levels of difficulty
for patients to perform (e.g. ‘bending, kneeling
or stooping’); by contrast, there are only a few
items that almost everyone could do with no
Health Technology Assessment 1998; Vol. 2: No. 14
difficulty (e.g. ‘bathing and dressing yourself’)
and only a few items that were difficult for the
majority to perform (e.g. ‘walking more than a
mile’). A direct consequence of this is that patients
passing a difficulty level in the middle of this scale
of the SF-36 are more likely to have larger change
scores than patients undergoing change at either
the top or bottom of the range of difficulty of
items, simply because of the larger number of items
assessing moderate levels of difficulty. A similar set
of observations about the ‘maldistribution’ of items
of this scale of SF-36 was made by another group of
investigators (Haley et al., 1994). The most obvious
consequences of this effect are two-fold: (i) the
meaning of change scores for instruments may
need to be interpreted in the knowledge of baseline
scores of patients and (ii) instruments may
need to ensure a more even distribution of
items across the range of levels of severity
or difficulty.
The distribution of items for the physical scale
of SF-36 was examined by Stucki and colleagues
(1996) by a variety of statistical methods including
Rasch analysis (discussed on page 37) to address
the issue of distribution. It should also be possible
for investigators to inspect instruments at a more
informal and intuitive level to consider whether
there may be problems of the distribution of items
in relation to the intended trial and patient group,
to see whether particular levels of severity of illness
are under-represented.
Ceiling and floor effects in relation
to precision
The problem of ceiling and floor effects has already
been considered in the context of responsiveness.
They are mentioned again here because, essentially
they may be viewed as problems arising from the
precision and distribution of items in questionnaires.
Studies were cited above in the context
of responsiveness (Bindman et al., 1990; Gardiner
et al., 1993) in which convincing evidence was
found that some instruments did not allow patients
with poor health status to report further deterioration.
Questionnaires were found not to include
items to capture the poorest levels of health. Potential
solutions, depending on the overall format of
the instrument, include adding a response category
such as ‘extremely poor’ to questions and increasing
the range of items, particularly addressing
more severe experiences. Bindman and colleagues
(1990) suggest adding transition questions which
directly invite respondents to say whether they are
worse or better than at a previous assessment.
However, this is an unwieldy solution in terms
of the formatting of questionnaires.
35