Evaluating Patient-Based Outcome Measures - NIHR Health ...

and with the summed score of the total of items in
the same scale.
Internal consistency can be measured in a number
of different ways. One approach – split-half reliability
– is randomly to divide the items in a scale
into two groups and to assess the degree of agreement
between the two halves. The two halves
should correlate highly. An extension of this
principle is Coefficient alpha, usually referred
to as Cronbach’s alpha, which essentially estimates
the average level of agreement of all the possible
ways of performing split-half tests (Cronbach,
1951). The higher the alpha, the higher the
internal consistency. However, it is also possible
to increase Cronbach’s alpha by increasing the
number of items, even if the average level of correlation
does not change (Streiner and Norman,
1995). Also if the items of a scale correlate perfectly
with each other, it is likely that there is some redundancy
among items, and also a possibility that the
items together are addressing a rather narrow
aspect of an attribute. For these reasons, it is
suggested that Cronbach’s alpha should be above
0.70 but not higher than 0.90 (Nunnally and
Bernstein, 1994; Streiner and Norman, 1995).
Another approach to establish internal consistency
of items is simply to examine the correlation of
individual items to the scale as a whole, omitting
that item. Steiner and Norman (1995) cite a normal
rule of thumb that items should correlate
at least 0.20 with the scale.
A balance needs to be struck between satisfactory
internal consistency and a measure that is too
homogeneous because it measures a very restricted
aspect of a phenomenon. Kessler and Mroczek
(1995) provide an important argument with
illustrative evidence against excessive emphasis
upon internal reliability. Essentially, they advocate
a shift toward selecting items in a scale in a way
that maximises the additional information content
of each item, a principle of minimal redundancy.
As Kessler and Mroczek argue, investigators usually
use factor-analytic techniques to identify items
that particularly correlate and therefore yield
high internal reliability. They argue for the use
of regression and related techniques to replace
factor analytic methods of developing scales. Their
argument begins with a hypothetical long list of
questionnaire items that together may be considered
the complete and true measure of some
phenomenon, say pain, or mobility. The objective
of scale development is to produce a small sub-set
that reliably measures the full set. Factor analysis
will identify the items from the longer set that
Health Technology Assessment 1998; Vol. 2: No. 14
most correlate with each other to produce an
internally reliable scale. If, however, one conceives
of the full set of items as the dependent variable
and uses regression analysis with individual items
as the independent variables, it is possible to
identify a small sub-set of items that explains most
of the variance in the full set. As they express it,
‘most of the variance in the long form can then
usually be reproduced with a small subset of the
scale items’ (Kessler and Mroczek, 1995: AS112).
The argument is then illustrated with data comprising
32 items used to screen for psychological
distress in a population survey. They select items
to form a short version of the scale from the full
set by two methods: factor analytic methods to
maximise internal reliability and regression to
minimise redundancy. Results from regression
produce consistently higher correlations of the
sub-scale with the total variance in the full set
of 32 items. On the other hand, factor analytic
techniques produce consistently higher
internal reliability.
It has been argued that excessive attention to
internal reliability can result in the omission of
important items, particularly those that reflect
the complexity and diversity of a phenomenon
(Donovan et al., 1993). Certainly, obtaining the
highest possible reliability coefficient should not
be the sole objective in developing or selecting an
instrument because the reductio ad absurdum of
this principle would be an instrument with high
reliability produced by virtually identical items.
Reproducibility
Reproducibility more directly evaluates whether
an instrument yields the same results on repeated
applications, when respondents have not changed
on the domain being measured. This is assessed
by test–retest reliability. The degree of agreement
is examined between scores at a first assessment
and when reassessed. There is no exact agreement
about the length of time that should elapse; it
needs to be a sufficient length of time that
respondents are unlikely to recall their previous
answers, but not so long that actual changes in
the underlying dimension of health have
occurred. Streiner and Norman (1995) suggest
that the usual range of time elapsed between
assessments tends to be between 2 and 14 days.
One way of checking whether the sample has
experienced underlying changes in health that
would reduce the apparent reproducibility of
an instrument is also to administer a transition
question at the second assessment (‘Is your
health better, the same or worse than at the
last assessment?’).
23