Evaluating Patient-Based Outcome Measures - NIHR Health ...

26
Criteria for selecting a patient-based outcome measure
the product of these variables’ reliability coefficients.
Therefore, given typical levels of reliability
of patient-based variables, a correlation coefficient
of 0.60 may be strong evidence in support of
construct validity (McDowell and Newell, 1996).
Because there is considerable vagueness and
variability in the levels of correlation coefficients
that authors cite as evidence of construct validity
of new instruments, McDowell and Jenkinson
(1996) recommend that expected correlations
should be specified at the outset of studies to test
instruments’ validity in order that it be possible
for validity to be disproved.
The most sophisticated form of testing construct
validity is so-called ‘convergent and discriminant
validity’ (Campbell and Fiske, 1959). This approach
requires postulating that an instrument that we
wish to test should have stronger relationships
with some variables and weaker relationships with
others. A new measure of mobility should correlate
more strongly with existing measures of physical
disability than with existing measures of emotional
well-being. Essentially correlations are expected to
be strongest with most related constructs and weakest
with most distally related constructs. Typically,
construct validity is examined by inspecting correlations
of a new measure against a range of other
evidence such as, disease staging, performance
status, clinical or laboratory evidence of disease
severity, illness behaviour, use of health services and
related constructs of well-being (Spitzer et al., 1981;
Fletcher, 1988; Sullivan et al., 1990; Aaronson et al.,
1993). As an example of convergent and discriminant
validation, Sullivan and colleagues (1990)
examined validity of the scales of the SIP for use in
rheumatoid arthritis with the expectation that the
SIP physical function score should correlate most
with various measures of disease severity and the
SIP psychosocial scale should correlate most with
other measures of mood and psychological wellbeing.
Conversely, measures of physical function
were expected to correlate less with measures of
physical mood. Similarly, Morrow and colleagues
(1992) examined the convergent-discriminant
validity of the Functional Living Index-Cancer
(FLIC) by examining relationships to other
variables. As predicted the subscales of FLIC to
measure gastrointestinal symptoms was significantly
related to other ratings by patients of nausea and
vomiting, but correlations were close to zero
between psychological and social subscales of
FLIC and patients’ separate reports of nausea
and vomiting.
The most demanding form of convergent and
discriminant validity is the multitrait-multimethod
matrix (Campbell and Fiske, 1959) in which two
unrelated constructs are measured by two or more
methods. In essence, it is expected that different
measures of a single underlying trait should
correlate most and different measures of different
constructs least. Whilst potentially powerful in
psychometric test development, it is difficult to
apply to patient-based outcome measures because
of problems of obtaining alternative methods of
measuring constructs.
Most instruments to assess outcomes from the
patient’s point of view are multi-dimensional.
They, for example, assess physical, psychological
and social aspects of an illness within one questionnaire.
This internal structure of an instrument can
also be considered a set of assumed relationships
between underlying constructs. At the very least,
an instrument with sub-scales has implied that the
instrument measures different underlying constructs
by providing different sub-scales, rather than
requiring that all items should simply be added to
produce one score of one underlying construct.
Normally instruments with multiple scales also
assume particular underlying relationships for
the constructs measured by the instrument; for
example, scales of different aspects of emotional
response to illness will correlate more with each
other than with scales assessing physical function.
This internal structure of instruments has also to
be established by construct validation. The most
common of methods for this purpose is statistical,
particularly the use of factor analysis. Thus, factor
analysis is often considered an aspect of
construct validity.
To simplify, factor analysis is the analysis of patterns
of, in this field, items that go together to assess
single underlying constructs. Typically, statistical
analysis of answers of a sample of respondents to
a pool of questionnaire items is used to reveal two
or more sub-scales assessing distinct dimensions.
In essence, the data can be checked to see whether
individual questionnaire items correlate more with
the scale of which they are a part and less with
other scales to which they do not belong. Examples
of instruments in which factor analysis has played
a key role in establishing the internal structure
of sub-scales include the Profile of Mood States
(McNair et al., 1992) and the St George’s
Respiratory Questionnaire (Jones et al., 1992).
However, there are problems with factor analytic
methods used in this context. Fayers and Hand
(1997) provide important arguments and evidence
against excessive reliance upon factor analysis
alone to determine or evaluate the construct