Evaluating Patient-Based Outcome Measures - NIHR Health ...

that nothing relevant was lost to subsequent
examination. Time restrictions curtailed the
total number of articles reviewed to 209, with a
decision taken one month into reading, to time
order the remaining publications, giving precedence
to articles published after 1990. However,
all 247 articles were skim read and important
articles published in the 1980s and 1970s were
transcribed. The qualitative analysis involved
transcribing key points and summary statements
from the 209 articles into files under the following
topics: general issues and concepts, selection
criteria, validity, reliability, responsiveness,
acceptability, feasibility, utility, comparison
of instruments, numerical properties
and weights.
An additional benefit of this stage was the identification
of an extensive list of possible search terms
for the next phase of the literature search. Possible
search terms were sought from the full text of
articles and not just keywords.
Electronic searching (step 5)
The electronic literature search was used to achieve
two objectives. It would validate the first phase of
the literature review strategy, by cross checking how
many of the 223 articles (24 of the 247 references
were either books, book chapters or unpublished)
previously obtained, appeared in the results of the
electronic search. The inspection revealed that
58% (130/223) of articles were represented in the
electronic literature search. This low figure is
consistent with Chalmers et al (1992), who found
that MEDLINE searching only retrieved half of the
relevant studies, with those missed actually contained
within MEDLINE but inaccurately indexed
or described by the author or the coding procedure.
More importantly, the electronic search
provided a substantial number of abstracts to
review in order to identify any publications that
provided new dimensions or additional perspectives
to issues already uncovered in the first stage
of the literature review.
The electronic search did not initially have any date
restrictions and went back as far as the databases
would allow. However, the actual review of abstracts
was limited to 1991–1996 in order to capture only
up to date information. The electronic literature
search was limited to English introducing a
selective bias. An attempt was made to reduce the
total number of records retrieved by only searching
in title and keywords but this was found to be too
narrow and risked missing many references.
Health Technology Assessment 1998; Vol. 2: No. 14
The electronic search was carried out in MEDLINE
(1966–1996), CINAHL (Cumulative Index of
Nursing and Allied Health, 1982–1996), PsychLIT
(1974–1996), Sociofile (1974–1996), Econlit
(1969–1996), all of which were on the University
of Oxford’s electronic reference library (ERL) and
accessed using Winspirs/Silverplatter software via
the University network. Additionally, the EMBASE
database (1990–1996) was accessed using the Bath
Information & Data Services (BIDS). As the BIDS
and the ERL databases are assessed via two different
pathways it was not possible to perform a
combined search.
The electronic search strategy combined the term
‘patient-based outcome*’ and its synonyms with
related methodological terms to retrieve only
publications that looked at methods of evaluating
patient-based outcome measures. The search terms
used after refinement of an original set are shown
in Box 6. The terms in search one were combined
using the ‘or’ connector and the same done for
search two. The results of the two searches were
then combined using the ‘and’ connector.
BOX 6 Electronic search terms
Search one: retrieval of all records using the terms
Patient-based outcome *
Health status Subjective health status
Health status indicator * Health status assessment
Quality of life Disability scale
Health-related Performance status
quality of life
Functional status
Search two: retrieval of all records using the terms
Methodol *
Effect size
Psychometric * Sensitivity to change
Validity Reproducibility
Reliability Acceptability
Responsiveness Utility measure *
Combine search one and two with ‘and’ connector
* = truncation symbol
The original list of search terms was run in the
MEDLINE database and refined by eliminating
terms that retrieved a high number of falsepositives.
This was done by reviewing a sample of
the records retrieved from individual search terms
and estimating the number of false hits. The
exclusion of terms was then discussed and verified
at project meetings. A summary of the excluded
terms is shown in Table 5.
67