RESEARCH METHOD COHEN ok

VALIDITY AND RELIABILITY IN QUESTIONNAIRES 157
identified under the section on ‘Threats to
internal validity’.
Invalidity or unreliability of instruments: The
use of instruments which yield data in which
confidence cannot be placed (see below on
tests).
Ecological validity, anditspartner,theextent
to which behaviour observed in one context
can be generalized to another: Hammersley
and Atkinson (1983: 10) comment on the
serious problems that surround attempts to
relating inferences from responses gained under
experimental conditions, or from interviews, to
everyday life.
By way of summary, we have seen that an
experiment can be said to be internally valid
to the extent that, within its own confines, its
results are credible (Pilliner 1973); but for those
results to be useful, they must be generalizable
beyond the confines of the particular experiment.
In a word, they must be externally valid
also: see also Morrison (2001b) for a critique
of randomized controlled experiments and the
problems of generalizability. Pilliner (1973) points
to a lopsided relationship between internal and
external validity. Without internal validity an
experiment cannot possibly be externally valid.
But the converse does not necessarily follow; an
internally valid experiment may or may not have
external validity. Thus, the most carefully designed
experiment involving a sample of Welsh-speaking
children is not necessarily generalizable to a target
population which includes non-Welsh-speaking
subjects.
It follows, then, that the way to good
experimentation in schools, or indeed any other
organizational setting, lies in maximizing both
internal and external validity.
Validity and reliability in questionnaires
Validity of postal questionnaires can be seen
from two viewpoints (Belson l986). First, whether
respondents who complete questionnaires do
so accurately, honestly and correctly; and
second, whether those who fail to return their
questionnaires would have given the same
distribution of answers as did the returnees. The
question of accuracy can be checked by means
of the intensive interview method, a technique
consisting of twelve principal tactics that include
familiarization, temporal reconstruction, probing
and challenging. The interested reader should
consult Belson (1986: 35-8).
The problem of non-response – the issue of
‘volunteer bias’ as Belson (1986) calls it – can,
in part, be checked on and controlled for,
particularly when the postal questionnaire is sent
out on a continuous basis. It involves followup
contact with non-respondents by means of
interviewers trained to secure interviews with
such people. A comparison is then made between
the replies of respondents and non-respondents.
Further, Hudson and Miller (1997) suggest several
strategies for maximizing the response rate to
postal questionnaires (and, thereby, to increase
reliability). They involve:
including stamped addressed envelopes
organizing multiple rounds of follow-up to
request returns (maybe up to three follow-ups)
stressing the importance and benefits of the
questionnaire
stressing the importance of, and benefits to,
the client group being targeted (particularly if
it is a minority group that is struggling to have
avoice)
providing interim data from returns to nonreturners
to involve and engage them in the
research
checking addresses and changing them if
necessary
following up questionnaires with a personal
telephone call
tailoring follow-up requests to individuals
(with indications to them that they are
personally known and/or important to the
research – including providing respondents
with clues by giving some personal information
to show that they are known) rather than
blanket generalized letters
detailing features of the questionnaire itself
(ease of completion, time to be spent,
Chapter 6