RESEARCH METHOD COHEN ok

VALIDITY AND RELIABILITY IN EXPERIMENTS 155
interviewer. Powerful interviewees are usually busy
people and will expect the interviewer to have read
the material that is in the public domain.
The issues of reliability do not reside solely
in the preparations for and conduct of the
interview; they extend to the ways in which
interviews are analysed. For example, Lee (1993)
and Kvale (1996: 163) comment on the issue
of ‘transcriber selectivity’. Here transcripts of
interviews, however detailed and full they
might be, remain selective, since they are
interpretations of social situations. They become
decontextualized, abstracted, even if they record
silences, intonation, non-verbal behaviour etc.
The issue, then, is how useful they are to
researchers overall rather than whether they are
completely reliable.
One of the problems that has to be considered
when open-ended questions are used in the
interview is that of developing a satisfactory
method of recording replies. One way is to
summarize responses in the course of the interview.
This has the disadvantage of breaking the
continuity of the interview and may result in
bias because the interviewer may unconsciously
emphasize responses that agree with his or her
expectations and fail to note those that do not. It
is sometimes possible to summarize an individual’s
responses at the end of the interview. Although
this preserves the continuity of the interview, it
is likely to induce greater bias because the delay
may lead to the interviewer forgetting some of
the details. It is these forgotten details that are
most likely to be the ones that disagree with the
interviewer’s own expectations.
Validity and reliability in experiments
As we have seen, the fundamental purpose of
experimental design is to impose control over
conditions that would otherwise cloud the true
effects of the independent variables upon the
dependent variables.
Clouding conditions that threaten to jeopardize
the validity of experiments have been identified
by Campbell and Stanley (1963), Bracht and
Glass (1968) and Lewis-Beck (1993), conditions
that are of greater consequence to the validity
of quasi-experiments (more typical in educational
research) than to true experiments in which random
assignment to treatments occurs and where
both treatment and measurement can be more
adequately controlled by the researcher. The following
summaries adapted from Campbell and
Stanley (1963), Bracht and Glass (1968) and
Lewis-Beck (1993) distinguish between ‘internal
validity’ and ‘external validity’. Internal validity
is concerned with the question, ‘Do the
experimental treatments, in fact, make a difference
in the specific experiments under scrutiny’.
External validity, on the other hand, asks the
question, ‘Given these demonstrable effects, to
what populations or settings can they be generalized’
(see http://www.routledge.com/textbooks/
9780415368780 – Chapter 6, file 6.8. ppt).
Threats to internal validity
History: Frequently in educational research,
events other than the experimental treatments
occur during the time between pretest and
post-test observations. Such events produce
effects that can mistakenly be attributed to
differences in treatment.
Maturation: Between any two observations
subjects change in a variety of ways. Such
changes can produce differences that are
independent of the experimental treatments.
The problem of maturation is more acute in
protracted educational studies than in brief
laboratory experiments.
Statistical regression: Like maturation effects,
regression effects increase systematically with
the time interval between pretests and
post-tests. Statistical regression occurs in
educational (and other) research due to the
unreliability of measuring instruments and to
extraneous factors unique to each experimental
group. Regression means, simply, that subjects
scoring highest on a pretest are likely to score
relatively lower on a post-test; conversely,
those scoring lowest on a pretest are likely
to score relatively higher on a post-test. In
short, in pretest-post-test situations, there is
Chapter 6