RESEARCH METHOD COHEN ok

VALIDITY AND RELIABILITY IN TESTS 159
reliability apply. Here steps are taken to ensure that
observers enter data into the appropriate categories
consistently (i.e. intra- and inter-rater reliability)
and accurately. Further, to ensure validity, a
pilot must have been conducted to ensure
that the observational categories themselves are
appropriate, exhaustive, discrete, unambiguous
and effectively operationalize the purposes of the
research.
Validity and reliability in tests
The researcher will have to judge the place and
significance of test data, not forgetting the problem
of the Hawthorne effect operating negatively or
positively on students who have to undertake
the tests. There is a range of issues which might
affect the reliability of the test – for example,
the time of day, the time of the school year,
the temperature in the test room, the perceived
importance of the test, the degree of formality
of the test situation, ‘examination nerves’, the
amount of guessing of answers by the students
(the calculation of standard error which the test
demonstrates feature here), the way that the test is
administered, the way that the test is marked, the
degree of closure or openness of test items. Hence
the researcher who is considering using testing
as a way of acquiring research data must ensure
that it is appropriate, valid and reliable (Linn
1993; Borsboom et al.2004).
Wolf (1994) suggests four main factors that
might affect reliability: the range of the group that
is being tested, the group’s level of proficiency,
the length of the measure (the longer the test the
greater the chance of errors), and the way in which
reliability is calculated. Fitz-Gibbon (1997: 36)
argues that, other things being equal, longer tests
are more reliable than shorter tests. Additionally
there are several ways in which reliability might be
compromised in tests. Feldt and Brennan (1993)
suggest four types of threat to reliability:
individuals: their motivation, concentration,
forgetfulness, health, carelessness, guessing,
their related skills (e.g. reading ability, their
usedness to solving the type of problem set, the
effects of practice)
situational factors: the psychological and
physical conditions for the test – the context
test marker factors: idiosyncrasy and subjectivity
instrument variables: poor domain sampling,
errors in sampling tasks, the realism of the
tasks and relatedness to the experience of the
testees, poor question items, the assumption or
extent of unidimensionality in item response
theory, length of the test, mechanical errors,
scoring errors, computer errors.
Sources of unreliability
There are several threats to reliability in tests and
examinations, particularly tests of performance
and achievement, for example (Cunningham
1998; Airasian 2001), with respect to examiners
and markers:
errors in marking: e.g. attributing, adding and
transfer of marks
inter-rater reliability: different markers giving
different marks for the same or similar pieces
of work
inconsistency in the marker: e.g. being harsh in
the early stages of the marking and lenient in
the later stages of the marking of many scripts
variations in the award of grades: for work that
is close to grade boundaries, some markers may
place the score in a higher or lower category
than other markers
the Halo effect: a student who is judged
to do well or badly in one assessment is
given undeserved favourable or unfavourable
assessment respectively in other areas.
With reference to the students and teachers
themselves, there are several sources of unreliability:
Motivation and interest in the task have a
considerable effect on performance. Clearly,
students need to be motivated if they are
going to make a serious attempt at any test
that they are required to undertake, where
motivation is intrinsic (doing something for
its own sake) or extrinsic (doing something
for an external reason, e.g. obtaining a
Chapter 6