RESEARCH METHOD COHEN ok

VALIDITY AND RELIABILITY IN TESTS 161
Marking practices are not always reliable,
markers may be being too generous, marking
by effort and ability rather than performance.
The context in which the task is presented
affects performance: some students can perform
the task in everyday life but not under test
conditions.
With regard to the test items themselves, there may
be problems (e.g. test bias):
The task itself may be multidimensional,
for example, testing ‘reading’ may require
several components and constructs. Students
can execute a Mathematics operation in the
Mathematics class but they cannot perform the
same operation in, for example, a Physics class;
students will disregard English grammar in a
Science class but observe it in an English class.
This raises the issue of the number of contexts
in which the behaviour must be demonstrated
before a criterion is deemed to have been
achieved (Cohen et al. 2004).Thequestionof
transferability of knowledge and skills is also
raised in this connection. The context of the
task affects the student’s performance.
The validity of the items may be in question.
The language of the assessment and the
assessor exerts an influence on the testee, for
example if the assessment is carried out in the
testee’s second language or in a ‘middle-class’
code (Haladyna 1997).
The readability level of the task can exert an
influence on the test, e.g. a difficulty in reading
might distract from the purpose of a test which
is of the use of a mathematical algorithm.
The size and complexity of numbers or
operations in a test (e.g. of Mathematics) might
distract the testee who actually understands the
operations and concepts.
The number and type of operations and stages
to a task: the students might know how to
perform each element, but when they are
presented in combination the size of the task
can be overwhelming.
The form and presentation of questions affects
the results, giving variability in students’
performances.
Asingleerrorearlyoninacomplexsequence
may confound the later stages of the sequence
(within a question or across a set of questions),
even though the student might have been able
to perform the later stages of the sequence,
thereby preventing the student from gaining
credit for all she or he can, in fact, do.
Questions might favour boys more than girls or
vice versa.
Essay questions favour boys if they concern
impersonal topics and girls if they concern
personal and interpersonal topics (Haladyna
1997; Wedeen et al.2002).
Boys perform better than girls on multiple
choice questions and girls perform better than
boys on essay-type questions (perhaps because
boys are more willing than girls to guess in
multiple-choice items), and girls perform better
in written work than boys.
Questions and assessment may be culturebound:
what is comprehensible in one culture
may be incomprehensible in another.
The test may be so long, in order to
ensure coverage, that boredom and loss of
concentration may impair reliability.
Hence specific contextual factors can exert a
significant influence on learning and this has to be
recognised in conducting assessments, to render
an assessment as unthreatening and natural as
possible.
Harlen (1994: 140-2) suggests that inconsistency
and unreliability in teacher-based and
school-based assessment may derive from differences
in:
interpreting the assessment purposes, tasks and
contents, by teachers or assessors
the actual task set, or the contexts and
circumstances surrounding the tasks (e.g. time
and place)
how much help is given to the test-takers
during the test
the degree of specificity in the marking criteria
the application of the marking criteria and the
grading or marking system that accompanies it
Chapter 6