RESEARCH METHOD COHEN ok

CONSTRUCTING A TEST 429
demonstrate higher order and divergent cognitive
processes. Further, it is comparatively easy to
construct an essay title. On the other hand,
essays have been criticized for yielding unreliable
data (Gronlund and Linn 1990; Cunningham
1998), for being prone to unreliable (inconsistent
and variable) scoring and neglectful of intended
learning outcomes and prone to marker bias
and preference (being too intuitive, subjective,
holistic, and time-consuming to mark). To
overcome these difficulties the authors suggest the
following:
The essay question must be restricted to
those learning outcomes that are unable to
be measured more objectively.
The essay question must ensure that it is clearly
linked to desired learning outcomes and that
it is clear what behaviours the students must
demonstrate.
The essay question must indicate the field and
tasks very clearly (e.g. ‘compare’, ‘justify’, ‘critique’,
‘summarize’, ‘classify’, ‘analyse’, ‘clarify’,
‘examine’, ‘apply’, ‘evaluate’, ‘synthesize’, ‘contrast’,
‘explain’, ‘illustrate’).
Time limits are set for each essay.
Options are avoided, or, if options are to
be given, ensure that, if students have a list
of titles from which to choose, each title is
equally difficult and equally capable of enabling
the student to demonstrate achievement,
understanding etc.
Marking criteria are prepared and are explicit,
indicating what must be included in the
answers and the points to be awarded for such
inclusions or ratings to be scored for the extent
to which certain criteria have been met.
Decisions are agreed on how to address and
score irrelevancies, inaccuracies, poor grammar
and spelling.
The work is double marked, blind, and, where
appropriate, without the marker knowing (the
name of) the essay writer.
Clearly these are issues of reliability (see
Chapter 6). The issue here is that layout can exert
aprofoundeffectonthetest.
Consider the layout of the test
Deciding on the layout will include the following
factors (Gronlund and Linn 1990; Hanna
1993; Linn 1993; Cunningham 1998):
the nature, length and clarity of the
instructions, for example what to do, how long
to take, how much to do, how many items
to attempt, what kind of response is required
(e.g. a single word, a sentence, a paragraph, a
formula, a number, a statement etc.), how and
where to enter the response, where to show the
‘working out’ of a problem, where to start new
answers (e.g. in a separate booklet)
is one answer only required to a multiple choice
item, or is more than one answer required
spread out the instructions through the test,
avoiding overloading students with too much
information at first, and providing instructions
for each section as they come to it
what marks are to be awarded for which parts
of the test
minimizing ambiguity and taking care over the
readability of the items
the progression from the easy to the more
difficult items of the test (i.e. the location and
sequence of items)
the visual layout of the page, for example
avoiding overloading students with visual
material or words
the grouping of items – keeping together items
that have the same contents or the same format
the setting out of the answer sheets or locations
so that they can be entered onto computers and
read by optical mark readers and scanners (if
appropriate).
The layout of the text should be such that it
supports the completion of the test and that this is
done as efficiently and as effectively as possible for
the student.
Consider the timing of the test
The timing refers to two areas: when the test will
take place (the day of the week, month, time
of day) and the time allowances to be given to
Chapter 19