RESEARCH METHOD COHEN ok

CONSTRUCTING A TEST 421
Select the contents of the test
Here the test is subject to item analysis.Gronlund
and Linn (1990) suggest that an item analysis will
need to consider:
the suitability of the format of each item for
the (learning) objective (appropriateness)
the ability of each item to enable students
to demonstrate their performance of the
(learning) objective (relevance)
the clarity of the task for each item
the straightforwardness of the task
the unambiguity of the outcome of each item,
and agreement on what that outcome should be
the cultural fairness of each item
the independence of each item (i.e. where the
influence of other items of the test is minimal
and where successful completion of one item
is not dependent on successful completion of
another)
the adequacy of coverage of each (learning)
objective by the items of the test.
In moving to test construction the researcher will
need to consider how each element to be tested
will be operationalized:
what indicators and kinds of evidence of
achievement of the objective will be required
what indicators of high, moderate and low
achievement there will be
what will the students be doing when they are
working on each element of the test
what the outcome of the test will be (e.g.
a written response, a tick in a box of
multiple choice items, an essay, a diagram,
acomputation).
Indeed the Task Group on Assessment and Testing
(1988) in the UK suggest that attention will
have to be given to the presentation, operation
and response modes of a test:
how the task will be introduced (e.g.
oral, written, pictorial, computer, practical
demonstration)
what the students will be doing when they are
working on the test (e.g. mental computation,
practical work, oral work, written)
what the outcome will be – how they will show
achievement and present the outcomes (e.g.
choosing one item from a multiple choice
question, writing a short response, open-ended
writing, oral, practical outcome, computer
output).
Operationalizing a test from objectives can
proceed by stages:
Identify the objectives/outcomes/elements to
be covered.
Break down the objectives/outcomes/elements
into constituent components or elements.
Select the components that will feature in the
test, such that, if possible, they will represent
the larger field (i.e. domain referencing, if
required).
Recast the components in terms of specific,
practical, observable behaviours, activities and
practices that fairly represent and cover that
component.
Specify the kinds of data required to provide
information on the achievement of the criteria.
Specify the success criteria (performance
indicators) in practical terms, working out
marks and grades to be awarded and how
weightings will be addressed.
Write each item of the test.
Conduct a pilot to refine the language/
readability and presentation of the items, to
gauge item discriminability, item difficulty and
distractors (discussed below), and to address
validity and reliability.
Item analysis, Gronlund and Linn (1990: 255)
suggest, is designed to ensure that the items
function as they are intended, for example,
that criterion-referenced items fairly cover the
fields and criteria and that norm-referenced
items demonstrate item discriminability (discussed
below); the level of difficulty of the items
is appropriate (see below: item difficulty); the
test is reliable (free of distractors – unnecessary
information and irrelevant cues, see below:
distractors) (seeMillmanandGreene(1993).An
item analysis will consider the accuracy levels
available in the answer, the item difficulty, the
Chapter 19