RESEARCH METHOD COHEN ok

434 TESTS
discriminability and efficiency of testing. Aiken
(2003: 51) suggests that computer adaptive testing
can reduce the number of test items present to
around 50 per cent of those used in conventional
tests. Testees can work at their own pace, they
need not be discouraged but can be challenged,
the test is scored instantly to provide feedback
to the testee, a greater range of items can be
included in the test and a greater degree of
precision and reliability of measurement can be
achieved; indeed, test security can be increased
and the problem of understanding answer sheets is
avoided.
Clearly the use of computer adaptive testing has
several putative attractions. On the other hand, it
requires different skills from traditional tests, and
these might compromise the reliability of the test,
for example:
The mental processes required to work with a
computer screen and computer program differ
from those required for a pen and paper test.
Motivation and anxiety levels increase or
decrease when testees work with computers.
The physical environment might exert a
significant difference, e.g. lighting, glare from
the screen, noise from machines, loading and
running the software.
Reliability shifts from an index of the
variability of the test to an index of the standard
error of the testee’s performance. The usual
formula for calculating standard error assumes
that error variance is the same for all scores,
whereas in item response theory it is assumed
that error variance depends on each testee’s
ability – the conventional statistic of error
variance calculates a single average variance
of summed scores, whereas in item response
theory this is at best very crude, and at worst
misleading as variation is a function of ability
rather than test variation and cannot fairly be
summed (see Thissen (1990) for an analysis of
how to address this issue).
Having so many test items increases the chance
of inclusion of poor items.
Computer adaptive testing requires a large
item pool for each area of content domain
to be developed (Flaugher 1990), with sufficient
numbers, variety and spread of difficulty. All items
must measure a single aptitude or dimension, and
the items must be independent of each other,
i.e. a person’s response to an item should not
depend on that person’s response to another item.
The items have to be pretested and validated,
their difficulty and discriminability calculated,
the effect of distractors reduced, the capability
of the test to address unidimensionality and/or
multidimensionality to be clarified, and the rules
for selecting items to be enacted.