RESEARCH METHOD COHEN ok

162 VALIDITY AND RELIABILITY
how much additional information about the
student or situation is being referred to in the
assessment.
Harlen (1994) advocates the use of a range of
moderation strategies, both before and after the
tests, including:
statistical reference/scaling tests
inspection of samples (by post or by visit)
group moderation of grades
post-hoc adjustment of marks
accreditation of institutions
visits of verifiers
agreement panels
defining marking criteria
exemplification
group moderation meetings.
While moderation procedures are essentially posthoc
adjustments to scores, agreement trials and
practice-marking can be undertaken before the
administration of a test, which is particularly
important if there are large numbers of scripts
or several markers.
The issue here is that the results as well as the
instruments should be reliable. Reliability is also
addressed by:
calculating coefficients of reliability, split-half
techniques, the Kuder-Richardson formula,
parallel/equivalent forms of a test, test/retest
methods, the alpha coefficient
calculating and controlling the standard error
of measurement
increasing the sample size (to maximize the
range and spread of scores in a normreferenced
test), though criterion-referenced
tests recognize that scores may bunch around
the high level (in mastery learning for
example), i.e. that the range of scores might
be limited, thereby lowering the correlation
coefficients that can be calculated
increasing the number of observations made
and items included in the test (in order to
increase the range of scores)
ensuring effective domain sampling of items
in tests based on item response theory (a
particular issue in Computer Adaptive Testing
see chapter 19: Thissen 1990)
ensuring effective levels of item discriminability
and item difficulty.
Reliability has to be not only achieved but also
seen to be achieved, particularly in ‘high stakes’
testing (where a lot hangs on the results of the test,
e.g. entrance to higher education or employment).
Hence the procedures for ensuring reliability must
be transparent. The difficulty here is that the
more one moves towards reliability as defined
above, the more the test will become objective,
the more students will be measured as though they
are inanimate objects, and the more the test will
become decontextualized.
An alternative form of reliability, which is
premissed on a more constructivist psychology,
emphasizes the significance of context, the
importance of subjectivity and the need to engage
and involve the testee more fully than a simple test.
This rehearses the tension between positivism and
more interpretive approaches outlined in Chapter
1ofthisbook.Objectivetests,asdescribedin
this chapter, lean strongly towards the positivist
paradigm, while more phenomenological and
interpretive paradigms of social science research
will emphasize the importance of settings, of
individual perceptions, of attitudes, in short, of
‘authentic’ testing (e.g. by using non-contrived,
non-artificial forms of test data, for example
portfolios, documents, course work, tasks that
are stronger in realism and more ‘hands on’).
Though this latter adopts a view which is closer
to assessment rather than narrowly ‘testing’,
nevertheless the two overlap, both can yield marks,
grades and awards, both can be formative as well
as summative, both can be criterion-referenced.
With regard to validity, it is important to note
here that an effective test will adequately ensure
the following:
Content validity (e.g. adequate and representative
coverage of programme and test objectives
in the test items, a key feature of domain
sampling): this is achieved by ensuring that
the content of the test fairly samples the class
or fields of the situations or subject matter