RESEARCH METHOD COHEN ok

420 TESTS
syllabus; then it indicates the marks/percentages
to be awarded for each area. This indicates several
points:
The least emphasis is given to the build-up
to and end of the war (10 marks each in the
‘total’ column).
The greatest emphasis is given to the invasion
of France (35 marks in the ‘total’ column).
There is fairly even coverage of the objectives
specified (the figures in the ‘total’ row only vary
from 9 to 13).
Greatest coverage is given to objectives 2a and
3a, and least coverage is given to objective 1c.
Some content areas are not covered in the test
items (the blanks in the matrix).
Hence we have here a test scheme that indicates
relative weightings, coverage of objectives and
content, and the relation between these two latter
elements. Gronlund and Linn (1990) suggest that
relative weightings should be addressed by first
assigning percentages at the foot of each column,
then by assigning percentages at the end of each
row, and then completing each cell of the matrix
within these specifications. This ensures that
appropriate sampling and coverage of the items
are achieved. The example of the matrix refers to
specific objectives as column headings; of course
these could be replaced by factual knowledge,
conceptual knowledge and principles, and skills
for each of the column headings. Alternatively
they could be replaced with specific aspects of
an activity, for example (Cohen et al. 2004: 339):
designing a crane, making the crane, testing the
crane, evaluating the results, improving the design.
Indeed these latter could become content (row)
headings, as shown in Box 19.2. Here one can
see that practical skills will carry fewer marks
than recording skills (the column totals), and that
making and evaluating carry equal marks (the row
totals).
This exercise also enables some indication to
be gained on the number of items to be included
in the test, for instance in the example of the
history test the matrix is 9 × 6 = 54 possible
items, and in the crane activity example the
matrix is 5 × 4 = 20 possible items. Of course,
there could be considerable variation in this, for
example more test items could be inserted if it were
deemed desirable to test one cell of the matrix with
more than one item (possible for cross-checking),
or indeed there could be fewer items if it were
possible to have a single test item that serves
more than one cell of the matrix. The difficulty
in matrix construction is that it can easily become
a runaway activity, generating very many test
items and, hence, leading to an unworkably long
test – typically the greater the degree of specificity
required, the greater the number of test items there
will be. One skill in test construction is to be able
to have a single test item that provides valid and
reliable data for more than a single factor.
Having undertaken the test specifications, the
researcher should have achieved clarity on the
exact test items that test certain aspects of achievement
of objectives, programmes, contents etc., the
coverage and balance of coverage of the test items
and the relative weightings of the test items.
Box 19.2
Compiling elements of test items
Content area Identifying key concepts Practical skills Evaluative skills Recording results Total
and principles
Designing a crane 2 1 1 3 7
Making the crane 2 5 2 3 12
Testing the crane 3 3 1 4 11
Evaluating the results 3 5 4 12
Improving the design 2 2 3 1 8
Total 12 11 12 15 50