I__. - International Military Testing Association

For each administration, the tests were generated with a total test and each domain
mean difficulty index (p-value) of -60. The tests are essentially power tests with
three hours allowed. The cutting score for each test is based on 62.5% of the
number of test questions (a score of 75) or the group mean, whichever is higher. The
cutting score was 75 for each of the tests for each administration. The test items
were selected in accordance with the following parameters, p-values between -25
and .90 and biserials between .15 and .99.
The tests are administered twice yearly, in the spring and fall, to enlisted Navy
personnel in pay grades E-5 through E-9, with a minimum of nine months experience
in an IMA activity. BM-0110 was developed in the summer of 1987 and placed into
operational use in the fall of 1987. EM-4613 was developed in the fall of 1987 and
placed into operational use in the spring of 1988. All tests in this program *were
developed by subject-matter experts from each trade under the tutelage of a testing
specialist. All of the tests are computer generated by an automated test processing
system (TPS) that includes item banking, scoring, and analysis and updating of all
test and item data.
Procedure
Three different administrations -- Spring 1989 (l-89), Fall 1989 (2-89), and Spring
1990 (l-90) --were used for this study for both the BM-0110 and EM-4613 tests. Both
the l-89 and I-90 tests were constructed under normal procedures, i.e., with items
grouped by domain and presented from easiest to most difficult within each
domain. For the 2-89 administrations, the test items were randomized without
regard for content area or difficulty level.
The items for each administration were generated by the TPS from the total item
pool available for each test and therefore the items were not identical across
administrations. Table II presents the number of items common to each pair of test
administrations.
Table II
Common Items Between Administrations
1-89 - 2-89 1-89 - l-90 2-89 - l-90
BM-0110
I
71 77
I
89
EM-461 3 66 67
I
67
Under ideal conditions, the research design would have used the same items for each
administration and both forms of the test would have been administered at the
same time. However, due to a number of factors including fairly small N’s and
numerous repeat candidates from one test administration to another, the ideal
design was not possible. The test populations do tend to be quite stable from one
administration to another, however, in terms of trade experience and numbers from
each paygrade.
The test results and item statistics from each administration for each test were
compared with the other administrations from four different perspectives -- total
test results, part test scores, common item comparisons, and individual item
statistics. As previously stated, the objectives were to determine if randomizing the
items would have any effect on total test performance, part (domain) test
performance, and individual item statistics. A variety of statistical procedures were
employed to analyze the data including Z-tests, two-tailed t-tests, and ANOVAs.
275