I__. - International Military Testing Association

The general knowledge test consisted of 98 items. It was not separated
by task or functional area, and in administration and analysis was treated as
a single test.
Test Administration
The field tests were administered to 61 Radiomen, all of whom were
graduates of the Class A Radioman School, were in paygrades E-2, E-3, and E-4,
and had graduated from the School between 1 and 59 months prior to testing.
(Of the tested population, 79% were in paygrade E-3 and 60% were in the 12
months to 35 months experience window.) Twenty-eight of the 61 sailors tested
were assigned to shore installations at the time of testing and 33 were aboard
ships.
Testing was conducted at two locations and about a month apart. Testing
lasted for 8 hours for each 'examinee and the three components of the test were
sequentially counterbalanced. Five hands-on scorers were used. All scorers
were project staff and had received extensive task/test training and
calibration. Each Radioman was scored independently by at least two scorers
for each hands-on test.
Field Test Results
Although a wide variety of analyses were conducted (Ford, Doyle,
Schultz, & Hoffman, 1987), this paper will focus on four main areas of
interest. Specifically:
� Interrater reliability of the hands-on tests.
� Internal consistency within test methods.
� Intercorrelations among test methods.
� Assignment effect (ship vs. shore).
Interrater Reliabilitvsf the Hands-On Tests
Interrater reliability estimates were computed from a generalizability
theory in which absolute generalizability coefficients were produced (SAS,
1982; Brennan, Jarjoura & Deaton, 1980). Generalizability estimates were
obtained as if only one rater score were produced and for an average of the
two raters, as shown in Table 2.
The reliabilities are exceptionally high. This is attributed to the
influence of the firm control over the scorers that was possible because they
were members of project staff and, secondly, due to the high incidence of
product scoring among the tested tasks.
Internal Consistencv Within Test Methods
Intertask correlations were conducted for the hands-on, written, and
general knowledge tests (the general knowledge test was analyzed for interitem
correlations since it was treated as a single test) and are presented in
Table 3. The obtained coefficients demonstrate acceptable levels of internal
consistency.
531