I__. - International Military Testing Association

Hands-on managers used these reports extensively. Differences
among TAs were discussed, and ambiguities in interpretation of
scoring rules were resolved through discussion and, if required,
additional training. Individual and group trends could be
detected. Individual TA counselling focused on adherence to the
original training standards and the definition and interpretation
of scoreable steps. Hands-on managers avoided overemphasis
on consistency to prevent artificially high levels of agreement.
Interrater Reliability Results
Interrater reliability, or TA agreement, can indicate the
presence of several possible error sources: test design, time,
t environmental, or other effects. Interrater reliability is the
percentage agreement between primary and shadow scorers on
individual task steps, It is computed by dividing the number of
steps on which the primary and shadow scorer agreed by the total
number they both graded, summed across all examinees and all
tasks. It was calculated using all obsenrations where both
primary and shadow step scores were available.
Fig. 1: Agreement by task
Age-t bctrrem ,nimwy md rhodow test odminirtrotor,
b y time intcrvd
Fig. 2: Agreement by Time
Period
Figure 1 shows scorer agreement across tasks for automotive
mechanics. Agreement ranged from .873 to .971 indicating that
TAs could reliably differentiate tlG~t' and "No Go" performance.
The lowest reliabilities at both sites were on troubleshooting
tasks, indicating some ambiguity in scoring the steps on those
tasks. Three of the lowest four reliabilities occurred on tasks
which were hard to observe because of confined spaces. The fact
that the relative reliabilities among tasks were the same between
sites also indicates a good training program and suggests that
reliability differences were due to test effects.
538