I__. - International Military Testing Association

might influence task performance. TAs were trained on how to
perform each task they were to evaluate and practiced them under
the supervision of active duty subject matter experts. This
included role playing and deliberate errors on the part of the
V'examineelt to check TA consistency and develop standardized
scoring of irregular responses. Once test administration was
begun, there were periodic review of steps with low interrater
reliabilities, with retraining where necessary.
Rotation of TAs Across Tasks
TAs were trained in multiple tasks to allow them to rotate
among test stations. This lessened the effect of boredom,
provided a cross check on the standardization of scoring in each
task, and reduced the impact of TA differences on scoring.
Shadow Scorinq
Perhaps the most important quality control procedure, shadow
scoring involved independent evaluation of an individual's task
performance by two TAs simultaneously. Shadow scorers were used
to monitor TA performance and test reliability, and were
systematically scheduled to capture interactions among testing
order and individual TA characteristics.
On-Site Data Entrv Trend Analvsis
A Hands-On Score Entry System (HOSES) was developed to enter,
verify, and report analyses-of collected data. Daily on-site
data entry enhanced completeness of data and allowed for early
identification of problems with the tests, TA consistency, and
score drift over time. HOSES generated three reports which were
used by site hands-on managers to improve scoring reliability.
1. Data Entry Report. All data were entered twice. This report
verified that there were no discrepancies between the two
entries. It also reported any missing data so the information
could be tracked down on the day of original testing. This
greatly reduced the amount of missing data.
2. The Detailed Discrepancy Report listed al'1 steps where
primary and shadow scorers disagreed. It also gave percent
disagreement for each task, and overall daily total by TA.
3. The Summary Report presented cumulative historical summaries
by TA and task. TA summaries showed leniency and reliability
information for each task administered by the TA. Leniency was
measured as a deviation from the mean percentage of "GOATS for
all TAs on each task. Reliability indicated disagreement with
all other TA's on each task. These were valuable in identifying
individual TA problems. Since this report could be broken out by
time, it also provided trend information. Task summaries showed
percent "Go" and disagreement for each step. This helped focus
on test effect problems, i.e. those common across all TAs.
537