2003 IMTA Proceedings - International Military Testing Association

As depicted in Table 3, half of participants chose D&D as their second choice,
whereas frequency was a common third choice. Rating scale was a less common third
Distance and
Outcomes Checklist Frequency Discrepancy Rating Scale
Accuracy 10% 10% 60% 20%
Timeliness - 11% 67% 22%
Productivity - 80% 10% 10%
Efficiency - 10% 70% 20%
Safety - 50% 10% 40%
Effects 33% - 11% 56%
choice.
Table 2. First Choice Rating Formats for Outcome Measures
Table 3. Second Choice Rating Formats for Outcome Measures
Distance and
Outcomes Checklist Frequency Discrepancy Rating Scale
Accuracy 45% 22% 11% 22%
Timeliness 30% 10% 20% 40%
Productivity 56% - 11% 33%
Efficiency 13% 13% 13% 61%
Safety 62% 25% 13% -
Effects 12% 25% 50% 13%
Process Measures
Procedural Taskwork
As highlighted in Table 4, nearly every respondent chose checklist as the most
appropriate way to measure procedural taskwork because it follows a step-by-step
process. Participants felt that the steps lend themselves to a checklist format, which
allows for actions to be measured by using dichotomous variables, such as yes or no. In
addition, they felt that it allows for determining whether or not multiple outcomes were
accomplished. Paper and pencil job knowledge tests and hands-on performance tests
were also recommended for both procedural and non-procedural taskwork, given that
these tests effectively measure the expertise and skill sets of individuals.
667
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003