I__. - International Military Testing Association

surveyed job incumbent is requested to estimate the relative
amount of time that he/she spends in performing each task on a
nine-point scale that ranges from "very small amount of time" to
"very large amount of time." No response means that the
incumbent does not perform the task. Each of these ratings is
divided by the sum of the relative time spent values for all of
the tasks in the inventory to get a percentage of time spent
value for the incumbent on each task. From these responses, the
following values are computed for each task: (a) the percentage
of incumbents performing the task (PMP), (b) the percentage of
time spent by incumbents performing the task (PTM), and (c) the
average pay grade of incumbents performing the task (AG). -.
Another survey containing the same task list as the
occupational inventory is administered to a large sample of
senior NCOs in each job specialty, who use a nine-point scale to
estimate the difficulty in learning to perform each task
successfully (TD) and the emphasis that should be given in formal
training on each task for newly-hired employees (TE). Raters are
asked to respond to all tasks they are familiar with, even if
some of them are not part of their current job. The TD and TE
values for each task are the means of the responses.
CONVENTIONAL TEST OUTLINE DEVELOPMENT
The CT0 is organized according to broad job knowledge areas.
The test development teams spend one to two days to create CTOs
by specifying and weighting knowledge categories based on their
own expertise and their review of appropriate personnel
classification and training documents, such as the Specialty
Training Standard, which describes important duties and tasks for
each job specialty; the Position Classification, which describes
all duties and responsibilities for each job specialty; and the
SKT abstract, which furnishes the following information for each
task in the AFS: PMP, PTM, TE, AG, and TD. The SMEs decide on
the number of test questions to be written on each knowledge
area, based on their determination of the relative testing
importance of that area.
AUTOMATED TEST OUTLINE DEVELOPMENT
The first step in the AT0 process is to select those tasks
from the inventory that are performed by at least 50% of the
incumbents or have TE values at least one standard deviation
above the mean. The screening process selects approximately 150
to 250 tasks for each AFS. A survey containing the selected
tasks is administered to approximately 70 senior NCOs to obtain
their opinions on the importance of including a question on the
SKT concerning the knowledge required to successfully perform
each task. The rating scale for testing importance is a sevenpoint
scale that ranges from *Ino importance" to t'extremely high
importance.t' The interrater reliability of these ratings is
estimated and deviant raters are eliminated (Lindguist, 1953).
The testing importance (TI) value for each task is the mean of
the ratings after deviant raters have been eliminated.
311