2003 IMTA Proceedings - International Military Testing Association

soldiers who left the Army for avoidable reasons (e.g., failure to meet minimal performance or
behavioral criteria) during their first tour of duty. Because the majority of this avoidable attrition
occurs during the first 12 months, attrition was defined as leaving the military for avoidable
reasons during the first 12 months of service.
BASIC VALIDATION AND EMPIRICAL SCORING PROCEDURES
For the basic validation analyses, AVOICE item-level scores were summed to create the
22 interest scales. Based on principal components analysis, these scale scores were grouped into
eight summary composites (Campbell & Knapp, 2001; see Table 1). AVOICE validity was
assessed by computing multiple correlations between these eight AVOICE composites and the
performance criteria within each MOS, statistically adjusting the correlations for shrinkage
(Rozeboom, 1978), correcting them for range restriction, and then averaging across MOS.
Paullin, Bruskiewicz, Hanson, Logan, and Fellows (1995) conducted preliminary
analyses to determine whether empirical scoring procedures have potential for enhancing the
validity of the AVOICE beyond the levels obtained in the basic validation analyses. This work
focused on five MOS, selected to include both combat and noncombat MOS, some MOS that are
similar to each other and some that are quite different in terms of the tasks involved, and some
MOS that have a relatively large percentage of female soldiers. The MOS selected were
Infantryman (11B), Cannon Crewman (13B), Light Wheel Vehicle Mechanic (63B),
Administrative Specialist (71L), and Medical Specialist (91A). Only a subset of the criterion
variables were included in this work: Core Technical Proficiency (CTP) because it is quite well
predicted by the AVOICE (see results below); attrition because it is reasonable to expect that
soldiers whose interests do not match their jobs will be more likely to leave; and MOS
membership because it has been one of the most widely studied criteria in past research on
interests. Several different empirical keying procedures were explored for CTP for the 13Bs and
for occupational membership for the 91As. Based on results of these analyses, only the most
promising empirical keying strategies were evaluated for the remaining MOS and criteria.
Empirical keys that focus on response option level data do not assume a linear
relationship between item scores and the criterion of interest. Two of the most common response
option level approaches were tried: vertical percent and correlational. For the vertical percent
method, contrasting groups of 13Bs were formed based on the criterion variable (CTP): soldiers
whose CTP scores fell in the top 30 percent and those whose scores fell in the bottom 30 percent.
The differences between the percentages of soldiers choosing each response option in each
contrasting group were then used to assign “net weights” to each response option (Strong, 1943).
This essentially gives differences greater weight if they occur at either extreme of the response
distribution. Two vertical percent keys were developed: one including only those items that
showed at least a 5-point difference across groups and one including items that showed at least a
10-point difference. Similar procedures were followed for the MOS membership criterion for
91As, but all items were included in these keys. A second “correlational” method was also
explored for CTP. Each dichotomous response option score was correlated with the continuously
scored CTP criterion measure, and unit weights were assigned to response options with a
significant point-biserial correlation (p < .05).
Empirical scoring procedures focused on item-level data were also developed, by
computing correlations between AVOICE item-level scores and CTP. The items with the largest
487
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003