2003 IMTA Proceedings - International Military Testing Association

488
correlations were then selected for a unit weighted short (6-item), medium (12-item), and long
key. Each successive key contained all of the items from the shorter keys, and the long key
included all of the items that had a statistically significant relationship with the criterion variable
(p < .05). Item-level empirical keys to predict MOS membership were developed using similar
procedures. Differences between the mean AVOICE item-level score for members of the target
MOS and members of all remaining MOS (i.e., the “general population”) were computed for
each item. Items for which this mean difference was significantly different from zero (p < .01)
were then included in a unit-weighted key, with a negative weight if the general population
received a higher score and a positive weight if the target MOS received a higher score.
All analyses for CTP were conducted within MOS, because interests are expected to have
different relationships with performance (i.e., CTP) in different jobs. For MOS that included
women, empirical keys were developed separately for men and women. Attrition analyses were
conducted for the 13Bs and 91A men only, and again analyses were conducted within MOS. All
samples were divided into two subsamples randomly. Empirical keys were then developed in one
sample and cross validated in the other. For the analyses focused on MOS membership,
empirical keys were developed in the CV sample and applied and evaluated in the LV sample.
RESULTS FOR JOB PERFORMANCE
A clear finding from the Project A research is that AVOICE interest scales predict job
performance. For example, in the LV sample (N = 4,220), the mean multiple correlation, across
nine MOS, between AVOICE composites and CTP was .38 (corrected for range restriction and
adjusted for shrinkage). Multiple correlations for the other first-tour performance composites
were .37 with GSP, .17 with ELS, .05 with MPD, and .05 with PFB. AVOICE validities for
second-tour performance (N=1,217) were similar; the multiple correlations were .41 with CTP,
.29 with GSP, .09 with AE, .06 with MPD, .09 with PFB, and .35 with LDR. It is interesting that
the prediction of CTP is slightly better for second-tour soldiers, while the prediction of GSP is
somewhat worse. Also, AVOICE scores predicted leadership performance quite well, even
though there was not a strong a priori rationale for expecting this relationship.
Table 2. Comparison of the Validity of AVOICE Composites with Empirical Keys for First-
Tour Core Technical Proficiency (CTP)
Cross-Validation
Sample Size
Multiple
Correlation 1
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003
Empirical Keys
12-Item All-Sig.
CV Median across MOS 176 .20 .24 .25
CV Range across MOS 92-243 .00-.44 .14-.43 .16-.46
LV Median across MOS 240 .15 .24 .23
LV Range across MOS 113-365 .00-.17 .17-.28 .17-.25
1 Multiple correlation of eight rational composites adjusted for shrinkage using Rozeboom (1978).
Regarding empirical keys for CTP, the various item and response option level empirical
keying procedures did not yield appreciably different cross-validated results. Response option
level keys general yielded higher validities in the development sample, but greater shrinkage in