2003 IMTA Proceedings - International Military Testing Association

312
Classification and regression trees (CART; Breiman, Friedman, Olshen, & Stone, 1984)
is one algorithm associated with this approach. Through brute force, CART examines all
possible binary splits of the data (answers to “yes/no” questions) based on all of the predictor
variables. It places the best split at the root of the tree and continues this process until no further
splits can be made. Later, the resulting decision tree is “pruned” according to misclassification
rates, user-determined preferences (e.g., permitted number of cases in a terminal node), or by
eliminating redundant nodes. Additionally, competing trees may develop depending on the
nature of the data.
To assess classification accuracy, CART uses “v-fold cross-validation” in which the
sample is divided into v subsamples and grows a decision tree after combining v-1 subsamples
and assesses the classification accuracy using the hold-out subsample. This process is iterated so
that v-1 subsamples are combined and used to grow decision trees and each of the v samples is
used as the hold-out sample once. Classification accuracy is estimated as the average
classification accuracy across the v holdout subsamples.
As mentioned earlier, CART may delineate separate “paths” that lead to the same
outcome –identifying configural relationships in the data. Also, CART may reuse any number of
variables in separate parts of the tree, and thus, may capture non-linear relationships. Below we
describe a investigation that examined whether we could improve upon the Adaptability
Composite in predicting attrition with the AIM.
Applying Decision Tree Methodology to AIM Data
Sample, data, and software
A file containing the data from 22,328 enlisted U.S. Army personnel was created
containing the AIM scale scores and a retention variable to 12 months. The data for this file
came from the AIM Grand Research Database, managed by ARI and a contractor, the Human
Resources Research Organization (HumRRO). This database is the source of much of the recent
research surrounding the AIM and, for this database, the AIM was administered to these
personnel between 1998 and 1999 for research purposes only (Knapp, Heggestad, & Young, in
preparation). The 12-month time interval was selected because this provided the CART 4.0
software package (Breiman, Friedman, Olshen, & Stone, 1997) with a sufficient number of
respondents with which to “grow” a tree. The six content scales were used as input (predictor
variables) and the 12-month attrition variable (criterion) was treated dichotomously (i.e.,
“stayers” and “leavers”).
Results
The analysis yielded 39 trees ranging in complexity from two terminal nodes to a tree
with 2802 terminal nodes and a depth of 51 levels or tiers. However, the larger trees exhibited
high rates of misclassification among the stayers (as much as 60 percent). Of particular interest
were five trees resulting from this analysis. Table 1 summarizes the misclassification rates for
these five “best” trees using the v-fold cross-validation approach (v=10).
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003