2003 IMTA Proceedings - International Military Testing Association

stimulus t, and is considered a positive response; an outcome of {0,1} indicates that stimulus t
was preferred to s (a negative response). Thus, the response data for this model are
dichotomous. Note that this model makes no assumption about item dimensionality. The
statements involved in a pair may be on the same or different dimensions and, in fact, a small
number of unidimensional pairings is required to identify the latent trait metric and permit
interindividual comparisons. In addition, because the stimuli in each pair are assumed to be
evaluated independently, stimulus parameters can be estimated, for each dimension separately,
by using software for calibrating unidimensional single stimulus responses, such as the
GGUM2000 computer program (Roberts, Donoghue, & Laughlin, 2000b). Therefore, this model
is not referred to as a multidimensional model, but rather a multi-unidimensional model called
MUPP (Multi-Unidimensional Pairwise Preferences; Stark, 2002).
Scoring respondents. Once the fake-resistant tests have been administered and the
dichotomous pairwise preference data have been collected, a multidimensional Bayes modal
estimation procedure can be used to obtain scores for each respondent on each dimension. This
amounts to maximizing the following equation:
ui
⎧ n
⎪ 1−u
⎫ i ⎪
Lu ( % , θ % ) = ⎨ ⎡P⎤ ⎡ ( s> t) 1 −P ⎤ ( ) * ( )
i s> t ⎬ f θ%
∏ ⎣ ⎦ ⎣ i⎦
, (2)
⎪⎩ i=
1
⎪⎭
% ( , , ..., ) represents a vector of latent trait values (one for each
dimension), u%represents a dichotomous response pattern, P ( > ) ( θ , θ ) is the probability of
where θ= θd'= 1 θd'= 2 θd'=
D
s t i ds dt
preferring stimulus s to stimulus t in item i, and f ( θ ) % represents the prior density, whose
dimensions, d ' = 1 to D, are assumed uncorrelated.
Equation 2 can be solved numerically to obtain a vector of latent trait estimates for each
respondent using subroutine DFPMIN (Press, Flannery, Teukolsky, & Vetterling, 1990) in
conjunction with functions that compute the log likelihood and its first derivatives. DFPMIN
performs a D-dimensional minimization, using a Broyden-Fletcher-Goldfarb-Shanno (BFGS)
algorithm, so the first derivatives and log likelihood values must be multiplied by –1 when
maximizing the likelihood of a response pattern. The primary advantage of this approach, over
Newton-Raphson iterations, is DFPMIN does not require an analytical solution for the second
derivatives of the log likelihood. Instead, it provides an approximation to the inverse Hessian
matrix of second derivatives, from which standard errors of the latent trait estimates can be
obtained by taking the square roots of the diagonal elements
A Monte Carlo Study to Examine MUPP Scoring Accuracy
Constructing Tests for Simulations
To examine the accuracy of latent trait estimation, one- and two- dimensional tests were
constructed using AIM pretest data provided by the U.S. Army Research Institute (ARI) through
Human Resources Research Organization (HumRRO). Specifically, in the early stages of AIM
development, nearly 500 stimuli, representing 6 temperament dimensions, were administered to
738 recruits who were instructed to indicate their level of agreement, using a scale of 1 (very
325
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003