2003 IMTA Proceedings - International Military Testing Association

328
This finding is important from a substantive perspective, because fewer unidimensional pairings
means a more fake-resistant test. As a final note, the correlations between the estimated and
known thetas, ranged from .77 in the most unfavorable situation (a 20-item test with 10%
unidimensional pairings) to .96 in the most favorable (80-item test with 40% unidimensional).
The average correlation was about 0.9 for the 40-item tests, regardless of the percentage of
unidimensional pairings.
Discussion and Conclusions
This paper outlines a method of constructing and scoring fake-resistant multidimensional
pairwise preference items. Individual statements are administered and calibrated using a
unidimensional single stimulus model. Social desirability ratings are obtained for statements
using, say, a panel of judges, and fake-resistant items are created by pairing similarly desirable
statements representing different dimensions. Tests are created by combining multidimensional
items with a small number of unidimensional pairings needed to identify the latent metric. Trait
scores are then obtained using a multidimensional Bayes modal estimation procedure based on
the MUPP model developed by Stark (2002).
As shown here, the MUPP approach to test construction and scoring provides accurate
parameter recovery in both one- and two-dimensional cases, even with relatively few (say 15%)
unidimensional pairings. Accuracy of this approach generally improves as a function of test
length. Even with nonadaptive tests, good estimates may be attained using only 20 to 30 items
per dimension, meaning that a 5-D test would require 100 to 150 items. If adaptive item
selection were used to improve efficiency, the required number of items might decrease by as
much as 40%. We are currently developing and validating a 5-D inventory, using this approach,
and comparing the scores to those obtained using traditional methods.
Acknowledgements
We wish to thank the U.S. Army Research Institute for access to the AIM data and for supporting
this research. Assistance from the Human Resources Research Organization (HumRRO) was
particularly helpful with data management and recordkeeping. The Consortium of Universities
of the Washington Metropolitan Area was also helpful in securing research funds. All statements
expressed in this document are those of the authors and do not necessarily reflect the official
opinions or policies of the U.S. Army Research Institute, the U.S. Army, the Department of
Defense, HumRRO, or the Consortium of Universities.
References
Press, W.H., Flannery, B.P., Teukolsky, S.A., & Vetterling, W.T. (1990). Numerical
recipes: The art of scientific computing. New York: Cambridge University Press.
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003