2003 IMTA Proceedings - International Military Testing Association

items involving stimuli on dimension 1 (Adjustment) only, dimension 2 (Agreeableness) only,
and dimensions 1 and 2. For illustration, a conventional test of 80 items (pairings), 40% of
which are unidimensional, must contain 16 1-1, 16 2-2, and 48 1-2/2-1 items.
80 stimuli representing Adjustment (dimension “1”) and 49 stimuli representing
Agreeableness (dimension “2”) were chosen, as in the 1-D case. Multidimensional items (1-2
and 2-1) were created by pairing Adjustment and Agreeableness stimuli that had similar
desirability; unidimensional (1-1 and 2-2) items were created by pairing stimuli, representing the
same dimension, which had different location parameters, but fairly similar desirability ratings.
Once the three 80-item tests were constructed, three 20- and three 40-item tests were created
using the first 20 and 40 items, respectively, of the 80-item test in each condition.
Investigating Parameter Recovery
1-D simulations. To determine if the accuracy of parameter recovery varied across theta
points on the unidimensional trait continuum, each conventional test was administered to 50
simulated examinees (simulees) at 31 points on the interval [ −3.0, − 2.8,..., + 3.0] . At each grid
point, the average estimated theta and standard error were computed over 50 replications and
used to compute error statistics, which were compared across conditions using MANOVA.
2-D simulations. Each of the nine conventional tests was administered to 50 simulees at
points on a ( θ1, θ 2)
grid, where θd ranged from –3 to +3 in increments of 0.5. As above, error
statistics for the estimated thetas and standard errors were compared using MANOVA.
Results
1-D simulations. Bias and root mean square errors of the latent trait estimates decreased
as test length increased, but accurate parameter recovery was observed across a wide range of
theta even for the short 10-item test. The estimated standard errors were also accurate,
approaching zero at moderate thetas for tests of 20 and 40 items. Overall, the results suggested
that latent trait and standard error estimation was quite accurate. In fact, a follow-up simulation,
examining the correlation between estimated and known thetas for 1000 simulees, sampled from
a standard normal distribution, showed correlations between estimated and known thetas for the
10-, 20-, and 40- item tests of .90, .95, and .97, respectively.
2-D simulations. Two independent variables, test length (TESTLEN) and the percent of
unidimensional pairings (UNIPCT) were fully crossed to produce 9 tests. Error statistics were
computed for each dimension separately and averaged for comparison using MANOVA. As
before, a follow-up simulation was conducted for each test by sampling known thetas for 1000
simulees from independent standard normal distributions and computing the correlations
between the estimated and known thetas.
As in the 1-D study, bias in the latent trait estimates decreased as test length increased,
and the largest bias statistics occurred at the endpoints of the trait continua, where the regression
toward the mean effect was greatest and the items provided the least information. In addition,
while the accuracy of latent trait estimation did increase as the percentage of unidimensional
pairings increased from 10% to 20%, there was little improvement by going from 20% to 40%.
These results were supported by the MANOVA, which showed main effects for both
independent variables, but only a weak linear trend for UNIPCT (eta-squared was about .10).
327
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003