2003 IMTA Proceedings - International Military Testing Association

324
different personality dimensions, with a high degree of accuracy, meaning that interindividual
comparisons are possible.
An IRT Approach to Constructing and Scoring Pairwise Preference Items
In his dissertation, Stark (2002) proposed a general IRT approach for constructing and
scoring pairwise preference items involving statements on different dimensions. A multi-step
procedure is required. 1) Develop a large number of statements representing different
personality dimensions. 2) Administer the statements to a group of respondents instructed to
indicate how well on, say, a scale of 1 to 5, each statement describes him/her. Also administer
the statements to a separate group of judges instructed to rate the desirability of each statement
using a similar scale. 3) Estimate stimulus parameters for the individual statements representing
each dimension separately, using a unidimensional IRT model that provides good model-data fit;
one possibility would be the Generalized Graded Unfolding Model (GGUM; Roberts, Donoghue,
& Laughlin, 2000a). 4) Create fake-resistant items by pairing statements similar in desirability
but representing different dimensions; also create a small proportion of unidimensional items by
pairing statements that are similar in desirability, but having different stimulus location
parameters. These pairings constitute the fake-resistant test. 5) Administer the resulting test to
respondents, instructed to choose the statement in each pair that better describes him/her. 6)
Score the pairwise preference data using a Bayes modal latent trait estimation procedure, based
on the following general model:
Pst{1, 0} Ps{1} Pt{0}
P( s> t) ( θ , )
i d θ s d = ≈
, (1)
t P {1, 0} + P {0,1} P{1} P{0} + P{0} P{1}
st st s t s t
where:
i = index for items (pairings), where i = 1 to I,
d = index for dimensions, where d = 1, …, D,
s, t = indices for first and second stimuli, respectively, in a pairing,
θd , θ =
s d latent trait values for a respondent on dimensions d t
s and dt respectively,
Ps{ 1},
Ps{
0}
= probability of endorsing/not endorsing stimulus s at θ d , s
Pt { 1},
Pt
{ 0}
= probability of endorsing/not endorsing stimulus t at θ d , t
Pst{ 1,
0}
= joint probability of endorsing stimulus s, and not endorsing stimulus t at ( θ d , θ )
s d , t
Pst{ 0,
1}
= joint probability of not endorsing stimulus s, and endorsing stimulus t at ( θ d , θ )
s d , and
t
P ( θ , θ ) = probability of respondent j preferring stimulus s to stimulus t in pairing i.
( s>
t)
i d s d t
In essence, the model above assumes that when a respondent is presented with a pair of
statements (stimuli), s and t, and is asked to indicate a preference, he/she evaluates each stimulus
separately and makes independent decisions about endorsement. If a respondent endorses both
stimuli, or does not endorse either, he/she must reevaluate the stimuli, independently, until a
preference is reached. A preference is represented by the joint outcome {Agree (1), Disagree (0)}
or {Disagree (0), Agree (1)}. An outcome of {1,0} indicates that stimulus s was preferred to
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003