Technical Report - International Military Testing Association

. .
‘-
-.-
.
.I :
.
-..
(. .-
,.’
.‘;. . . .
:
.._
. .
../
./’
:
.
-.
:’
. .
. ‘,\
.’ ‘\
.
,
.
.
.
._
������ ������ �������� ���������� ⌧ ���� ������ � �� �� � ��
��
- ~- . ---. .*._ -..-___ _,- .__^.. ____.
.
.
���
� �
�� � ���� � �������� � � �������
.
.I..-.-..--.---. .._... _ __; _.., ._ ._
c
������� ��
The third general area fall8 into the Human Error category. ~x~plea of
thir type of error will vary from the use of the wrong cnsver key (eaey
to spot vicually) to the inclusion in the ramplt of answer cards from
another examination, or lack of coordination portafning to different
elements of the program.
In short, diacrcpancier do occur in machine-produced item analysis.
and it is generally agreed that human inspection of item analysis rt6UltiB
ir ncca88ary.
Hethodr
At this point it might be useful to conaider scme methods upon which
vieual fnrpection of machine itear � naly8i8 re8ulte may be baaed.
The firrrt method considered might be called the u8e of an outside
criteria. One such outside criterion vould concern the Item difficulty
index or item p-value. The p-value criterion 8tAte8 that a p-value may
not be less than 0.00, nor greater than +l.OO. That In, not leaa than
zero percent of the rample may an8wer an item correctly, nor more than
100 percent. Another outside criterion would 8tate that the item-teat
correlation or r-value (vhich 18 generalized a8 the discrimination index)
may not be lese than -1.00 nor more than +l.OO without further invertigat
ion. Another method of check:ng itan analyrir might be called the
intra-item conristency method. If the item r and p values are bared on
the total sample, and if the correct and incorrect item alternative8
r and y ertimatar are baaed on a high and low group (upper and lower
27% for fnrtrncc) then the correct alternative is being mearur.-d indcpendently
on two overlapping groups to yield similar although not identical
rerults. Empirical studier based on a large number of item8 will yeild
probability table8 for variourr degree8 of diacrepancice betvecn there
tvo ret8 of r and p valuer.
A more involved method of checking item analysis might be termed
the Item history method. This method is rertricted to pre-tested or
control items, and COn8it3t8 of observing change8 in the p and r value8
of the right and wrong alternative8 of an item from one examination
period to another. Although the p-value of an item is canputed independently
of other items, it8 value may change due to change8 in the examiaee
population or varying degrees of item coatpromise or obsolescence. The
r-value vould also be expected to have a history showing a degree of
fluctuation which would be further affected by its dependence on the
action of other itanu in the examination.
Another method of checking item analysis data ir based on trends in
the item results. When the p and r value8 of the examination items arc
viewed am a whole, trendo may become apparent. For instance, if one or
more clusters of negatfve r-values are obrerved wfthout cmpcnsatfng
group8 of high r-values and the rav score standard deviation io adequate
on the ballsi of experience or a rtatistical model, these trend8 may
indicate error8 in the item analysis resultr.
. .
.
1 0 3
I
� �
� �
f