Insight Psychometrics Technical Manual - Canadian Test Centre

Validity analysesCheck on gender biasIn each of the stages of item and test development, statistical tests werecarried out to identify any items that seemed to be biased according togender; that is, any items that had particularly higher probabilities of correctresponses for boys or for girls, controlling for ability. Items that showedproblems were eliminated from consideration when the statistical evidencewas strong and when the differences were considered to demonstrate bias.The “differential item functioning” or DIF statistic is calculated for each itemas the standardized difference in Rasch difficulty comparing boys (positive)with girls (negative). Items that had DIF statistic values equal to or larger than 2standard errors were omitted from the final forms whenever possible. Items showingthis degree of bias that couldn’t be omitted were usually counterbalanced with anitem of comparable difficulty and a comparable degree of bias in the other direction.For example, an item with DIF statistic around −2 was usually counterbalancedwith an item of comparable difficulty and a DIF statistic around +2.For the final development forms at each level for subtests Gc, Gv, and Gf, thedistribution of DIF statistics is shown over all items in Figure 19. Most itemsare in the normal range between +/– 2. There are some items that showsignificant DIF, but the fact that they are about evenly balanced betweenbias in favour of boys and bias in favour of girls suggests that there is littleoverall bias in the tests.18