Identifying rater effects using latent trait models Abstract

Rater effects 45
Results
Harshness/Leniency
Figure 4 displays the rater location parameter estimates, sorted from the most lenient
rater to the harshest rater, each with a 95% confidence band drawn around it. A Bonferroni
correction for multiple comparisons was applied to these confidence bands to control for the
experiment-wise Type I error rate. Note that six of the raters (6%) exhibit leniency that is
statistically significant in its difference from the group mean of zero. Similarly, five of the
raters (5%) exhibit statistically significant harshness. For illustration, the average rating
assigned by the harshest rater equals 3.42 while the average rating assigned by the most
lenient rater equals 5.68. The average of all ratings assigned to the set of 28 essays by the
101 raters equals 4.39. From these figures, it is clear that the rater harshness and leniency
may have a profound impact on the rating assigned to an examinee. In fact, if an arbitrary
cut-point was imposed on the raw ratings (e.g., 5 or greater is a passing rating – the 50 th percentile
of the expert ratings), 75% of the examinees rated by the most lenient rater would
pass while only 14% of the examinees rated by the harshest rater would pass 5 .
2.50
2.00
1.50
1.00
Logit
0.50
0.00
-0.50
-1.00
-1.50
-2.00
Rater
Figure 4:
Rater Harshness and Leniency Estimates