NAVY SCHOOL MANAGEMENT MANUAL

Alternative "c" is more deceiving to the high group
than to the low group. This item can be improved by
making this response less plausible to the high group.
Analyzing Procedures for Performance and Essay Test Items.
Performance and essay items almost always require a checklist or
rating scale. First, check the reliability of the rating scale.
Once you are confident the checklist or rating scale is
reliable, student responses can be analyzed. The following
sections describe the types of errors that raters can make and
procedures for checking the reliability of checklists and rating
scales for performance and essay test items.
Types of Rating Errors. One problem with rating scales is that
different raters often make different judgments about the same
performance. These differences or rating errors can be
classified into four categories:
Error of Standards. Errors are sometimes made because of
differences in different raters' standards. If rating is
done without any specified standards, there may be as many
different standards as there are observers. This is why it
is important that rating scales be "anchored" with
descriptions of the behaviors for each value on the rating
scale. The more complete these descriptions, the better
the inter-rater agreement.
Error of Halo. A rater’s ratings may be biased because
he/she allows his/her general impression of an individual
to influence his/her judgment. This results in a shift of
the rating and is known as a "halo" effect. If a rater is
favorably impressed, the shift is toward the high end of
the scale. If the rater is unfavorably impressed, the
shift is toward the low end. This type of error frequently
goes undetected unless it is extreme. It is therefore a
difficult error to overcome. Error of halo is reduced by
reminding each rater that he/she is judging specific
performances and should NOT take into consideration of
his/her overall impression of a student.
Logical Error. A logical error may occur when a rater uses
a series of rating scales. When a rater tends to give
similar ratings on scales that are not necessarily related,
he/she is making a logical error. The way to minimize
logical errors is to make clear the distinctions among
different performances or aspects of a product that are to
be measured. Again, behavioral "anchors" help.
Error of central tendency. An error of central tendency is
demonstrated when different raters tend to rate most
C-18
NAVEDTRA 135C