NAVY SCHOOL MANAGEMENT MANUAL

students near the middle of a scale. If, for example, a
scale has seven points and you get a large number of "4s"
from the raters, they may be making this error. One way to
counter this is to use scales with an even number of points
(so there is no middle point). Also, behavioral "anchors"
again help.
Determining Reliability of Rating Scales and Checklists
Rating Scales. Rating scales are used in items that
involve decisions more complicated than "yes-no" or "go-nogo".
It is important that different raters use the scale
in the same way. To determine how well different raters
agree, construct a chart similar to the one below. The
chart should show the score that each rater gave to each
student on each item. In the example below, three raters
rated five students on five items. The rating scale for
each item was 1 to 5.
Sample Rating Scale Data for Five Item
Student 1 Student 2 Student 3 Student 4 Student 5
Item R1 R2 R3 R1 R2 R3 R1 R2 R3 R1 R2 R3 R1 R2 R3
Number
1 5 5 5 3 3 3 4 4 4 2 2 2 1 1 1
2 5 4 4 4 4 4 3 4 3 1 2 2 2 3 2
3 5 4 5 4 3 3 3 3 3 3 2 2 4 4 4
4 3 5 2 3 1 4 2 4 3 1 2 4 3 2 1
5 4 4 3 3 2 3 4 3 4 3 2 2 3 3 3
R1=Rater 1, R2=Rater 2, R3=Rater 3
By looking across a row, you can compare the scores given
by the different raters to each student. In the example
above, you can see that for item 1, there is perfect
agreement among raters. For items 2, 3, and 5, there is
some disagreement and for item 4, there is considerable
disagreement. A good guideline is that, if the majority of
raters agree and the raters disagreeing are only off by one
point on a scale, the rating scale is reliable. If,
however, there is no majority agreement or if raters differ
by 2 or more points on the scale, a review is necessary.
The rating scales should be checked to make sure that the
"anchoring" statements are as clear as possible, and the
instructions to scorers should be checked to make sure they
are not misleading some of the raters. It is best to do
this with the raters, because they can tell you what they
thought they were doing.
C-19
NAVEDTRA 135C