Introduction to Categorical Data Analysis

8.5 ANALYZING RATER AGREEMENT 261
that they both classify a subject in category i. The sum �
i πii is the total probability
of agreement. Perfect agreement occurs when �
i πii = 1.
Many categorical scales are quite subjective, and perfect agreement is rare. This
section presents ways to measure strength of agreement and detect patterns of
disagreement. Agreement is distinct from association. Strong agreement requires
strong association, but strong association can exist without strong agreement. If
observer X consistently classifies subjects one level higher than observer Y , the
strength of agreement is poor even though the association is strong.
8.5.1 Cell Residuals for Independence Model
One way of evaluating agreement compares the cell counts {nij } to the values
{ni+n+j /n} predicted by the loglinear model of independence (7.1). That model provides
a baseline, showing the degree of agreement expected if no association existed
between the ratings. Normally it would fit poorly if there is even only mild agreement,
but its cell standardized residuals (Section 2.4.5) provide information about patterns
of agreement and disagreement.
Cells with positive standardized residuals have higher frequencies than expected
under independence. Ideally, large positive standardized residuals occur on the main
diagonal and large negative standardized residuals occur off that diagonal. The sizes
are influenced, however, by the sample size n, larger values tending to occur as n
increases.
In fact, the independence model fits Table 8.7 poorly (G2 = 118.0,df = 9).
Table 8.7 reports the standardized residuals in parentheses. The large positive standardized
residuals on the main diagonal indicate that agreement for each category
is greater than expected by chance, especially for the first category. The off-maindiagonal
residuals are primarily negative. Disagreements occurred less than expected
under independence, although the evidence of this is weaker for categories closer
together. Inspection of cell counts reveals that the most common disagreements refer
to observer Y choosing category 3 and observer X instead choosing category 2 or 4.
8.5.2 Quasi-Independence Model
A more useful loglinear model adds a term that describes agreement beyond that
expected under independence. This quasi-independence model is
log μij = λ + λ X i + λY j + δiI(i = j) (8.12)
where the indicator I(i = j) equals 1 when i = j and equals 0 when i �= j. This
model adds to the independence model a parameter δ1 for cell (1, 1) (in row 1 and
column 1), a parameter δ2 for cell (2, 2), and so forth. When δi > 0, more agreements
regarding outcome i occur than would be expected under independence. Because of