Introduction to Categorical Data Analysis

264 MODELS FOR MATCHED PAIRS
8.5.5 Kappa Measure of Agreement
An alternative approach describes strength of agreement using a single summary
index, rather than a model. The most popular index is Cohen’s kappa. It compares
the agreement with that expected if the ratings were independent. The probability
of agreement equals �
i πii. If the observers’ ratings were independent, then
πii = πi+π+i and the probability of agreement equals �
i πi+π+i.
Cohen’s kappa is defined by
�
πii −
κ =
� πi+π+i
1 − � πi+π+i
The numerator compares the probability of agreement with that expected under
independence. The denominator replaces � πii with its maximum possible value of
1, corresponding to perfect agreement. Kappa equals 0 when the agreement merely
equals that expected under independence, and it equals 1.0 when perfect agreement
occurs. The stronger the agreement is, for a given pair of marginal distributions, the
higher the value of kappa.
For Table 8.7, � ˆπii = (22 + 7 + 36 + 10)/118 = 0.636, whereas � ˆπi+ ˆπ+i =
[(26)(27) + (26)(12) + (38)(69) + (28)(10)]/(118) 2 = 0.281. Sample kappa equals
ˆκ = (0.636 − 0.281)/(1 − 0.281) = 0.49
The difference between the observed agreement and that expected under independence
is about 50% of the maximum possible difference.
Kappa treats the variables as nominal, in the sense that, when categories are
ordered, it treats a disagreement for categories that are close the same as for categories
that are far apart. For ordinal scales, a weighted kappa extension gives more
weight to disagreements for categories that are farther apart.
Controversy surrounds the usefulness of kappa, primarily because its value
depends strongly on the marginal distributions. The same diagnostic rating process
can yield quite different values of kappa, depending on the proportions of cases of the
various types. We prefer to construct models describing the structure of agreement
and disagreement, rather than to depend solely on this summary index.
8.6 BRADLEY–TERRY MODEL FOR PAIRED PREFERENCES ∗
Table 8.9 summarizes results of matches among five professional tennis players during
2004 and 2005. For instance, Roger Federer won three of the four matches that he
and Tim Henman played. This section presents a model that applies to data of this
sort, in which observations consist of pairwise comparisons that result in a preference
for one category over another. The fitted model provides a ranking of the players. It
also estimates the probabilities of win and of loss for matches between each pair of
players.