Statistical Methods in Medical Research 4ed

An alternative set of weights, which penalizes disagreements by the square of
the number of categories of disagreement, is
wi ˆ 1
2 : …19:50†
…k 1†
Although k provides a measure of agreement within a particular set of data
its use as a more general measure is limited. This is because its value is
dependent not only on the intrinsic agreement of the two raters but also on the
prevalence of the condition being assessed or, for multicategorical conditions,
the distribution across the categories (Kraemer, 1979). Thus k is not a general
measure of agreement between raters but only of their agreement in a particular
situation.
The problems discussed in the previous paragraph are analysed in detail by
Cicchetti and Feinstein (1990) and Feinstein and Cicchetti (1990). They proposed
that, just as agreement between a diagnostic test and disease status would not
normally be summarized in terms of a single index, but in terms of sensitivity for
the disease positives and specificity for the disease negatives, so also agreement
between raters should be expressed by two measures. The situations are not
identical, since in the former disease status represents the true situation against
which the diagnostic test is assessed, whereas in the latter the two raters have to
be treated on an equal basis. The measures proposed are ppos, the number of
agreed positives divided by the average number of positives for the two raters,
and pneg defined similarly for the negatives. That is,
ppos ˆ
pneg ˆ
For the three situations considered earlier
1
2
19.10 Kappa measure of agreement 703
i 2
a
2a
1
ˆ
2 ‰…a ‡ b†‡…a ‡ c†Š 2a ‡ b ‡ c
d
2d
ˆ
‰…c ‡ d†‡…b ‡ d†Š 2d ‡ b ‡ c :
A B Example 19.9
ppos 0 50 0 88 0 94
pneg 0 94 0 92 0 78
k 0 44 0 79 0 72
In A and B the two raters have similar agreement when the condition is absent,
but in B the raters have much better agreement when the condition is present.
The two measures provide information about the areas of agreement and
disagreement which is not identifiable from a single measure.