Russel-Research-Method-in-Anthropology

Qualitative Data Analysis I: Text Analysis 513
just by chance. To adjust for this possibility, many researchers use a statistic
called Cohen’s kappa (Cohen 1960), or k.
Cohen’s kappa
Kappa is a statistic that measures how much better than chance is the agreement
between a pair of coders on the presence or absence of binary (yes/no)
themes in texts. Here is the formula for kappa:
Observed Chance
k Formula 17.1
1 Chance
When k is 1.0, there is perfect agreement between coders. When k is zero,
agreement is what might be expected by chance. When k is negative, the
observed level of agreement is less than what you’d expect by chance. And
when k is positive, the observed level of agreement is greater than what you’d
expect by chance. Table 17.7 shows the data in table 17.6 rearranged so that
we can calculate kappa.
Coder 1
TABLE 17.7
The Coder-by-Coder Agreement Matrix for the Data in Table 17.6
Coder 2
Yes No Coder 1 totals
Yes 1 (a) 1 (b) 2
No 3 (c) 5 (d) 8
Coder 2 totals 4 6 10 (n)
The observed agreement between Coder 1 and Coder 2 is:
a d
n
Here, Coder 1 and Coder 2 agreed that the theme was present in the text once
(cell a) and they agreed that the theme was absent five times (cell d), for a
total of 6, or 60% of the 10 texts.
The probability that Coder 1 and Coder 2 agree by chance is:
a b
n
a c
n
c d
n
b d
n
Here, the probability that Coder 1 and Coder 2 agreed by chance is
.08.48.56. Using formula 17.1, we calculate kappa: