Russel-Research-Method-in-Anthropology

514 Chapter 17
.6 .56
k
1 .56 .0909
In other words, the 60% observed agreement between the two coders for the
data in table 17.6 is about 9% better than we’d expect by chance. Whether
we’re talking about agreement between two people who are coding a text or
two people who are coding behavior in a time allocation study, 9% better than
chance is nothing to write home about.
Carey et al. (1996) asked 51 newly arrived Vietnamese refugees in New
York State 32 open-ended questions about tuberculosis. Topics included
knowledge and beliefs about TB symptoms and causes, as well as beliefs
about susceptibility to the disease, prognosis for those who contract the disease,
skin-testing procedures, and prevention and treatment methods. The
researchers read the responses and built a code list based simply on their own
judgment. The initial codebook contained 171 codes.
Then, Carey et al. broke the text into 1,632 segments. Each segment was
the response by one of the 51 respondents to one of the 32 questions. Two
coders independently coded 320 of the segments, marking as many of the
themes as they thought appeared in each segment. Segments were counted as
reliably coded if both coders used the same codes on it. If one coder left off a
code or assigned an additional code, then this was considered a coding disagreement.
On their first try, only 144 (45%) out of 320 responses were coded the same
by both coders. The coders discussed their disagreements and found that some
of the 171 codes were redundant, some were vaguely defined, and some were
not mutually exclusive. In some cases, coders simply had different understandings
of what a code meant. When these problems were resolved, a new,
streamlined codebook was issued, with only 152 themes, and the coders
marked up the data again. This time they were in agreement 88.1% of the
time.
To see if this apparently strong agreement was a fluke, Carey et al. tested
intercoder reliability with kappa. The coders agreed perfectly (k 1.0) on
126 out of the 152 codes that they’d applied to the 320 sample segments. Only
17 (11.2%) of the codes had final k values 0.89. As senior investigator,
Carey resolved any remaining intercoder discrepancies himself (Carey et al.
1996).
How much intercoder agreement is enough? As with so much in real life,
the correct answer, I think, is: it depends. It depends, for example, on the level
of inference required. If you have texts from single mothers about their efforts
to juggle home and work, it’s easier to code for the theme ‘‘works full-time’’
than it is to code for the theme ‘‘enjoys her job.’’