Russel-Research-Method-in-Anthropology

512 Chapter 17
sexual men and women offer each other in personal ads in the United States
hasn’t changed much since 1983 (Wiederman 1993; Willis and Carlson 1993;
Goode 1996; Butler-Smith et al. 1998; Cameron and Collins 1998). Of course,
we expect that people of different sexual orientations and people in different
cultures and subcultures will seek and offer different resources in personal
ads, which sets up a program of work on cross-cultural content analysis. (See
de Sousa Campos et al. [2002] for a content analysis of personal ads in Brazil;
see Parekh and Beresin [2001] for an analysis of ads in the United States,
India, and China. And see Yancey and Yancey [1997] for a content analysis of
personal ads of people seeking interracial relationships in the United States.
See Smith and Stillman 2002 on differences in the personal ads of heterosexual,
lesbian, and bisexual women; and see Kaufman and Phua 2003 on differences
in the personal ads of gay and straight men.)
Intercoder Reliability
It is quite common in content analysis to have more than one coder mark
up a set of texts. The idea is to see whether the constructs being investigated
are shared—whether multiple coders reckon that the same constructs apply to
the same chunks of text. There is a simple way to measure agreement between
a pair of coders: you just line up their codes and calculate the percentage of
agreement. This is shown in table 17.6 for two coders who have coded 10 texts
for a single theme, using a binary code, 1 or 0.
TABLE 17.6
Measuring Simple Agreement between Two Coders on a Single Theme
Units of Analysis (Documents/Observations)
1 2 3 4 5 6 7 8 9 10
Coder 1 0 1 0 0 0 0 0 0 1 0
Coder 2 0 1 1 0 0 1 0 1 0 0
Both coders have a 0 for texts 1, 4, 5, 7, and 10, and both coders have a 1
for text 2. These two coders agree a total of six times out of ten—five times
that the theme, whatever it is, does not appear in the texts, and one time that
the theme does appear. On four out of ten texts, the coders disagree. On text
9, for example, Coder 1 saw the theme in the text, but Coder 2 didn’t. Overall,
these two coders agree 60% of the time.
The total observed agreement, though, is not a good measure of intercoder
reliability because people can agree that a theme is present or absent in a text