l I - National Criminal Justice Reference Service

REPORT ON THE PLAYBOY CARTOON PILOT STUOY
COMPUTATION OF INTER-RATER RELIABILITY
I. Conceptual Description
A pilot study was conducted using version eight oF the
Cartoon Content Analysis Coding Instrument (CCACI), wherein
40 cartoons were each coded ~y two indiviOual coders.
Following completion of the coding process, Inter-coOer
Reliability (IRR) was determined in order to proviOe a
quantitative check on the overall level of agreement
~etween the responses of individual coders in this
cartoon coding process. Specifically, IRR was
operationalized as the total number of cases where the
responses of the two coders agreed (for a given question
pertainlng to a given cartoon) divided by the total number
of required responses. The score o~tained was 0.87 (See
Figure A).
The major objective in determining IRR was to identify
problems with coder bias, the coder training process, and/or
the CCACI. Such an analysis was necessary, since the
primary purpose of the pilot study was to serve as a
learning tool rather than..as a definitive test of the
current version of the CCACI.
II. Operational Description
Forty Playboy child cartoons examined within the context of
each total magazine, were each coded independently by two
separate coders, resulting in 80 completed data sheets.
(Four coders participated in the IRR test for the pilot
study; each participated In a training process during the
summer of 1984.) The Four coders followed the CCACI, which
called for 37 responses for each cartoon. For the IRR
check, nine coded magazines were randomly chosen from oaOyear,
1971-1979, Playboy magazines.
L
After completion of the coming process, the data sheets
for each cartoon were paired. Two people not invoIve~ in
the coding process cross-checke~ the 37 responses for each
pair of data sheets. The number o4 discrepancies per cartoon
was counted and logged on a form together with the following
information: the month/year o4 the magazine, the page
number o4 the cartoon, the question number of each
discrepant answer,, and each coder's indiviOual response.
Each pair o4 data sheets was checked by both the monitors So
guarantee accuracy. The number of discrepancies per magazine
was then counted and logged on a similar form together with
the following information: the month/year of the magazine,
the number o4 codeO cartoons, the page number o4 each
cartoon, and the number of discrepant answers per cartoon.
Once these data ~ere collecteO, recor~e~, and checkeO, the IRR
60