The Jeremiad Over Journalism

The code book and the coding scheme was specifically constructed to code for manifest, not latent,
meaning in the texts and after two rounds of pilot-coding, a full-scale content analysis with a
comprehensive intercoder-reliability-test by a second coder was conducted. 535
Based on previous content analysis studies, Neuendorf recommends a reliability subsample size
between 10% and 20% and advocates keeping this sample size larger than 50 and rarely ―larger than
about 300.‖ 536 Consequently, to adhere to Neuendorf‘s guidelines, a pool of 100 articles out of the
entire sample of 876 was randomly selected using the website random.org.
The 100 articles (11,4 percent of the total number of articles) selected for a test of intercoder-
reliability therefore falls well within Neuendorf‘s recommendation and follows her prescription that
―the establishment of intercoder reliability is essential, a necessary criterion for valid and useful
research when human coding is employed.‖ 537
Through the statistical program PASW, the latest version of SPSS, a reliability-test taking into
account chance agreement between coders was chosen. Neuendorf points out that no agreed-upon
standards for presenting intercoder-reliability is prevalent, and therefore ―the best we can expect at
present is full and clear reporting of at least one reliability coefficient for each variable measured in
a human-coded content analysis.‖ 538
To conform with Neuendorf‘s recommendations the reliability results in this study will be reported
using Krippendorff‘s alpha as the reliability-coefficient and reporting this for each variable coded
by the two coders. Furthermore, examples of the coding will be provided throughout the analysis, so
the readers have an opportunity to follow the research process and the line of reasoning developed
based on the content analysis results. Through this approach, a quantitative foundation for further
discussion of the hypotheses‘ validity will be laid. As Neuendorf has pointed out, reporting
intercoder-reliability variable by variable yields a more precise picture of the actual reliability, as
variables with extremely high reliability cannot balance out variables with low reliability scores.
535 Neuendorf, The Content Analysis Guidebook. Page 146.
536 Ibid. Page 158-159. In Daniel Riffe, Stephen Lacy, and Frederick G. Fico‘s writing a ―random selection of content
samples for reliability‖ is recommended. On the amount of text to be tested the authors state that advice has been
ambiguous, but, ―One text (Wimmer & Dominick, 2003) suggests that between 10% and 25% of the body of content
should be tested. Others (Kaid & Wadsworth, 1989) suggested that between 5% and 7% or the total is adequate.‖ Daniel
Riffe, Stephen Lacy, and Frederick G. Fico, Analyzing Media Messages: Using Quantitative Content Analysis, 2nd ed.
(New Jersey: Lawrence Erlbaum Associates, Publishers, 2005). Page 143.
537 Neuendorf, The Content Analysis Guidebook. Page 142.
538 Ibid. Page 144.
172