Download the eBook (8.25 MB) - ECREA Thematic Sections

Diversity of Journalisms. Proceedings of ECREA/CICOM Conference, Pamplona, 4-5 July 2011
Aims
We should recall that reliability is a necessary, although not sufficient, condition of
validity. When it comes to speaking of reliability, Stemler (2001) distinguishes between
(a) stability, the intra-rater reliability – can the same coder get the same results try after
try? or (b) reproducibility, the inter-rater reliability – do coding schemes lead to the
same text being coded in the same category by different people? However, as
Krippendorff (1990: 192) affirms, we must bear in mind that “in content analysis,
reliability and validity are related by the two following propositions (a) reliability sets
limits on the potential validity of the research results; and (b) reliability does not
guarantee the validity of the research results”. Thus, according to Andrén (1981: 46)
"we find a connection between the concepts of objectivity and of reliability. This is only
as it should be; it is natural to assume that an objective result is independent of the
subject who conducted the investigation. Here, however, we must distinguish between
the factual or ontological problem – what makes the result true? – and the epistemic or
methodological problem – how do we come to know that a result is true or false –”.
Intercoder reliability
Our communication, as we have said, is centered on analyzing the reproducibility of the
content analysis developed within the project as a whole. Therefore, with the aim of
providing our study with the necessary intercoder reliability, we consider it a priority to
carry out a test that measures that level of agreement before starting the analysis of
the entire sample.
As Lombard, Snyder and Campanella (2002:589) affirm “it is widely acknowledged that
intercoder reliability is a critical component of content analysis and (although it does not
ensure validity) when it is not established, the data and interpretations of the data can
never be considered valid”. The United States General Accounting Office, GAO,
(1996:64) states in one of its reports “this measure indicates how well two or more
coders reached the same judgments in coding the data”. In its turn, the GAO (1996:36)
considers its use necessary given that “in many circumstances, evaluators can make
numerical estimates of intercoder reliability and use the results to judge the readiness
of coders to proceed from training to actual coding”.
263