Contours of Conflict in the 21st Century

THE METAFORE PROTOCOLThe HCSS team went – within the scope of this project – to somelengths to ensure intercoder reliability 10 :• All teams were trained intensively by the HCSS team at the beginningof the project.• Analysts in the Chinese, Russian and English teams had prior codingexperience.• Extensive documentation on coding, both general and project-specific,was made available (and constantly updated based on lessons learned)on Google Wave.• At the start of the project, the different language domains teams eachinitially blind coded the same three English studies, discussed theirresults and calibrated their coding efforts.• The teams were subsequently supervised by experienced HCSS staffwhile analyzing their specific language domain studies. Whenever theyfelt uncertain about some coding issues, they discussed these on theGoogle Wave – and invariably came to some generally acceptablesolution.After the main coding effort was completed, the results of thecoding efforts of the English and Russian language domain teamswere checked by HCSS staff for accuracy and cross-languagecomparability. We randomly picked ten studies to determine thequality of the coding. We also picked a few variables (both driversand parameters) in Nvivo to see whether the studies that had codedthese variables had been coded in a uniform way. The results of thisvalidation effort were deemed satisfactory for the English and Russianlanguage domains – in the sense that HCSS staff felt comfortable thatthe coding results adequately reflected the key parameters and driverscovered in the studies. In the absence of another in-house nativeChinese speaker, the coding results of the Chinese language domainwere checked on the basis of translated fragments of coded text which10 K. Krippendorff, ‘Reliability in Content Analysis,’ Human Communication Research 30, no. 3(2004): 411-433; M. Lombard, ‘A Call for Standardization in Content Analysis Reliability,’Human Communication Research 30, no. 3 (2004): 434–437; M. Lombard, C. Bracken, and J.Snyder-Duch, ‘Content Analysis in Mass Communication: Assessment and Reporting ofIntercoder Reliability,’ Human communication research 28, no. 4 (2002): 587–604.HCSS Report 33