GPANet-Report-2012

23 Birger Heldt
Mass Atrocities Early Warning Systems:
Data Gathering, Data Verification, and Other Challenges
manner, independently of each other, and across any case, and that conclusions/
predictions can be replicated given the use of identical data. Unless this is achieved,
early warning falls back to being an art form, without transparency, without a
rigorous and systematic character. To exemplify, with the help of Harff’s (2003)
transparent model and findings, any policy planner can calculate new and replicate
old genocide risk scores for any country. For early warning models to become useful
for practitioners, it is important to raise the bar to this level.
The detailed knowledge of country and case experts is impressive and valuable, but
has in general been found to be an imperfect basis for predictions. Citing findings
from a study that covered 27,000 judgements/predictions covering 55 countries
over 20 years, Tetlock (2011) reports that 1/3 of the experts did not fare better than a
random guess generator, whereas 60% of the experts did not outperform the simple
decision rule “tomorrow will be just like today” (i.e., if it is peace this year, then predict
there will be peace next year too). Corroborating findings are reported by Green &
Armstrong (2007) in that expert forecasts built on the assumption that the case was
unique, were not more accurate than forecasts made by novices, and the accuracy was
pretty much identical to what could have been expected by chance alone (Ibid.: 12).
Interestingly, experts’ success rate increased sharply (39%) when they were asked to
base forecasts on outcomes from similar cases in the past - or retrospectively based
predictions – instead of treating the cases as unique. 13
These findings do not imply that ”unique case assumption” experts do not provide
accurate predictions, or that some case experts do not have a very good track record of
accurate predictions. What can be inferred is instead that in general retrospectively
based predictions - and hence predictions that are based on the assumption that there
are empirical patterns and general causes - outperform the unique case assumption
approach.
Given that retrospectively based predictions based on general patterns are overall
superior to the unique case assumption approach, should predictions be based on an
inventory of findings from case studies, or on findings from large comparative studies
(i.e., statistical studies)? Large comparative studies offer broad retrospective lessons
and empirical scope, but at the cost of detailed, case-specific insights and texture that
are the hallmark of case studies. The predictions are also far from perfect. But scope,
broad and sweeping lessons are valuable in that the “big picture” helps us think more
13 An analogy is medical doctors that regard every patient as unique, yet approach symptoms, analysis,
treatments and predictions on the retrospectively based knowledge and experience of the medical field.