Analytic Culture in the U.S. Intelligence Community (PDF) - CIA

FINDINGS
I’m looking for links and patterns. Once I figure out the pattern, I
can figure out where to look next.
I use patterns. If things start happening, out of the ordinary things, I
pay attention to them.
I try to build patterns out of the data. It helps me predict what will
happen next.
I come up with a few scenarios and see what the evidence supports.
I look for data that are diagnostic: some piece of evidence that rules
out certain possibilities.
I try to weigh the evidence to see which scenario it supports.
Although anomaly-detection, pattern-recognition, and weighing data may
appear to be idiosyncratic tradecraft based on individual expertise and cognitive
skills, these methods can be formalized and replicated if the operating
parameters, variables, and rules of evidence are made explicit. 10 This is to say
that intelligence analysis can be reconstructed in the context of a scientific
method, which is merely an articulated, formal process by which scientists,
collectively and over time, endeavor to put together a reliable, consistent, and
nonarbitrary representation of some phenomena. Broadly, the steps include:
• observation and description of phenomena;
• formulation of hypotheses to explain phenomena;
• testing of hypotheses by independent experts;
• refutation or confirmation of hypotheses.
These steps do not suggest that any specific scientific methodology results
in what is ultimately the truth, rather that scientific methods are merely formal
processes used to describe phenomena, make predictions, and determine
which hypothesis best explains those phenomena. The principal value of any
type of methodological formalism is that it allows other researchers to test the
validity and reliability of the findings of any other researcher by making
explicit, and therefore replicable, the means by which anyone reaches a specific
conclusion. 11
The idea that intelligence analysis is a collection of scientific methods
encounters some resistance in the Intelligence Community. The interview data
analyzed in this study highlight many subtle—and not so subtle—prejudices
10
A corollary to these methods can be found in the practice of radiologists. See Chapter Five for
more on expertise.
19