Analytic Culture in the U.S. Intelligence Community (PDF) - CIA

CHAPTER THREE
load model, David Meister’s behavioral model, and the cognitive process
model of Gary Klein and his colleagues. 25 Each model focuses on a different
aspect of human performance: White’s examines the actual task and task
requirements; Meister’s looks at the behavior of individuals performing a
task; and Klein’s uses verbal protocols to identify the cognitive processes of
individuals performing a task.
Surveying the literature. My research began with a review of the literature,
both for background information and for the identification of variables. The
intelligence literature produced by academics and practitioners tends to be
episodic, or case-based. This is not unique to the field of intelligence. A number
of disciplines—medicine, business, and law, for example—are also casebased.
Many of the texts were general or theoretical rather than episodic.
Again, this is not an uncommon phenomenon. The review yielded 2,432 case
studies, journal articles, technical reports, transcripts of public speeches, and
books related to the topic. I then narrowed the list to 374 pertinent texts on
which a taxonomy of intelligence analysis could be built, and I analyzed them
to identify individual variables and categories of variables that affect intelligence
analysis. 26
Using a methodology known as “Q-Sort,” by which variables are sorted and
categorized according to type, I read each text and recorded the variables that
each author identified. 27 These variables were then sorted by similarity into
groups. Four broad categories of analytic variables emerged from this process.
28
Refining the prototype. Next, I used the preliminary taxonomy derived from
my reading of the literature to structure interviews with 51 substantive experts
and 39 intelligence novices. In tandem, I conducted two focus group sessions,
with five individuals in each group. As a result of the interviews and focus
group discussions, I added some variables to each category, moved some to
different categories, and removed some that appeared redundant.
Testing in a controlled setting. Finally, to compare the taxonomy with specific
analytic behaviors, I watched participants in a controlled intelligence
analysis–training environment. Trainees were given information on specific
24
Alexander Ervin, Applied Anthropology. See Chapter One, note 4 for a definition of triangulation.
25
Robert White, Task Analysis Methods; David Meister, Behavioral Analysis and Measurement
Methods; G. Klein, R. Calderwood, and A. Clinton-Cirocco, Rapid Decision Making on the Fire
Ground.
26
A copy of the list and search criteria is available from the author.
27
William Stephenson, The Study of Behavior: Q-Technique and its Methodology. See Chapter
Eleven for additional information on this methodology.
28
I would like to credit Dr. Forrest Frank of the Institute for Defense Analyses for his suggestions
regarding the naming convention for the categories of variables in the accompanying chart.
38