Analytic Culture in the U.S. Intelligence Community (PDF) - CIA

CHAPTER NINE
Development and Validation of Analytic Methods. The Intelligence Community
routinely generates ad hoc methodologies to solve specific analytic
problems or crises. However, once the problem has been solved, or the crisis
averted, the new analytic method may or may not become part of the institutional
memory. Often these new methods are lost and need to be re-created to
address the next problem or crisis. In addition, these methods are seldom
tested against other competing analytic methods for validity or reliability. It is
difficult for an analyst to know which analytic method to employ in a given
situation or requirement.
There are obvious inefficiencies in the current model. First, there is the loss
of corporate knowledge each time an innovative analytic method is generated
and subsequently abandoned. Second, there is no effectiveness testing center
where analytic methods can be compared for specific cases. Although there
are hundreds of analytic strategies, there is no way to determine which strategy
is the most effective for any particular problem set.
The lack of an analytic methodology research agenda leads analysts to
choose methods with which they are most familiar or to choose those dictated
by circumstance, such as deadlines. Moreover, instead of advancing the concept
that intelligence analysis is science and needs to be engaged in like any
other scientific discipline, the paucity of effectiveness data supports a deepseated
community bias that analytic methods are idiosyncratic and, therefore,
akin to craft.
The development of a research agenda for analytic methodology that is
focused on collecting effectiveness and validation data is the first step in moving
intelligence analysis from a tradecraft model to a scientific model. This
may be the most culturally difficult recommendation to implement: there is
cultural resistance to adopting a science-based model of intelligence analysis
that is rooted in the traditions, norms, and values of the Intelligence Community.
Another difficult step will be to introduce effectiveness data and corresponding
analytic methods to the community at large and to incorporate these
in future training programs.
Training Effectiveness. Successful analysis demands group cohesion and
the implementation of consistent, effective analytic methods within the Intelligence
Community. The best way to achieve this is through formal basic and
advanced training programs. As noted earlier, several agencies within the
community have invested resources in formal training programs, but these
programs are unique to each agency and are often missing evaluations of student
performance. Although most formal courses include a written subjective
evaluation of the instructor, as well as the student’s perception of the value of
the course, the evaluation of student performance has yet to be formalized.
114