Analytic Culture in the U.S. Intelligence Community (PDF) - CIA

RECOMMENDATIONS
to determine and predict the effect on organizational performance. The same
is true with the Intelligence Community. Although there have been numerous
proposals to reorganize the Intelligence Community—including those that
resulted from the hearings of the Kean 9/11 commission—few have addressed
the question of why one change would be any more effective than any other
change. Merely asserting, based on some a priori notion of effectiveness, that
organizational scheme X is more effective than organizational scheme Y is
insufficient evidence. What is needed is a posteriori data, such as case studies,
to support or refute the proposed change. 1
Organizational Requirements. Many large organizations distribute performance
improvement responsibilities throughout the organization at a supervisory
or midlevel of management, but the group most often charged with
collecting and analyzing performance data is the human resources department.
This task generally involves developing task-specific performance standards
and metrics based on expert performance models and in accordance with corporate
policy.
The human resources department also becomes the central repository for
pre-, periodic, and post-performance measurements. As this department generally
has contact with employees throughout their careers, this is the most
efficient way to manage, analyze, and inform senior leadership about aggregate
changes in performance over time. Although data are collected at the
individual level, it is the aggregation of performance data that allows leadership
to determine the effectiveness of any organizational change or job-related
intervention.
Baseline Data. Measuring actual analytic performance is essential to the
establishment of a data driven performance infrastructure. The analysts in this
study perceived their performance to be tied directly to the quantity of written
products they produced during each review period. Counting the number of
analytic publications is one metric, of course, but it is hardly indicative of analytic
quality. Surgeons are a useful example of this problem.. They may
count the number of patients they treat, but this metric says more about system
throughput and salesmanship than it does about surgical performance. Unlike
the purely cognitive work of intelligence analysts, surgeons have the advantage
of multiple physical outputs, which makes measurement an easier task.
In particular, surgeons have patient outcomes, or morbidity and mortality
ratios, which become a grounded end-state for all measurements. 2 Other
things being equal, these data then ought to inform a prospective patient about
where to take his or her business.
1
William Nolte, a deputy assistant director of central intelligence for analysis and production proposed
such an idea in “Preserving Central Intelligence: Assessment and Evaluation in Support of
the DCI ” in Studies in Intelligence 48, no. 3 (2004): 21–25.
109