learning with professionals - Higgins Counterterrorism Research ...

Regardless of its use, “no piece of evidence can be used in the state in which it is
found.” 86 Whenever an analyst considers new evidence, the critical mind must follow a
systematic interrogatory:
Are these data genuine?
How do I know?
What does this evidence state or imply?
Who is its author or maker and what degree of access does he have? 87
What is the relation in time and space between the author and the information,
overt or implied, that is conveyed by the information?
How does the statement compare with other statements on the same point?
What do we know independently about the author and his credibility? 88
The product of critical inquiry is a judgment about the credibility and reliability of the
evidence and its source. 89 Credibility (a measure of confidence) is only partly based on
reliability (a measure of consistency), since past performance is never perfectly repeated.
Pertinent data may not be available when an analyst begins to verify a piece of evidence.
Under these circumstances, common sense and educated guesswork may be the
only means of answering the questions asked in the evaluation of evidence. The specific
basis of the answers is important and must be recorded. In this way, resort to intuition as
the basis for decisionmaking is avoided. The key to the process is not that all of the
answers to the inquiry are based on fully verified facts; rather, that the process is systematic
and each answer is documented.
Systematic Analytical Method. When an analyst systematically verifies data, authenticates
a source, or accepts the likelihood of a hypothesis, important benefits ensue if the
details of the process are recorded. Aside from serving as a paper trail that proves an evaluation
was accomplished, documentation will hold crucially important information: the degree
of rigor involved, what was known and unknown, what the analyst had confidence or doubts
in, and a level of plausibility or acceptability of the evidence. A record of the manner in
which the analyst evaluated the evidence also exposes the analyst’s assumptions and values
for consistent application when other evidence arises. Finally, these records can be available
for periodic review and reassessment by both the analyst and others who need to understand
the analyst’s line of reasoning to evaluate the acceptability of the conclusions.
86 Barzun and Graff, 156.
87 There are four levels of verifiability: personal observations, reports, inferences, and assumptions. These
levels correspond to the directness of the source to the information being reported. Karl Albrecht, Brain Power:
Learn to Improve Your Thinking Skills (Englewood Cliffs, New Jersey: Prentice-Hall, 1980), 121.
88 Barzun and Graff, 157-158.
89 Credibility is a level of confidence in someone’s or something’s credit-worthiness. Reliability refers to the
repeatability, including intrasubjective replicability, of observation. Suppose a technical surveillance system or
human source produces approximately the same set of responses on repeated trials (and with different analysts
or debriefers), we can say that the observational technique/source has high reliability, regardless of the actual
validity of the findings. Pertti J. Pelto, Anthropological Research: The Structure of Inquiry (New York: Harper
and Row, 1970), 42.
60