Analytic Culture in the U.S. Intelligence Community (PDF) - CIA

INSTRUCTIONAL TECHNOLOGY
synthesize the quantitative measures and describe the findings of the analysis.
Meta-analysis appears to be especially suited for synthesizing the results of
instructional research, and it has been widely used for this purpose since its
introduction in 1976.
Meta-analysis is still being developed as a technique, and some matters
concerning its use, notably the “file-drawer” problem and calculation of effect
size, remain unsettled. Chapter Twelve presents a more detailed explanation
and these considerations. Briefly, however, meta-analytic reviews of instructional
technology effectiveness have found substantial results favoring its use
over traditional technologies of classroom instruction.
Overall, effect sizes for post-secondary school instruction average about
0.42, which is roughly equivalent to raising the achievement of 50th percentile
students to that of 66th percentile students. 15 Reviews of more elaborate forms
of instructional technology, such as those using applied artificial intelligent
techniques, have found effect sizes in excess of 1.0, which is roughly equivalent
to raising the achievement of 50th percentile students to that of the 84th
percentile. 16 It seems reasonable to conclude that the reduced costs and
reduced time to learn obtained in applications of instructional technology are
not achieved at the expense of instructional effectiveness.
Encouraging as these favorable results are, our ability to apply instructional
technology efficiently may be in its infancy. Findings thus far have been based
on instructional applications intended to teach facts (e.g., What is the capital
of Brazil? What is the Spanish word for chapel? Who was the first director of
the Central Intelligence Agency?) concepts (e.g., What is a mass spectrometer
used for? What is the difference between micro- and macro-economics? When
must you use a torque wrench?), and procedures (e.g., How do you record a
movie from television? How do you prepare a purchase requisition? How do
you calibrate a radar repeater?). All intelligence analysts must possess a repertoire
of facts, concepts, and procedures to perform their craft, and instructional
technology holds great promise for increasing both the efficiency with which
they might develop this repertoire and their access to instructional resources
for doing so.
However, the capabilities analysts may seek through instruction are likely
to include more abstract, or “higher,” cognitive processes. For instance, in
addition to learning a procedure, analysts may need the capability to recognize
15
Chen-Lin Kulik., James Kulik and Barbara Shwalb, “Effectiveness of Computer-Based Adult
Education: A Meta-Analysis”; Chen-Lin Kulik and James Kulik, “Effectiveness of Computer-
Based Education in Colleges”; Rob Johnston and J. D. Fletcher, A Meta-Analysis of the Effectiveness
of Computer-Based Training for Military Instruction; J. D. Fletcher, “Evidence for Learning
from Technology-Assisted Instruction.”
16
Sherrie P. Gott, R. S. Kane, and Alan Lesgold , Tutoring for Transfer of Technical Competence.
91