monitoring

UNCLASSIFIED
DEFENSE SCIENCE BOARD | DEPARTMENT OF DEFENSE
Utilization of risk as an end‐to‐end problem space metric comes with a set of inherent
challenges as well. Common criticisms of formal risk assessment methodologies in decision
processes include:
• Conflating stochastic processes and adversary decisions – Rigorous risk assessment
methodologies rely on estimates of the probabilities of events of concern. In systems
where outcomes are determined by truly random processes, this works quite well,
especially when the system is well characterized. However, well characterized stochastic
processes do not govern intelligent adversaries; instead, they make informed decisions.
Although frequently used, probabilistic representations of adversary decisions are, for
the most part, meaningless. However, characterization of uncertainty about adversary
decisions in a probabilistic analysis can be beneficial, if carefully developed.
• Focusing on absolute values rather than relative impacts and sensitivities – The
absolute values of risk are, in most formulations, arbitrary, as they are built upon the
assumptions and values of the analyst or decision maker for whom they are
constructed. Additionally, the models upon which risk is calculated often cannot be truly
validated. Therefore, the absolute values of risk are less important and somewhat
meaningless, while the differences and relative comparisons can be more telling.
• Inability to define “acceptable” – A key component of making decisions in a risk‐based
framework is to define acceptable risks within the timeframe of the investment decision
itself. If “acceptable” cannot be defined, the goal becomes to simply minimize risk,
expending all resources, rather than achieve minimum risk using appropriate resources.
• Examining risk trade‐offs too narrowly – One of the most common criticisms of
implementations of risk‐based approaches is the lack of definition around uncertainties.
Especially in problems of high uncertainty, the error in risk calculations can be so large,
that close trade‐offs can be interpreted as essentially the same. Not having well defined
uncertainties can allow false comparisons to drive decision processes. At the same
time, ignoring the uncertainty (or error bars) in the analysis denies they opportunity to
prioritize potential efforts to improve understanding.
• Misuse of probability and statistics – While it may seem like an elementary mistake,
bad assumptions or interpretations, especially around dependence or independence,
can lead to mathematical operations and inferences that may be numerically correct,
yet meaningless––or worse, incorrectly calculated.
In the face of those challenges, it is certainly possible, although difficult, to articulate a risk
metric that serves as an end‐to‐end metric within the M&V problem space for evaluating
proposed solution architectures. It may not be necessary to always use the classical definitions
of risk as the primary measure. Most risk studies use the arithmetic product of probabilities of
occurrence and consequences as the standard definition of risk. However, the strictest
interpretation of risk may not always be the most useful or accurate. Instead, it is possible to
use measures that are proximate to risk when performing analyses. For example, other studies
have proposed and formalized using assessments of the difficulty an adversary would face in
DSB TASK FORCE REPORT Appendix A: Unabridged Description | 89
Nuclear Treaty Monitoring Verification Technologies
UNCLASSIFIED