monitoring

UNCLASSIFIED
DEFENSE SCIENCE BOARD | DEPARTMENT OF DEFENSE
metrics through this process is not a trivial task. Moving down the chain through strategic
capability areas, functional objectives, tasks, and assets, metrics of increasing resolution and
granularity are derived. At the asset level, the metrics center largely on performance
specifications that are technology specific. These metrics are familiar in radiation detector
assessments, for example, but on their own, are only implicitly related to the overall goals of
risk‐reduction. A metrics derivation process such as this places each metric in the context of the
layer above it, explicitly linking it to overall architecture performance.
4.5. Portfolio Decision Methodologies
In order to render the analytical results that produce well‐characterized architectural options
from the approach described above into investment roadmaps, a decision framework must be
established. While this section does not attempt to propose a decision framework, it does
provide some considerations for doing so.
4.5.1. End‐to‐End Metrics
Risk (or risk minimization) is most often the implicit or explicit top‐level metric for the decision
maker. It provides a metric for endogenous trade‐offs within the M&V problem space, allowing
for the comparison of very different solution sets and examination of benefit between
investments both within and across different components of the problem space itself. Utilizing
risk as an end‐to‐end metric in the M&V problem space can also enable exogenous trades, as
governments face economic challenges and must make tougher decisions about where to
invest resources.
Utilization of risk as an end‐to‐end metric comes with a set of inherent challenges, however.
Common criticisms of formal risk assessment methodologies in decision processes include:
1. Conflating stochastic processes and adversary decisions – Well characterized
stochastic processes do not govern intelligent adversaries; instead, they make
informed decisions. Although frequently used, probabilistic representations of
adversary decisions are, for the most part, meaningless. However, characterization of
uncertainty about adversary decisions in a probabilistic analysis can be beneficial, if
carefully developed.
2. Focusing on absolute values rather than relative impacts and sensitivities – The
absolute values of risk are, in most formulations, arbitrary, as they are built upon the
assumptions and values of the analyst or decision maker for whom they are
constructed. Additionally, the models upon which risk is calculated often cannot be
truly validated.
3. Inability to define “acceptable” – A key component of making decisions in a riskbased
framework is to define “acceptable” risks within the timeframe of the
investment decision itself, something often difficult to achieve, especially when
multiple equities are impacted.
DSB TASK FORCE REPORT Chapter 4: Address the Problem | 46
Nuclear Treaty Monitoring Verification Technologies
UNCLASSIFIED