14-1190b-innovation-managing-risk-evidence

level was announced on 22 January 2010 the government’s
Home Secretary felt obliged to say: “This means that a
terrorist attack is highly likely, but I should stress that there
is no intelligence to suggest than an attack is imminent” 1 .
This shows that using words to express uncertainty, without
a numerical reference scale, can be misleading unless
audiences fully understand their usage in the particular
context.
Words can also be used as direct translations of
numerical probability assessments. For example, the
Intergovernmental Panel on Climate Change (IPCC) has
standardized its verbal terms for likelihood using the scale in
Table 1 (ref.2).
The IPCC sometimes use these terms for communicating
their confidence in scientific conclusions, for example in
stating that: “It is extremely likely that human influence
has been the dominant cause of the observed warming since
the mid-20th century”. The IPCC only uses these terms of
likelihood in situations of ‘high’ or ‘very high’ confidence
(see below for the interpretation of this term), although it
can be argued that such a restriction is unnecessary.
It is important that any numerical scale broadly reflects
usage in that domain. The case study in this chapter, Accurate
Communication of Medical Risk, notes that drug side-effects
that are described as ‘common’ are intended to correspond
to between 1% and 10%. This may be reasonable usage for
pharmacologists, but not for patients.
4. Being imprecise about numbers
When it comes to expressing uncertainty in quantitative
terms, a hierarchy of levels of precision can be assigned,
appropriate to the confidence of the assessors. For example:
• Numbers to appropriate levels of precision.
• A distribution or range.
• A list of possibilities.
• A rough order of magnitude.
TABLE 1
IPCC LIKELIHOOD SCALE
Term
Likelihood
of the outcome
(probability)
Virtually certain 99–100%
Extremely likely 95–100%
Very likely 90–100%
Likely 66–100%
More likely than not 50–100%
About as likely as not 33–66%
Unlikely 0–33%
Very unlikely 0–10%
Exceptionally unlikely 0–1%
No analysis can claim to
produce the risk of an
innovation.
5. Expressing (lack of) confidence in the science
Our uncertainty does not just concern the likelihood of
future outcomes. For a whole modeling process, we may
have to contend with assumptions about the inputs, the
model structure, and the issues of concern. Again, lists of
possible values or scenarios can be provided as a sensitivity
analysis. These can be supplemented by acknowledging that
there are aspects that have been left out of the model, with
a qualitative assessment of their potential impact.
Some domains have tried to use a summary qualitative
scale to communicate the confidence in the analysis. For
example, the IPCC uses the quality of the evidence, and
the degree of scientific agreement, to assess a level of
confidence in their scientific conclusions.
Seeking a common language
There is room for substantial improvement in the quality
of the discourse between the participants involved in
innovation and risk, including the public, regulators,
innovators, NGOs and the media. But “seeking a common
language” does not refer to a restricted use of specific
words, such as risk, uncertainty and ambiguity — that would
be impossible to impose and counter-productive to try.
Rather, by “common language” I mean the acceptance of a
set of principles regarding the presentation of arguments or
analyses. Five such principles are outlined below.
1. Acknowledge that any risk and uncertainty assessment is
contingent and provisional. No analysis can claim to produce
the risk of an innovation. Such assessments are constructed
depending on assumptions, and may change in receipt of
further information. The reasons for the uncertainty should
also be given. And since there will be competing opinions,
not all of which will be based on an adequate consideration
of the evidence, the pedigree of an analysis is also important.
It would be useful to have an appraisal of the quality of
the analytic framework: this could be self-assessed by the
analysts (although this would require considerable humility)
but also assessed by the team responsible for formulating
the policy in question. The appraisal should be based on
the quality, quantity, consistency and coverage of evidence,
as well as the quality of the process that collected and
analyzed the evidence, and the quality of deliberation.
Policymakers can then make a holistic summary of the state
of uncertainty and how it influences the confidence in the
conclusion (see Table 2 for a possible rough categorization
of such an assessment, roughly modeled on the Grading of
Recommendations Assessment, Development and Evaluation
(GRADE) scale used in health 3 ).
Clearly the strength of any recommendation does not
75