Contents - Max-Planck-Institut für Physik komplexer Systeme

2.15 Probabilistic Forecasting
JOCHEN BRÖCKER, STEFAN SIEGERT, HOLGER KANTZ
A ubiquitous problem in our life (both as individuals
as well as a society) is having to make decisions in the
face of uncertainty. Forecasters are supposed to help in
this process by making statements (i.e. forecasts) about
the future course of events. In order to allow the forecast
user to properly assess potential risks associated
with various decisions, forecasters should, in addition,
provide some information as to the uncertainty associated
with their forecasts. Unequivocal or “deterministic”
forecasts are often misleading as they give the false
impression of high accuracy. Probabilities, in contrast,
allow to quantify uncertainty in a well defined and consistent
manner, if interpreted correctly.
Forecasts in terms of probabilities have a long and
successful history in the atmospheric sciences. In the
1950’s, several meteorological offices started issuing
probability forecasts, then based on synoptic information
as well as local station data. On a scientific (non–
operational) level, probabilistic weather forecasts were
discussed even much earlier.
Evaluation of probabilistic forecasts
Since the prediction in a probabilistic forecast is a probability
(distribution) whereas the observation is a single
value, quantifying the accuracy of such forecasts is
a nontrivial issue. This is of interest not only for quality
control of operational forecasting schemes, but also for
a more fundamental understanding of predictability of
dynamical systems; see for example [5]. Nowadays,
probabilistic weather forecasts are often issued over a
long period of time under (more or less) stationary conditions,
allowing archives of forecast–observation pairs
to be collected. Such archives permit to calculate observed
frequencies and to compare them with forecast
probabilities. The probability distribution of the observation
y, conditioned on our probability forecast being
p, ideally coincides with p; a forecast having this property
is called reliable or consistent. If a large archive of
forecast–observation pairs is available, reliability can
be tested statistically. It is not difficult though to produce
reliable probabilistic forecasts. The overall climatological
frequency for example will always be a reliable
forecast; this constant forecast is not very informative
though. The forecast should delineate different observations
from each other.
The question arises how virtuous forecast attributes
(reliability and information content) can be quantified
and evaluated. One therefore seeks for “scoring rules”
which reward both reliable and informed forecasters.
A scoring rule is a function S(p,y) where p is a probability
forecast and y a possible observation. Here, y
is assumed to be one of a finite number of labels, say
{1...K}. A probability forecast would then consist
of a vector p = (p1 ...pK) with pk ≥ 0 for all k, and
k pk = 1. The idea is that S(p,y) quantifies how well
p succeeded in forecasting y. The general quality of a
forecasting system is ideally measured by the mathematical
expectation E[S(p,y)], which can be estimated
by the empirical mean
E[S(p,y)] ∼ = 1
N
N
S(p(n),y(n)) (1)
n=1
over a sufficiently large data set {(p(n),y(n));n =
1...N} of forecast–observation pairs.
Two important examples for scoring rules are the logarithmic
score [8] (also called Ignorance)
and the Brier score [1]
S(p,y) = −log(py), (2)
S(p,y) =
(pk − δy,k) 2 . (3)
k
The convention here is that a smaller score indicates a
better forecast.
Although these two scoring rules seem ad hoc, they
share an interesting property which arguably any scoring
rule should possess in order to yield consistent results.
Suppose that our forecast for some observation y
is q, then the score of that forecast will be S(q,y). If the
correct distribution of y is p, then the expectation value
of our score is
E(S(q,y)) =
S(q,k)pk. (4)
The right hand side is referred to as the scoring function
s(q,p). Arguably, as p is the correct distribution of y,
the expected score of q should be worse (i.e. larger in
our convention) than the expected score of p. This is
equivalent to requiring that the divergence function
k
d(q,p) := s(q,p) − s(p,p) (5)
be non-negative and zero only if p = q. A scoring
rule with this property (for all p,q) is called strictly
proper [6, 7]. The divergence function of the Brier score
for example is d(q,p) :=
k (qk − pk) 2 , demonstrating
that this score is strictly proper. The Ignorance
is proper as well, since (5) is then just the Kullback–
Leibler–divergence, which is well known to be positive
definite.
The expectation value E[S(p,y)] with strictly proper
scoring rule S allows for the following decomposition
[4]:
ES(p,y) = s(¯π, ¯π) + Ed(p,π) − Ed(¯π,π), (6)
70 Selection of Research Results