Contents - Max-Planck-Institut für Physik komplexer Systeme

where π and ¯π are probability distributions with πk =
P(y = k|p), and ¯πk = P(y = k), respectively. The first
term is the inherent uncertainty of y—this quantity is
not affected by the forecasts. The second term quantifies
reliability; note that this term is always positive unless
p = π, which is the mathematical definition of reliability.
The third term quantifies the information content
of π. It contributes negatively to (i.e. improves) the
score, unless π = ¯π, in which case π is constant. Hence,
this term penalises lack of variability of π; the larger the
variability of π, the better the score. As a whole, the decomposition
If constructed yields credence from a to finite thenumber practiceof ofsamples, assessingthe
forecast Talagrand quality diagram through is subject properto scoring randomrules. fluctuations. In
-diagram, deviations from flatness due to finite
Ensemble samples forecasts are taken into account. Under the assumption of
Modern consistency,thenumberofcasesinwhichtheverification
weather forecasts are generated using large
dynamical atmospheric should models, followrunning a binomial on supercom- distribution
puters. In order to initialise these simulations properly, is
the current state of the atmosphere has to be known
at least approximately, and subsequently projected into
the state space of the model; this process is known as
data assimilation. The fact that the initial condition is
not known with certainty (and also that the model is incorrect)
is accounted for by generating not one but several
simulations with minutely perturbed initial conditions,
resulting in an ensemble of forecasts.
Although ensemble forecasts already provide vital information
as to the inherent uncertainty, they need to
be post-processed before they can be interpreted as
probabilities. The interpretation of ensembles and how
to generate useful forecast probabilities using ensembles
is a very active area of research.
Several different interpretations exist, a very common
one being the following Monte–Carlo interpretation:
An ensemble is a collection x1,...,xK of random variables,
drawn independently from a common distribution
function p, the forecast distribution. The forecast distribution
p can be considered as the distribution of the
ensemble members conditioned on the internal state
of the forecasting scheme. The forecast distribution
p however is but a mental construct and not operationally
available.
In this interpretation, the forecasting scheme is called
reliable if the observation y along with the ensemble
members x1 ...xK are independent draws from the
forecast distribution p. Less formally stated, the observation
behaves like just another ensemble member.
A necessary consequence of reliability is that the rank
of y among all ensemble members assumes the values
1,...,K +1 with equal probability (namely 1/(K +1)).
This means that the histogram of rank(y) should be flat,
which can be statistically tested, see for example [3].
In operational ensembles though, histograms are often
found to be u–shaped, with the outermost ranks
being too heavily populated (see Fig.1 for an example);
in other words, outliers happen more often than
they should in a reliable ensemble. This can have several
reasons, such as insufficient spread or conditional
bias. On the other hand, this means that we should be
able to predict such outliers by looking at characteristic
patterns in the ensemble. Indeed, as we could show,
even for reliable ensemble forecasts, the spread of the
actual specific ensemble is indicative of the probability
that the future observation will be an outlier [10].
v l
0.999
0.99
0.9
0.5
0.1
0.01
0.001
1 52
Figure 1: Rank diagram for temperature forecasts in Hannover: The
verification falls much too often into the first and last bins, indicating
that outliers are too frequent for reliability. This ensemble features
51 members. The y–axis shows Binomial probabilities, rather than
actual counts.
Unfortunately, rank based reliability tests are restricted
to scalar predictions. In [9], a rank analysis for vector
valued predictions was suggested by measuring the
length of a minimum spanning tree. Thereby, a new
scalar ensemble is created which however ceases to be
independent, that is, the Monte–Carlo interpretation no
longer applies. This puts the assumptions behind the
entire rank histogram analysis into question. However,
as it could be shown in [2] the rank based reliability
analysis can still be applied to such ensembles, due to
some inherent symmetries called exchangeability. In
particular, these investigations demonstrated that the
minimum spanning tree approach is mathematically
sound.
[1] Glenn W. Brier. Monthly Weather Review, 78(1):1–3, 1950.
[2] J. Bröcker and H. Kantz. Nonlinear Processes in Geophysics,
18(1):1–5, 2011.
[3] Jochen Bröcker. Nonlinear Processes in Geophysics, 15(4):661–673,
2008.
[4] Jochen Bröcker. Quarterly Journal of the Royal Meteorological Society,
135(643):1512 – 1519, 2009.
[5] Jochen Bröcker, David Engster, and Ulrich Parlitz. Chaos, 19,
2009.
[6] Jochen Bröcker and Leonard A. Smith. Weather and Forecasting,
22(2):382–388, 2007.
[7] Thomas A. Brown. Technical Report RM–6299–ARPA, RAND
Corporation, Santa Monica, CA, June 1970.
[8] I. J. Good. Journal of the Royal Statistical Society, XIV(1):107–114,
1952.
[9] J.A. Hansen and L.A. Smith. Monthly Weather Review,
132(6):1522–1528, 2004.
[10] S. Siegert, J. Bröcker, and H. Kantz. Quarterly Journal of the Royal
Meteorological Society, 2011 (submitted).
2.15. Probabilistic Forecasting 71