Chapter 7 - Ensemble methods.pdf

Chapter 7
Ensemble methods

Each of the following introduces some
error into the modeling process
• Initial conditions
• Lateral and upper boundary conditions
• Specification of the lower-boundary conditions
• Numerical approximations to the equations
• Parameterizations of physical processes
The sum of the errors can be very case/date
dependent

The essence of ensemble prediction
• Perform parallel forecasts or simulations using
different arbitrary choices for the previous
imperfect data or methods.
• This “samples the uncertainty space”
associated with the modeling process, to see
how this uncertainty projects onto the
forecast.

Example ensemble forecast – track
prediction for hurricane Katrina, from the ECMWF
ensemble prediction system. The heavy line is from the
deterministic model forecast

Deterministic vs. ensemble prediction
• Without ensemble prediction (deterministic
forecast) – there is one forecast state of the
atmosphere, and the user has no information
about the confidence we should place in it.
– Compared to yesterday’s forecast
– One part of the model grid compared to others
• With ensemble prediction (probabilistic forecasts)
– we get a better forecast of the most probable
conditions, and we get uncertainty information
(“spread” of ensemble members).

Benefit of ensemble method
• 1) The mean of the ensemble of forecasts is generally
more accurate than the forecast from an individual
member, for a large number of forecasts.
• 2) The spread or variance or dispersion of the
ensemble members can be related to the uncertainty
in the ensemble mean.
• 3) The probability distribution (or density) function
(PDF) can provide information about extreme events.
• 4) Quantitative probabilistic products can be moreeffectively
employed in decision-support systems

Historical context
• Since the 1960’s it has been shown that
combining human forecasts from different
forecasters produces a group-mean forecast
that is superior – “consensus forecasting”.
• Forecasting paradigm
– When models agree…good confidence in forecast
– When models disagree…low confidence

Compromises with
ensemble prediction
• Running many parallel models is computationally
expensive.
• Must use coarser horizontal grid increment for
the ensemble, than you would use for a single
deterministic forecast.
• If you increased the grid increment by a factor of
4, how many ensemble members could you run
with the same hardware?

Compromises with
ensemble prediction
• Running many parallel models is computationally
expensive.
• Must use coarser horizontal grid increment for
the ensemble, than you would use for a single
deterministic forecast.
• If you increased the grid increment by a factor of
4, how many ensemble members could you run
with the same hardware?
64

The spread among ensemble
members, and uncertainty
Cyclogenesis?
No Cyclogenesis?
Eight member ensemble with slightly different initial conditions.
• The “x” shows the ensemble mean at each time.
• Circles show the trajectory of individual ensemble members.
• Solid line shows the “trajectory” of the initial mean of the ensemble.

High
predictability
Example of
different spreads
of ensemble
members
London 2-m AGL
temperature forecasts
from ECMWF, initialized
on different dates.
Low
predictability
Different initial
conditions and model
configurations

Another way of looking at day-to-day variability in predictability
that can be captured by ensemble methods: Temporal variability
of single model predictive skill - 500-mb heights in NH

Defining the ensemble members
• Initial-condition uncertainty – how should it
be done???
• Physical-process parameterization uncertainty
– again, how?
• Errors in the dynamical core – i.e., numerical
algorithms – how?
• LBC uncertainty (If a ALM) – how?
• Surface BC uncertainty

Some comments
• With sequential data-assimilation systems, where the
model forecast is used as a first-guess in the next cycle,
model error and initial-condition error are closely
related.
• Multi-model ensembles - using completely different
models to construct the ensemble - make sense if the
models are already being run for the same geographic
area.
• Super-ensembles involve running ensembles for
individual models, and then combining the multiple
sets of ensemble members into a single ensemble.

Verification of ensemble predictions
• The ensemble mean – can be verified the same
way as a deterministic forecast.
• The uncertainty information (e.g., 2 nd moment)
– Reliability diagrams

Reliability diagrams – reliability is an
important attribute of ensemble forecasts of
dichotomous events (ones that occur or do not
occur at a grid point or over an area)
Using a large number of archived ensemble forecasts, select the forecasts
where the forecasted probability of occurrence of an event was p f . Of
those events, what was the observed frequency of occurrence p o ? Do this
for all the different thresholds in the forecast probabilities, and plot as
above.

An example of a real reliability diagram
Single-model IC ensemble
Multi-model super-ensemble

Verification of ensemble predictions
• The ensemble mean – can be verified the same
way as a deterministic forecast.
• The uncertainty information (e.g., 2 nd moment)
– Reliability diagrams
– Rank histograms

Rank histograms
• Also called verification rank histograms and
Talagrand diagrams.
• Define the bias of ensemble predictions
• For a specific variable and location of an
observation, take the ensemble forecast for
that variable at that location and rank-order
the forecasts from each of the members.
• Then define the n+1 intervals that are
bounded by the n ordered forecast variables.

Example with 4 ensemble members
and 5 intervals, for variable P
For this location and
time t forecast , the
observed P (X obs ) is
lower than any of
the forecast Ps and
the observation is
thus in I 1 .

• Follow the same process for all other pairs of
observations and forecasts at this time.
• Calculate the total number of observations in
each of the five intervals, or ranks, and plot a
histogram of the frequency.
• This provides a graphical view of how the
ensemble of forecasts relates to the observations.
• A non-uniformity in the distribution reveals
systematic errors in the ensemble.

• Over-forecasting bias – Observations tend to fall in the
lower intervals of the ensemble. Forecast values are
generally too high.
• Under-forecasting bias – Observations tend to fall in
the upper intervals of the ensemble. Forecast values
are generally too low.
• Rank uniformity – Ideal situation
• Under-dispersion – Observations tend to fall in the
upper and lower intervals. Forecasts are too similar to
each other…need more spread.
• Over-dispersion – Observations tend to fall in the
center intervals. Forecasts are too dispersed.

Quantitatively interpreting
• Many approaches
ensemble forecasts
• Example: Democratic voting method – e.g., for
visibility or wind speed at an airport, pick a
relevant value, count the number of ensemble
members above and below the value, and
translate to a probability.
15 m/s
4 forecasts 6 forecasts
Wind speed
Probability of winds exceeding threshold is 60%

Calibration of ensembles
• Calibration is a post-processing step that removes the bias
from the first and possibly the higher moments.
• Calibration is important because it:
– provides greater accuracy in the ensemble mean,
– provides improved estimates of the probabilities of extreme
events,
– represents ensemble spread in terms of quantitative measures
of the uncertainty in the forecast of the ensemble mean.
• A history of observations and ensemble forecasts is needed
to perform the calibration.
• Historical archives of operational forecasts are not ideal for
this. WHY?

Calibration of ensembles
• Calibration is a post-processing step that removes the bias
from the first and possibly the higher moments.
• Calibration is important because it:
– provides greater accuracy in the ensemble mean,
– provides improved estimates of the probabilities of extreme
events,
– represents ensemble spread in terms of quantitative measures
of the uncertainty in the forecast of the ensemble mean.
• A history of observations and ensemble forecasts is needed
to perform the calibration.
• Historical archives of operational forecasts are not ideal for
this. Because the model is continually being updated, and
thus the calibration changes also.
• Ideally, reforecasts with the current version of the model
should be used.

Example skill of calibrated versus
uncalibrated ensembles
• Uncalibrated ensemble – probability of
precipitation occurring over a threshold amount
is calculated based on the number of ensemble
members that produce precip above and below
the threshold (democratic voting method).
skill
no skill
One more day of
skill from calibration

Time lagged ensemble
• Advantage: It is based on deterministic forecasts from
different initial times, that are valid at the same time.
• Not as good as conventional ensemble, but some
benefits have been shown.

Spread of time-lagged ensembles
• Forecasters frequently look at how consistent
forecasts from different cycles.
• If the forecast for a certain time remains the
same for different cycles (different initial
times) – forecast has confidence in the
prediction.
• If the forecast changes for a new cycle, the
forecaster is alerted that there may be
increasing uncertainty.

Short-range ensemble prediction
with high-resolution LAMs
• LBCs may cause excessively small ensemble
dispersion, even when they are perturbed.
• Near-surface processes are sometimes not
very predictable, limiting the usefulness of an
ensemble.
• Methods for generating IC error have been
designed for larger-scale models – it is unclear
how to do this on the mesoscale.

Graphically displaying
ensemble-model products

The ensemble mean
• These fields look like any map of a model
dependent variable, and can be displayed in
the same way

The spread or dispersion
of the ensemble

12 h
Spaghetti plots:
36 h
84 h
5520-m contour
of the 500-mb
height based on
a 17-member
ensemble
forecast by
NCEP

Meteograms
Show one variable
for one point

Probability of exceedance plots
The probability
that near-surface
wind gusts will
exceed 50 m/s at
the 42-hour lead
of a forecast for 26
December 1999,
based on an
ECMWF 50-
member
ensemble

Plots of ensemble variance
19-member
physics ensemble
(solid line) and a
19-member
initial-condition
ensemble
(dashed line)
from MM5, for a
long-lived MCS.

“Stamp maps”

Probability of the dosage exceeding a
threshold – democratic voting method

An “electricity-gram” – an example of a specialized
plot: uncertainty in 10-day forecast of electricity demand
ECMWF ensemble prediction used as input to an energy-demand
model. The middle 50% of predictions falls within the box, and the
whiskers contain all the values

Using probabilistic information from
ensemble predictions
• The cost-loss approach
• Given an uncertain prediction of whether an
event will or will not occur, a decision maker has
the option to either protect against the
occurrence of the wx event, or not to protect.
• Simple decision problem – two actions (protect,
or not) and two outcomes (event occurs, or not)
• Example events – freeze, flood, high winds, heavy
snow

• Decision to protect will have cost C, whether
or not the event occurs.
• A decision to not protect will result in a loss L,
if the event occurs

• Assume that a calibrated ensemble forecast
predicts that the probability of an event occurring
is p.
• The optimal decision will be the one resulting in
the smallest expense.
• If the decision is to protect, expense = C with a
probability of 1.
• If the decision is to not protect, the expense will
be pL.
• Thus, protecting against the risk will result in the
smallest expense when C < pL. Or
C/L < p

• Such an approach to decision making is
straightforward when economic value is
used…it is more problematic when societal or
environmental “values” must be considered.