Chapter 7 - Ensemble methods.pdf

rap.ucar.edu

Chapter 7 - Ensemble methods.pdf

Chapter 7

Ensemble methods


Each of the following introduces some

error into the modeling process

• Initial conditions

• Lateral and upper boundary conditions

• Specification of the lower-boundary conditions

• Numerical approximations to the equations

• Parameterizations of physical processes

The sum of the errors can be very case/date

dependent


The essence of ensemble prediction

• Perform parallel forecasts or simulations using

different arbitrary choices for the previous

imperfect data or methods.

• This “samples the uncertainty space”

associated with the modeling process, to see

how this uncertainty projects onto the

forecast.


Example ensemble forecast – track

prediction for hurricane Katrina, from the ECMWF

ensemble prediction system. The heavy line is from the

deterministic model forecast


Deterministic vs. ensemble prediction

• Without ensemble prediction (deterministic

forecast) – there is one forecast state of the

atmosphere, and the user has no information

about the confidence we should place in it.

– Compared to yesterday’s forecast

– One part of the model grid compared to others

• With ensemble prediction (probabilistic forecasts)

– we get a better forecast of the most probable

conditions, and we get uncertainty information

(“spread” of ensemble members).


Benefit of ensemble method

• 1) The mean of the ensemble of forecasts is generally

more accurate than the forecast from an individual

member, for a large number of forecasts.

• 2) The spread or variance or dispersion of the

ensemble members can be related to the uncertainty

in the ensemble mean.

• 3) The probability distribution (or density) function

(PDF) can provide information about extreme events.

• 4) Quantitative probabilistic products can be moreeffectively

employed in decision-support systems


Historical context

• Since the 1960’s it has been shown that

combining human forecasts from different

forecasters produces a group-mean forecast

that is superior – “consensus forecasting”.

• Forecasting paradigm

– When models agree…good confidence in forecast

– When models disagree…low confidence


Compromises with

ensemble prediction

• Running many parallel models is computationally

expensive.

• Must use coarser horizontal grid increment for

the ensemble, than you would use for a single

deterministic forecast.

• If you increased the grid increment by a factor of

4, how many ensemble members could you run

with the same hardware?


Compromises with

ensemble prediction

• Running many parallel models is computationally

expensive.

• Must use coarser horizontal grid increment for

the ensemble, than you would use for a single

deterministic forecast.

• If you increased the grid increment by a factor of

4, how many ensemble members could you run

with the same hardware?

64


The spread among ensemble

members, and uncertainty

Cyclogenesis?

No Cyclogenesis?

Eight member ensemble with slightly different initial conditions.

• The “x” shows the ensemble mean at each time.

• Circles show the trajectory of individual ensemble members.

• Solid line shows the “trajectory” of the initial mean of the ensemble.


High

predictability

Example of

different spreads

of ensemble

members

London 2-m AGL

temperature forecasts

from ECMWF, initialized

on different dates.

Low

predictability

Different initial

conditions and model

configurations


Another way of looking at day-to-day variability in predictability

that can be captured by ensemble methods: Temporal variability

of single model predictive skill - 500-mb heights in NH


Defining the ensemble members

• Initial-condition uncertainty – how should it

be done???

• Physical-process parameterization uncertainty

– again, how?

• Errors in the dynamical core – i.e., numerical

algorithms – how?

• LBC uncertainty (If a ALM) – how?

• Surface BC uncertainty


Some comments

• With sequential data-assimilation systems, where the

model forecast is used as a first-guess in the next cycle,

model error and initial-condition error are closely

related.

• Multi-model ensembles - using completely different

models to construct the ensemble - make sense if the

models are already being run for the same geographic

area.

• Super-ensembles involve running ensembles for

individual models, and then combining the multiple

sets of ensemble members into a single ensemble.


Verification of ensemble predictions

• The ensemble mean – can be verified the same

way as a deterministic forecast.

• The uncertainty information (e.g., 2 nd moment)

– Reliability diagrams


Reliability diagrams – reliability is an

important attribute of ensemble forecasts of

dichotomous events (ones that occur or do not

occur at a grid point or over an area)

Using a large number of archived ensemble forecasts, select the forecasts

where the forecasted probability of occurrence of an event was p f . Of

those events, what was the observed frequency of occurrence p o ? Do this

for all the different thresholds in the forecast probabilities, and plot as

above.


An example of a real reliability diagram

Single-model IC ensemble

Multi-model super-ensemble


Verification of ensemble predictions

• The ensemble mean – can be verified the same

way as a deterministic forecast.

• The uncertainty information (e.g., 2 nd moment)

– Reliability diagrams

– Rank histograms


Rank histograms

• Also called verification rank histograms and

Talagrand diagrams.

• Define the bias of ensemble predictions

• For a specific variable and location of an

observation, take the ensemble forecast for

that variable at that location and rank-order

the forecasts from each of the members.

• Then define the n+1 intervals that are

bounded by the n ordered forecast variables.


Example with 4 ensemble members

and 5 intervals, for variable P

For this location and

time t forecast , the

observed P (X obs ) is

lower than any of

the forecast Ps and

the observation is

thus in I 1 .


• Follow the same process for all other pairs of

observations and forecasts at this time.

• Calculate the total number of observations in

each of the five intervals, or ranks, and plot a

histogram of the frequency.

• This provides a graphical view of how the

ensemble of forecasts relates to the observations.

• A non-uniformity in the distribution reveals

systematic errors in the ensemble.


• Over-forecasting bias – Observations tend to fall in the

lower intervals of the ensemble. Forecast values are

generally too high.

• Under-forecasting bias – Observations tend to fall in

the upper intervals of the ensemble. Forecast values

are generally too low.

• Rank uniformity – Ideal situation

• Under-dispersion – Observations tend to fall in the

upper and lower intervals. Forecasts are too similar to

each other…need more spread.

• Over-dispersion – Observations tend to fall in the

center intervals. Forecasts are too dispersed.


Quantitatively interpreting

• Many approaches

ensemble forecasts

• Example: Democratic voting method – e.g., for

visibility or wind speed at an airport, pick a

relevant value, count the number of ensemble

members above and below the value, and

translate to a probability.

15 m/s

4 forecasts 6 forecasts

Wind speed

Probability of winds exceeding threshold is 60%


Calibration of ensembles

• Calibration is a post-processing step that removes the bias

from the first and possibly the higher moments.

• Calibration is important because it:

– provides greater accuracy in the ensemble mean,

– provides improved estimates of the probabilities of extreme

events,

– represents ensemble spread in terms of quantitative measures

of the uncertainty in the forecast of the ensemble mean.

• A history of observations and ensemble forecasts is needed

to perform the calibration.

• Historical archives of operational forecasts are not ideal for

this. WHY?


Calibration of ensembles

• Calibration is a post-processing step that removes the bias

from the first and possibly the higher moments.

• Calibration is important because it:

– provides greater accuracy in the ensemble mean,

– provides improved estimates of the probabilities of extreme

events,

– represents ensemble spread in terms of quantitative measures

of the uncertainty in the forecast of the ensemble mean.

• A history of observations and ensemble forecasts is needed

to perform the calibration.

• Historical archives of operational forecasts are not ideal for

this. Because the model is continually being updated, and

thus the calibration changes also.

• Ideally, reforecasts with the current version of the model

should be used.


Example skill of calibrated versus

uncalibrated ensembles

• Uncalibrated ensemble – probability of

precipitation occurring over a threshold amount

is calculated based on the number of ensemble

members that produce precip above and below

the threshold (democratic voting method).

skill

no skill

One more day of

skill from calibration


Time lagged ensemble

• Advantage: It is based on deterministic forecasts from

different initial times, that are valid at the same time.

• Not as good as conventional ensemble, but some

benefits have been shown.


Spread of time-lagged ensembles

• Forecasters frequently look at how consistent

forecasts from different cycles.

• If the forecast for a certain time remains the

same for different cycles (different initial

times) – forecast has confidence in the

prediction.

• If the forecast changes for a new cycle, the

forecaster is alerted that there may be

increasing uncertainty.


Short-range ensemble prediction

with high-resolution LAMs

• LBCs may cause excessively small ensemble

dispersion, even when they are perturbed.

• Near-surface processes are sometimes not

very predictable, limiting the usefulness of an

ensemble.

• Methods for generating IC error have been

designed for larger-scale models – it is unclear

how to do this on the mesoscale.


Graphically displaying

ensemble-model products


The ensemble mean

• These fields look like any map of a model

dependent variable, and can be displayed in

the same way


The spread or dispersion

of the ensemble


12 h

Spaghetti plots:

36 h

84 h

5520-m contour

of the 500-mb

height based on

a 17-member

ensemble

forecast by

NCEP


Meteograms

Show one variable

for one point


Probability of exceedance plots

The probability

that near-surface

wind gusts will

exceed 50 m/s at

the 42-hour lead

of a forecast for 26

December 1999,

based on an

ECMWF 50-

member

ensemble


Plots of ensemble variance

19-member

physics ensemble

(solid line) and a

19-member

initial-condition

ensemble

(dashed line)

from MM5, for a

long-lived MCS.


“Stamp maps”


Probability of the dosage exceeding a

threshold – democratic voting method


An “electricity-gram” – an example of a specialized

plot: uncertainty in 10-day forecast of electricity demand

ECMWF ensemble prediction used as input to an energy-demand

model. The middle 50% of predictions falls within the box, and the

whiskers contain all the values


Using probabilistic information from

ensemble predictions

• The cost-loss approach

• Given an uncertain prediction of whether an

event will or will not occur, a decision maker has

the option to either protect against the

occurrence of the wx event, or not to protect.

• Simple decision problem – two actions (protect,

or not) and two outcomes (event occurs, or not)

• Example events – freeze, flood, high winds, heavy

snow


• Decision to protect will have cost C, whether

or not the event occurs.

• A decision to not protect will result in a loss L,

if the event occurs


• Assume that a calibrated ensemble forecast

predicts that the probability of an event occurring

is p.

• The optimal decision will be the one resulting in

the smallest expense.

• If the decision is to protect, expense = C with a

probability of 1.

• If the decision is to not protect, the expense will

be pL.

• Thus, protecting against the risk will result in the

smallest expense when C < pL. Or

C/L < p


• Such an approach to decision making is

straightforward when economic value is

used…it is more problematic when societal or

environmental “values” must be considered.

More magazines by this user
Similar magazines