Low (web) Quality - BALTEX

152
Weighting multi-models in seasonal forecasting – and what can be learnt for
the combination of RCMs
Andreas P. Weigel, Mark A. Liniger and C. Appenzeller
Federal Office of Meteorology and Climatology, MeteoSwiss, Switzerland
Email: andreas.weigel@meteoswiss.ch
1. Introduction
Multi-model combination (MMC) is a pragmatic approach
to estimate the range of uncertainties induced by model
error. MMC is now routinely applied on essentially all time
scales, ranging from the scale of weather forecasts to the
scale of climate-change scenarios, and its success has been
demonstrated in many studies (e.g. Palmer et al. 2004).
Simple multi-models can be easily constructed by pooling
together the available single model predictions with equal
weight. However, given that models may differ in their
quality and prediction skill, it has been suggested to further
optimize the effect of MMC by weighting the participating
models according to their prior performance (e.g. Giorgi and
Mearns 2002). How can such weights be objectively
estimated? How robust are such estimates, and what are the
consequences if “wrong” weights are applied, i.e. if the
weights are not fully consistent with the true model skill? It
is the aim of this study to discuss these questions at the
example of seasonal forecasts as well as synthetic toy model
forecasts, i.e. prediction contexts where a sufficient number
of training and verification data is available. Potential
implications of the results for the construction of weighted
RCM multi-models will be discussed.
2. Model weighting in seasonal forecasting
The weighting method presented in this study has been
described in detail in Weigel et al. (2008). Essentially,
optimum weights are obtained by minimizing a universal
skill metric called ignorance (Roulston and Smith 2002), a
score which is derived from information theory. The
ignorance quantifies the information deficit (measured in
bits) of a user who is in possession of a probabilistic
prediction, but does not yet know the true outcome. This
metric has its fundamental justification in the fact that it is
the aim of any prediction strategy to maximize the available
information about a future event, respectively to minimize
the information deficit. The method has been applied and
tested in cross-validation mode on a set of 40 years of
hindcast data from two models of DEMETER database
(Palmer et al. 2004). The models have been combined (a)
with equal weights and (b) with optimum weights. The
weights have been determined grid-pointwise. The
corresponding skill maps (skill metric: debiased ranked
probabilitiy skill score, RPSSd, Weigel et al. 2007) are
shown in Fig. 1 and reveal that, on average, model
weighting can improve the prediction skill significantly.
3. Weight robustness
The success of weighted MMC depends on the robustness of
the weights obtained. For the example shown above the
average uncertainty of the weight estimates is, despite 40
years of available reforecasts, on the order of 10%. The
uncertainties grow if the number of independent training
data decreases. Model weighting in the context of climate
change scenarios is particular problematic, since the
verification context is strongly limited to a few, rather
dependent samples. To evaluate the potential effects of these
uncertainties more systematically, a stochastic and
Gaussian forecast generator has been applied to generate
large numbers of multi-model forecasts (consisting of two
models) and corresponding verifying observations. The
combination has been carried out (i) with equal weights
(this is the benchmark), (ii) with optimum weights, and
(iii) with perturbed weights.
(a)
(b)
-1 0
Figure 1. Skill maps (RPSSd) of combined seasonal
forecasts (June, July, August; initialization 1 May)
of 2m temperature for the period 1960-2001. Two
models (ECMWF’s System 2 and the UK Met
Office’s GloSea 2) from the DEMETER database
have been combined (a) with equal weights, and (b)
with optimum weights (determined grid-pointwise).
The toy model experiments confirm that the forecast
information content can be substantially improved by
weighted multi-models. However, they also show that
even more information can be lost if the uncertainty in the
weights becomes too large, i.e. if the weights which are
applied are not consistent with the true model skill. This
1