The tenth IMSC, Beijing, China, 2007 - International Meetings on ...

<strong>The</strong> ranked probability score and its corresponding skill score are widely used metrics
of the quality of forecasts of weather and climate that are expressed as probabilities of three or
more ordinal categories.<strong>The</strong> ranked probability score considers the ordering of the categories
so that for two forecasts with equal probabilities assigned to the observed category, a better
score will be achieved by the forecast that gives higher probabilities to nearby categories. It is
argued here that when considering the mathematical quality of forecasts as opposed to their
social or economic value, the credit given to probabilities assigned to categories close to the
observed category ('distance') is an undesirable feature of the ranked probability score
because we have no knowledge of what the actual probabilities are for these categories for
any specific case. <strong>The</strong> reliability of the probabilities assigned to the categories that did not
verify can be tested only over a set of forecasts by considering the observed relative
frequencies of verifications within these categories conditional upon the forecast probabilities
assigned.It is thus inappropriate to give credit for 'near-misses' in the context of probabilistic
forecasts. A desirable feature of a scoring metric is that of 'locality': the forecast is scored only
on the basis of the probability assigned to what actually occurred. <strong>The</strong> ranked probability score
does not have this feature. Alternative scores include the quadratic and the logarithmic scores,
but the latter is to be preferred since the former does not generalise to continuous forecasts
(infinite number of categories).
Verification of Seasonal to Secular Climate Forecasts
Speaker: David Stephenson
Professor David Stephenson
University of Exeter, UK
Verification is a key aspect in developing good scientific forecast systems. Without
reliable procedures for forecast evaluation, we are unable to judge whether a forecasting
system is better than just chance or whether a new forecasting system is really an
improvement compared to previous schemes.
Climate forecasts pose particular problems for forecast verification. Small sample sizes and
non-stationarity due to long-term trends can make verification scores extremely unreliable and
uninformative and put serious limits on how well we can assess such types of forecast.
This talk will discuss various issues that affect the verification of climate forecasts and will
present simple statistical models for understanding verification of climate forecasts on
lead-times from seasons to centuries
104