Rating Models and Validation - Oesterreichische Nationalbank

Chart 70 shows the reliability diagram for the example used here. In the
illustration, a double logarithmic representation was selected because the
default probabilities are very close together in the good rating classes in particular.
Note that the point for rating class 1 is missing in this graph. This is
because no defaults were observed in that class.
The points of a well-calibrated system will fall close to the diagonal in the
reliability diagram. In an ideal system, all of the points would lie directly on the
diagonal. The ÒcalibrationÓ term of the Brier Score represents the average
(weighted with the numbers of cases in each rating class) squared deviation
of points on the calibration curve from the diagonal. This value should be as
low as possible.
The resolution of a rating model is indicated by the average (weighted with
the numbers of cases in the individual rating classes) squared deviation of points
in the reliability diagram from the broken line, which represents the default rate
observed in the sample. This value should be as high as possible, which means
that the calibration curve should be as steep as possible. However, the steepness
of the calibration curve is primarily determined by the rating modelÕs discriminatory
power and is independent of the accuracy of default rate estimates.
An ideal trivial rating system with only one rating class would be represented
in the reliability diagram as an isolated point located at the intersection
of the diagonal and the default probability of the sample.
Like discriminatory power measures, one-dimensional indicators for calibration
and resolution can also be defined as standardized measures of the area
between the calibration curve and the diagonal or the sample default rate. 97
Checking the Significance of Deviations in the Default Rate
In light of the fact that realized default rates are subject to statistical fluctuations,
it is necessary to develop indicators to show how well the rating model estimates
the parameter PD. In general, two approaches can be taken:
— Assumption of uncorrelated default events
— Consideration of default correlation
Empirical studies show that default events are generally not uncorrelated.
Typical values of default correlations range between 0.5% and 3%. Default correlations
which are not equal to zero have the effect of strengthening fluctuations
in default probabilities. The tolerance ranges for the deviation of realized
default rates from estimated values may therefore be substantially larger when
default correlations are taken into account. In order to ensure conservative estimates,
therefore, it is necessary to review the calibration under the initial
assumption of uncorrelated default events.
The statistical test used here checks the null hypothesis ÒThe forecast default
probability in a rating class is correctÓ against the alternative hypothesis ÒThe
forecast default probability is incorrectÓ using the data available for back-testing.
This test can be one-sided (checking only for significant overruns of the forecast
default rate) or two-sided (checking for significant overruns and underruns of
the forecast default probability). From a management standpoint, both significant
underestimates and overestimates of risk are relevant. A one-sided test can
97 See also HASTIE/TIBSHIRANI/FRIEDMAN, Elements of statistical learning.
Rating Models and Validation
Guidelines on Credit Risk Management 119