Rating Models and Validation - Oesterreichische Nationalbank

Chart 62: ROC Curve with Simultaneous Confidence Bands at the 90% Level for the Data Example
Chart 63: Table of Upper and Lower AUC Limits for Selected Confidence Levels in the Data Example
Given a sufficient number of subsamples, this procedure yields estimates of
the average and variance of the AUC value. However, this procedure often only
shows an apparently exact AUC parameter calculated from the same sample, as
especially in homogenous samples the AUC fluctuations in subsamples are
rather low. However, the procedure is useful for small samples where the confidence
bands are very wide due to the small number of cases.
Bayesian Error Rate
As a measure of discriminatory power, the Bayesian error rate is defined as the
minimum error rate occurring in the sample examined (a error plus b error).
In this technique, the minimum is search for among all cutoff values (the score
value beyond which the debtor is classified as bad):
ER ¼ min½
p ðCÞþð1 pÞ ðCÞŠ:
C
The a and b errors for each cutoff value C are weighted with the sampleÕs
default rate p or its complement (1-p). For the example with 10 rating classes
used here, the table below (chart 64) shows the a and b errors for all cutoff
values as well as the corresponding Bayesian error rates for various values of
p. The Bayesian error rate means the following: ÒIn the optimum use of the rating
model, a proportion ER of all cases will still be misclassified.Ó
Rating Models and Validation
Guidelines on Credit Risk Management 111