Rating Models and Validation - Oesterreichische Nationalbank

Rating Models and Validation
In the analytical estimation of confidence levels, confidence bands are placed
around the CAP or ROC curve. These bands indicate the area of the diagram in
which the overall curve is located at a predefined probability (i.e. the confidence
level). As the Gini Coefficient and AUC area measures are summary properties
of the overall curve, simultaneous confidence bands are preferable to pointbased
confidence bands.
We will now demonstrate the process of constructing confidence bands
around the ROC curve. For each point in the ROC curve, we use Kolmogorov
distribution to define the upper and lower limits of the x and y values for the
desired confidence level (1-q), thus creating a rectangle around each point. The
probability that the overall ROC curve will be located within these rectangles is
(1-q) 2 . Linking the outer corners of all confidence rectangles forms the ROC
curve envelope for the confidence level (1-q) 2 . In turn, the upper and lower
limits of the parameter AUC can be calculated on the basis of this envelope.
These values form the confidence interval for the AUC value.
The confidence rectangles are constructed by adding the value
to the x values and the value
Dq pffiffiffiffiffiffiffi
Nþ Dq pffiffiffiffiffiffiffi
N
to the y values of the points on the ROC curve. N þ and N refer to the number
of good and bad cases in the sample examined. The values for Dq can be found in
the known Kolmogorov distribution table.
Chart 61: Kolmogorov Distribution Table for Selected Confidence Levels
Chart 62 shows simultaneous confidence bands around the ROC curve at
the confidence level (1-q) 2 ¼ 90% for the example used here. The confidence
interval for the value AUC ¼ 82.8% (calculated using the sample) is between
69.9% and 91.5%.
The table below (chart 63) shows the confidence intervals of the parameter
AUC calculated analytically using simultaneous confidence bands for various
confidence levels.
Heuristic estimation of confidence intervals for the parameter AUC is based
on resampling methods. In this process, a large number of subsamples are drawn
from the existing sample. These subsamples can be drawn without replacement
(each case in the sample occurs exactly once or not at all in a subsample) or with
replacement (each case from the sample can occur multiple times in a subsample).
The resulting subsamples should each contain the same number of
cases in order to ensure sound comparability. The ROC curve is drawn and
the parameter AUC is calculated for each of the subsamples.
110 Guidelines on Credit Risk Management