Rating Models and Validation - Oesterreichische Nationalbank

Chart 35: Creating the Analysis and Validation Samples
— If restrictions apply to the number of cases which banks can collect in the
data collection stage, a higher proportion of bad cases should be collected.
In practice, approximately one fourth to one third of the analysis sample
comprises bad cases. The actual definition of these proportions depends
on the availability of data in rating development. This has the advantage
of maximizing the reliability with which the statistical procedure can identify
the differences between good and bad borrowers, even for small quantities
of data. However, this approach also requires the calibration and
rescaling of calculated default probabilities (cf. section 5.3).
As an alternative or a supplement to splitting the overall sample, the bootstrap
method (resampling) can also be applied. This method provides a way of
using the entire database for development and at the same time ensuring the
reliable validation of scoring functions.
In the bootstrap method, the overall scoring function is developed using the
entire sample without subdividing it. For the purpose of validating this scoring
function, the overall sample is divided several times into pairs of analysis and
validation samples. The allocation of cases to these subsamples is random.
The coefficients of the factors in the scoring function are each calculated
again using the analysis sample in a manner analogous to that used for the overall
scoring function. Measuring the fluctuation margins of the coefficients resulting
from the test scoring functions in comparison to the overall scoring function
makes it possible to check the stability of the scoring function.
The resulting discriminatory power of the test scoring functions is determined
using the validation samples. The mean and fluctuation margin of the
resulting discriminatory power values are likewise taken into account and serve
as indicators of the overall scoring functionÕs discriminatory power for unknown
data, which cannot be determined directly.
In cases where data availability is low, the bootstrap method provides an
alternative to actually dividing the sample. Although this method does not
Rating Models and Validation
Guidelines on Credit Risk Management 73