Rating Models and Validation - Oesterreichische Nationalbank

Rating Models and Validation
ing development stages. However, data pooling in the validation and continued
development of rating models is usually less comprehensive, as in this case it is
only necessary to retrieve the rating criteria which are actually used. However,
additional data requirements may arise for any necessary further developments
in the pool-based rating model.
5.1.3 Definition of the Sample
The data gathered in the data collection and cleansing stages represent the overall
sample, which has to be divided into an analysis sample and a validation sample.
The analysis sample supports the actual development of the scoring functions,
while the validation sample serves exclusively as a hold-out sample to test
the scoring functions after development. In general, one can expect sound discriminatory
power from the data records used for development. Testing the
modelÕs applicability to new (i.e. generally unknown) data is thus the basic prerequisite
for the recognition of any classification procedure. In this context, it is
possible to divide the overall sample into the analysis and validation samples in
two different ways:
— Actual division of the database into the analysis and validation samples
— Application of a bootstrap procedure
In cases where sufficient data (especially regarding bad cases) are available to
enable actual division into two sufficiently large subsamples, the first option
should be preferred. This ensures the strict separation of the data records in
the analysis and validation samples. In this way, it is possible to check the quality
of the scoring functions (developed using the analysis sample) using the
unknown data records in the validation sample.
In order to avoid bias due to subjective division, the sample should be split
up by random selection (see chart 35). In this process, however, it is necessary
to ensure that the data are representative in terms of their defined structural
characteristics (see section 5.1.2).
Only those cases which fulfill certain minimum data quality requirements
can be used in the analysis sample. In general, this is already ensured during
the data collection and cleansing stage. In cases where quality varies within a
database, the higher-quality data should be used in the analysis sample. In such
cases, however, the results obtained using the validation sample will be considerably
less reliable.
Borrowers included in the analysis sample must not be used in the validation
sample, even if different cutoff dates are used. The analysis and validation samples
thus have to be disjunct with regard to borrowers.
With regard to weighting good and bad cases in the analysis sample, two different
procedures are conceivable:
— The analysis sample can be created in such a way that the proportion of bad
cases is representative of the rating segment to be analyzed. In this case, calibrating
the scoring function becomes easier (cf. section 5.3). For example,
the result of logistic regression can be used directly as a probability of
default (PD) without further processing or rescaling. This approach is advisable
whenever the number of cases is not subject to restrictions in the data
collection stage, and especially when a sufficient number of bad cases can be
collected.
72 Guidelines on Credit Risk Management