Rating Models and Validation - Oesterreichische Nationalbank

Rating Models and Validation
Chart 64: Table of Bayesian Error Rates for Selected Sample Default Rates in the Data Example
As the table shows, one severe drawback of using the Bayesian error rate to
calculate a rating modelÕs discriminatory power is its heavy dependence on the
default rate in the sample examined. Therefore, direct comparisons of Bayesian
error rate values derived from different samples are not possible. In contrast,
the measures of discriminatory power mentioned thus far (AUC, the Gini Coefficient
and the Pietra Index) are independent of the default probability in the
sample examined.
The Bayesian error rate is linked to an optimum cutoff value which minimizes
the total number of misclassifications (a error plus b error) in the rating
system. However, as the optimum always occurs when no cases are rejected
(a ¼ 100%, b ¼ 0%), it becomes clear that the Bayesian error rate hardly allows
the differentiated selection of an optimum cutoff value for the low default rates
p occurring in the validation of rating models.
Due to the various costs of a and b errors, the cutoff value determined by
the Bayesian error rate is not optimal in business terms, that is, it does not minimize
the overall costs arising from misclassification, which are usually substantially
higher for a errors than for b errors.
For the default rate p ¼ 50%, the Bayesian error rate equals exactly half the
sum of a and b for the point on the a—b error curve which is closest to point
(0, 0) with regard to the total a and b errors (cf. chart 65). In this case, the
following is also true (a and b errors are denoted as a and b): 91
ER ¼ min½ þ Š¼ max½1 1Š ¼ max½F bad
cum
¼ ð1 Pietra IndexÞ
good F cum
Šþ ¼
However, this equivalence of the Bayesian error rate to the Pietra Index only
applies where p ¼ 50%. 92
91 The Bayesian error rate ER is defined at the beginning of the equation for the case where p ¼ 50%. The error values a and b
are related to the frequency distributions of good and bad cases, which are applied in the second to last expression. The last
expression follows from the representation of the Pietra Index as the maximum difference between these frequency distributions.
92 Cf. LEE, Global Performances of Diagnostic Tests.
112 Guidelines on Credit Risk Management