Machine Learning - DISCo

A final dimension is the particular definition of rule PERFORMANCE used to
guide the search in LEARN-ONE-RULE. Various evaluation functions have been used.
Some common evaluation functions include:
0 Relative frequency. Let n denote the number of examples the rule matches
and let nc denote the number of these that it classifies correctly. The relative
frequency estimate of rule performance is
Relative frequency is used to evaluate rules in the AQ program.
0 m-estimate of accuracy. This accuracy estimate is biased toward the default
accuracy expected of the rule. It is often preferred when data is scarce and
the rule must be evaluated based on few examples. As above, let n and nc
denote the number of examples matched and correctly predicted by the rule.
Let p be the prior probability that a randomly drawn example from the entire
data set will have the classification assigned by the rule (e.g., if 12 out of
100 examples have the value predicted by the rule, then p = .12). Finally,
let m be the weight, or equivalent number of examples for weighting this
prior p. The m-estimate of rule accuracy is
Note if m is set to zero, then the m-estimate becomes the above relative frequency
estimate. As m is increased, a larger number of examples is needed
to override the prior assumed accuracy p. The m-estimate measure is advocated
by Cestnik and Bratko (1991) and has been used in some versions of
the CN2 algorithm. It is also used in the naive Bayes classifier discussed in
Section 6.9.1.
0 Entropy. This is the measure used by the PERFORMANCE subroutine in the
algorithm of Table 10.2. Let S be the set of examples that match the rule
preconditions. Entropy measures the uniformity of the target function values
for this set of examples. We take the negative of the entropy so that better
rules will have higher scores.
-Entropy (S) =
C
pi logl pi
where c is the number of distinct values the target function may take on,
and where pi is the proportion of examples from S for which the target
function takes on the ith value. This entropy measure, combined with a test
for statistical significance, is used in the CN2 algorithm of Clark and Niblett
(1989). It is also the basis for the information gain measure used by many
decision tree learning algorithms.