Artificial Intelligence and Soft Computing: Behavioral ... - Arteimi.info

Fly =false. Further if living status =alive, we check whether Z=A/W ≥ 2.5. If
yes, Fly =true, else Fly is declared false.
A question then naturally arises: in what order should we check the
attribute value of the instances in a decision tree? In answering this question
we need to know: which attribute is more vital? To measure the vitality of the
attributes we require a statistical property called information gain, which will
be defined shortly. The information gain depends on a parameter, called
entropy, which is defined as follows:
Given a collection S of positive and negative instances for a target
concept (decision). The entropy S with respect to this Boolean classification
is:
Entropy (S) ≡ -pos log 2 (pos) – neg log 2 (neg) (13.1)
where ‘pos’ and ‘neg’ denote the proportion of positive and negative instances
in S. While calculating entropy, we define 0log2 (0) =0 .
For illustration, let us consider S that has 3 positive and 6 negative
instances. We, following Mitchell [10], adopt the notation [3+, 6-] to
summarize this sample of data. The entropy of S with respect to the
Boolean classifier Fly is given by
Entropy [3+, 6-]
= -(3 / 9) log 2 (3 / 9) - (6 / 9) log 2 (6 / 9)
= -(1/3) log 2 (1/3) - (2/3) log 2 (2/3)
= -(1/3) log 2 (1/3) - (1/3)log 2 (4/9)
= - (1/3) [ log 2 (1/3) + log 2 (4/9) ]
= - (1/3) [log 2 ( 4 /27 ) ]
= 0.9179.
It is to be noted that when all the instances are either positive or
negative, entropy is zero, as neg or pos =1. Further, when neg = pos = 0.5,
entropy is 1; when neg ≠ pos, entropy must lie in the interval [0,1].
When the classifier has an output range that takes c different values,
then entropy of S with respect to this c-wise classification will be
c
Entropy (S) = ∑i= 1 -Pi log2 (Pi) (13.2)