Data Mining: Practical Machine Learning Tools and ... - LIDeCC

7.5 COMBINING MULTIPLE MODELS 327of iterations needed to arrive at a good additive model. Reducing the multipliereffectively damps down the learning process, increasing the chance of stoppingat just the right moment—but also increasing run time.Additive logistic regressionAdditive regression can also be applied to classification just as linear regressioncan. But we know from Section 4.6 that logistic regression outperforms linearregression for classification. It turns out that a similar adaptation can be madeto additive models by modifying the forward stagewise modeling method toperform additive logistic regression. Use the logit transform to translate theprobability estimation problem into a regression problem, as we did in Section4.6, and solve the regression task using an ensemble of models—for example,regression trees—just as for additive regression. At each stage, add the modelthat maximizes the probability of the data given the ensemble classifier.Suppose f j is the jth regression model in the ensemble and f j (a) is its predictionfor instance a. Assuming a two-class problem, use the additive model Sf j (a)to obtain a probability estimate for the first class:1p( 1a) =- f1 ej ( a)+ÂThis closely resembles the expression used in Section 4.6 (page 121), except thathere it is abbreviated by using vector notation for the instance a and the originalweighted sum of attribute values is replaced by a sum of arbitrarily complexregression models f.Figure 7.9 shows the two-class version of the LogitBoost algorithm, which performsadditive logistic regression and generates the individual models f j . Here,y i is 1 for an instance in the first class and 0 for an instance in the second. Ineach iteration this algorithm fits a regression model f j to a weighted version ofmodel generationFor j = 1 to t iterations:For each instance a[i]:Set the target value for the regression toz[i] = (y[i] – p(1 | a[i])) / [p(1 | a[i]) (1 – p(1 | a[i])]Set the weight of instance a[i] to p(1 | a[i]) (1 – p(1 | a[i])Fit a regression model f[j] to the data with class values z[i] and weights w[i].classificationPredict first class if p(1 | a) > 0.5, otherwise predict second class.Figure 7.9 Algorithm for additive logistic regression.