Boosted Regression (Boosting): An introductory tutorial and a Stata ...

More documents

Recommendations

Info

of his/her race/ethnicity. The leaf with the largest values of the response, 0.8, correspondsto observations between 49 and 57 years of age, who are neither Hispanic nor Black. Asthe splits on age show, the tree can split on the same variable several times. If the 6 splitsin the tree in Figure 1 had split on six different variables the tree would represent a 6-level interaction because six variables would have to be considered jointly in order toobtain the predicted value.| age
4.3 Friedman’s gradient boosting algorithmEarly researchers had some difficulty explaining the success of the AdaBoostalgorithm. The computational learning theory roots of the AdaBoost puzzled statisticianswho have traditionally worked from likelihood-based approaches to classification and,more generally, to regression. Then Friedman et al. (2000) were able to reinterpret thisalgorithm in a likelihood framework, enabling the authors to form a boosted logisticregression algorithm, a formulation more familiar to the statistical community. Once theconnection to the likelihood existed, boosting could be extended to generalized linearmodels and further still to practically any loss criterion (Friedman 2001, Ridgeway,1999). This meant that a boosting algorithm could be developed for all the errordistributions in common practice: Gaussian, logistic, Poisson, Cox models, etc. With thepublication of Hastie et al. (2001), a book on statistical learning, modern regressionmethods like boosting caught on in the statistics community.The interpretation of boosting in terms of regression for a continuous, normallydistributed response variable is as follows: The average y-value is used as a first guess forpredicting all observations. This is analogous to fitting a linear regression model thatconsists of the intercept only. The residuals from the model are computed. A regressiontree is fit to the residuals. For each terminal node, the average y-value of the residualsthat the node contains is computed. The regression tree is used to predict the residuals.(In the first step, this means that a regression tree is fit to the difference between theobservation and the average y-value. The tree then predicts those differences.) Theboosting regression model - consisting of the sum of all previous regression trees - isupdated to reflect the current regression tree. The residuals are updated to reflect thechanges in the boosting regression model; a tree is fit to the new residuals, and so forth.This algorithm is summarized in more detail in Algorithm 2.1) Initialization: Set initial guess to y2) For all regressions trees m=1 to M:2a) Compute the residuals based on the current model9
Page 3 and 4: e “the best algorithm”. Friedma
Page 5 and 6: interaction=2, then each tree has 3
Page 7: Initialize weights to be equal w i
Page 11 and 12: The second parameter is the number
Page 13 and 14: 4 train test train5 train testFigur
Page 15 and 16: In what follows I give some suggest
Page 17 and 18: my experience the cross-validated R
Page 19 and 20: global trainn=e(trainn) /* using e(
Page 21 and 22: The influence shows one can learn h
Page 23 and 24: to one another. The response y is a
Page 25 and 26: has a lower value. Both values are
Page 27 and 28: Percentage Influence0 5 10 15 20123
Page 29 and 30: distribution, trainfraction=0.5, ba
Page 31 and 32: Shrinking does not affect runtime,
Page 33: RAND Corporation for his support an

Boosted Regression (Boosting): An introductory tutorial and a Stata ...

Create successful ePaper yourself

Delete template?

Save as template?