Boosted Regression (Boosting): An introductory tutorial and a Stata ...

More documents

Recommendations

Info

compared to linear regression. Linear regression is a good reference model because manyscientists initially fit a linear model. I simulated data from the following modely = 30( x − 0.5) + 2x + x + ε2 −0.51 2 3where ε~uniform(0,1) and 0 ≤ x i≤ 1 for i ∈ 1, 2, 3 . To keep things relatively simple, themodel has been chosen to be additive without interactions. It is quadratic in x 1 , nonlinearin x 2 , linear with a small slope in x 3 . The nonlinear contribution of x 2 is stronger than thelinear contribution of x 3 even though their slopes are similar. A fourth variable, x 4 , isunrelated to the response but is used in the analysis in an attempt to confuse the boostingalgorithm. Scatter plots of y vs x1 through x4 is shown in Figure 3.y0 20 40 60 80y0 20 40 60 800 .2 .4 .6 .8 1x10 .2 .4 .6 .8 1x2y0 20 40 60 80y0 20 40 60 800 .2 .4 .6 .8 1x30 .2 .4 .6 .8 1x4Figure 3: Scatter plots of y versus x1 through x4I chose the following parameter values shrink=0.01, bag=0.5 and maxiter=4000.My rule of thumb in choosing the maximal number of iterations is that the shrinkagefactor times the maximal number of iterations should be roughly between 10 and 100. In16
my experience the cross-validated R 2 as a function of the number of iterations isunimodal, i.e. there is only one maximum. If bestiter is too close to maxiter then thenumber of iterations that maximizes the likelihood may be greater than maxiter. It isrecommended to rerun the command with a larger value for maxiter. The command I amgiving is:boost y x1-x4, distribution(normal) train(0.5) maxiter(4000) seed(1)bag(0.5) interaction(`inter') shrink(0.01)where the commands only differ by inter ranging from 1 through 5. One of these boostcommands runs in 8.8 seconds on my laptop. Fixing the seed is only relevant forbagging. Figure 4 shows a plot of the test R 2 versus the number of interactions. The testR 2 is roughly the same regardless of the number of interactions (note the scale of theplot). The fact that the test R 2 is high even for the main effect model (interaction=1) doesnot surprise because our model did not contain any interactions. The actual number ofiterations that maximizes the likelihood, bestiter, varies. Here the number of iterations are(number of interactions in parenthesis): 3769 (1), 2171 (2), 2401 (3), 1659 (4), 1156 (5).17
Page 3 and 4: e “the best algorithm”. Friedma
Page 5 and 6: interaction=2, then each tree has 3
Page 7 and 8: Initialize weights to be equal w i
Page 9 and 10: 4.3 Friedman’s gradient boosting
Page 11 and 12: The second parameter is the number
Page 13 and 14: 4 train test train5 train testFigur
Page 15: In what follows I give some suggest
Page 19 and 20: global trainn=e(trainn) /* using e(
Page 21 and 22: The influence shows one can learn h
Page 23 and 24: to one another. The response y is a
Page 25 and 26: has a lower value. Both values are
Page 27 and 28: Percentage Influence0 5 10 15 20123
Page 29 and 30: distribution, trainfraction=0.5, ba
Page 31 and 32: Shrinking does not affect runtime,
Page 33: RAND Corporation for his support an

Boosted Regression (Boosting): An introductory tutorial and a Stata ...

Create successful ePaper yourself

Delete template?

Save as template?