2DkcTXceO

502 Sparsity and convexity Test statistic0 2 4 6 8 10 12 0 2 4 6 8 10 12Exp(1)(a) Forward stepwise(b) LassoFIGURE 42.3Asimpleexamplewithn = 100 observations and p = 10 orthogonal predictors.All true regression coefficients are zero, β ∗ = 0. On the left is a quantilequantileplot, constructed over 1000 simulations, of the standard chi-squaredstatistic R 1 ,measuringthedropinresidualsumofsquaresforthefirstpredictorto enter in forward stepwise regression, versus the χ 2 1 distribution. Thedashed vertical line marks the 95% quantile of the χ 2 (1)distribution. The rightpanel shows a quantile-quantile plot of the covariance test statistic T 1 for thefirst predictor to enter in the lasso path, versus its asymptotic distributionE(1). The covariance test explicitly accounts for the adaptive nature of lassomodeling, whereas the usual chi-squared test is not appropriate for adaptivelyselected models, e.g., those produced by forward stepwise regression.Wasserman and Roeder, 2009), but these may suffer from lower power due tothe decrease in sample size.But the lasso can help us! Specifically, we need the LAR (least angle regression)method for constructing the lasso path of solutions, as the regularizationparameter λ is varied. I won’t give the details of this construction here, but wejust need to know that there are a special set of decreasing knots λ 1 > ···>λ kat which the active set of solutions (the non-zero parameter estimates) change.When λ>λ 1 , the solutions are all zero. At the point λ = λ 1 , the variablemost correlated with y enters the model. At each successive value λ j , a variableenters or leaves the model, until we reach λ k where we obtain the fullleast squares solution (or one such solution, if p>N).We consider a test statistic analogous to R j for the lasso. Let y be thevector of outcome values and X be the design matrix. Assume for simplicity