Subsampling estimates of the Lasso distribution.

More documents

Recommendations

Info

Chapter 1 Introduction In this thesis we consider the linear regression model Y i = x ′ iβ + ε i , i = 1, . . . , n and focus on the Least Absolute Shrinkage and Selection Operator, or Lasso, as estimation method for the parameter β. The Lasso, defined as arg min φ∈R p ∑ n p∑ (x ′ iφ − Y i ) 2 + λ |φ| j , i=1 j=1 was introduced by Tibshirani (1996) and has become very popular over the last years. There are two main reasons for this gain in popularity. The first being, in a context of high dimensional data analysis, where the dimension of the parameter is very large, by introducing an l 1 -norm penalty in addition to the squared loss, the Lasso can induce sparsity on the estimated model while maintaining its prediction ability. That is, many coefficients will be set to zero, which is a desirable property if one believes that only few parameters are relevant. The second reason is that the solution can be computed efficiently by convex optimization. Efron, Hastie, Johnstone, and Tibshirani (2004) introduced the LARS algorithm which computes the entire solution path, that is, estimates corresponding to all values λ > 0 with the cost of a single least square regression. For the sole purpose of prediction, the Lasso proves to be very efficient. Several authors showed under various assumptions that it enjoys a so-called oracle property, see for instance Zou (2006), Bunea, Tsybakov, and Wegkamp (2007) or Van De Geer and Bühlmann (2009). Roughly speaking, an oracle result states that prediction based on the Lasso solution will be as accurate as if the true model was known. Variable selection properties of the Lasso are by now also quite well understood and it was shown that consistency in this mode can be characterized by a so called irrerepresentable condition, see for instance Zhao and Yu (2006) or Meinshausen and Bühlmann (2006). Nevertheless, the Lasso theory still needs to bridge a major gap with traditional statistical inference. Indeed, it is to this date difficult to assign a measure of incertainty to its estimates. One consequence is that the data analyst who goes beyond the goal of prediction and is interrested in selecting the exact generating model will typically face many noise variables for which no statistical tests are available. Also, there are no confidence intervals for the estimated coeffficents. Assigning confidence intervals to an estimator or testing for a null hypothesis typically involves understanding its distributional properties. In the situation where an analytic 1
2 Introduction solution to this problem doesn’t seem to be sight, as is the case when the estimator results from numerical optimization, one usually resorts to resampling techniques, of which Efron’s bootstrap is a prime example. However, proving the consistency (or the inconsistency) of the bootstrap often proves to be a hard task. Meanwhile, subsampling, or resampling without replacement, will yield consistency under very weak assumptions. Roughly speaking, the estimator, or an associated root needs to have a limiting distribution only. In this thesis we investigate this direction. Chapter 2 reviews the asymptotic theory for minimizers of convex processes. In Chapter 3, the results obtained are applied to the Lasso estimator in a low dimensional setting in the vein of Knight and Fu (2000) and continuity propperties of the limiting distribution are outlined. In Chapter 4, following Huang et al. (2008) the adaptive Lasso is studied in a sparse high dimensional setting and a partial oracle property (in distribution) is presented. In Chapter 5, the theory of subsampling is introduced and consistency results are presented, following the expositions of Politis, Romano, and Wolf (1999) and Romano and Shaikh (2010). Finally, a simulation study is conducted in Chapter 6 to acess the performance of subsampling applied to the aforementioned problems.
Page 1: ✄✄ ✄ ✄ ✄ ✄✄ ✄ ✄
Page 4 and 5: iv Abstract
Page 6 and 7: vi CONTENTS Contents Notation viii
Page 8 and 9: viii Notation List of Tables 6.1 Mo
Page 12 and 13: Chapter 2 Minimizers of convex proc
Page 14 and 15: 2.1 Convergence in probability 5 fo
Page 16 and 17: 2.2 Convergence in distribution 7 a
Page 18 and 19: 2.2 Convergence in distribution 9 L
Page 20 and 21: 2.2 Convergence in distribution 11
Page 22 and 23: 2.2 Convergence in distribution 13
Page 24 and 25: 2.3 A continous mapping theorem for
Page 30 and 31: Chapter 3 Application to the Lasso
Page 32 and 33: 3.2 Limit in distribution 23 almost
Page 34 and 35: 3.2 Limit in distribution 25 We als
Page 36 and 37: 3.2 Limit in distribution 27 3.2.2
Page 38 and 39: 3.2 Limit in distribution 29 Figure
Page 40 and 41: Chapter 4 The adaptive Lasso in a h
Page 42 and 43: 33 B.3 (Adaptive irrepresentable co
Page 44 and 45: 35 exists a constant K d depending
Page 46 and 47: 4.1 Variable selection consistency
Page 48 and 49: 4.2 Partial asymptotic normality 39
Page 50 and 51: 4.3 Marginal regressors as initial
Page 52 and 53: Chapter 5 Subsampling In his semina
Page 54 and 55: 5.1 Pointwise consistency for distr
Page 56 and 57: 5.2 Uniform consistency for quantil
Page 58 and 59: 5.2 Uniform consistency for quantil
Page 60 and 61:
5.2 Uniform consistency for quantil
Page 62 and 63:
5.2 Uniform consistency for quantil
Page 64 and 65:
Chapter 6 Numerical results Motivat
Page 66 and 67:
6.1 Low dimensinal setting 57 5. De
Page 68 and 69:
6.1 Low dimensinal setting 59 Figur
Page 70 and 71:
6.2 High dimensional setting 61 3.
Page 72 and 73:
6.2 High dimensional setting 63 (d)
Page 74 and 75:
6.2 High dimensional setting 65 Mod
Page 76 and 77:
6.2 High dimensional setting 67 Mod
Page 78 and 79:
6.2 High dimensional setting 69 Fig
Page 80 and 81:
Chapter 7 Concluding remarks We rev
Page 82 and 83:
Bibliography Andersen, P. and R. Gi
Page 84 and 85:
Appendix A R Codes A.1 Simulation c
Page 86 and 87:
A.1 Simulation code for the Lasso i
Page 88 and 89:
A.2 Simulation code for the adaptiv
Page 90 and 91:
A.2 Simulation code for the adaptiv
show all

Subsampling estimates of the Lasso distribution.

Create successful ePaper yourself

Delete template?

Save as template?