Subsampling estimates of the Lasso distribution.

✄✄ ✄ ✄
✄
✄✄ ✄ ✄ ✄
Swiss Federal Institute of Technology Zurich
Seminar for
Statistics
Department of Mathematics
Master Thesis Summer 2011
Emmanuel Payebto Zoua
Subsampling estimates of the Lasso distribution.
Submission Date: September 09th 2011
Adviser:
Prof. Dr. Peter Bühlmann

Acknowledgement
First and foremost, I want to thank my supervisor Prof. Dr. Peter Bühlmann for the
excellent guidance and for introducing me with this fascinating topic. The interest he
demonstrated for my work has been a great source of motivation.
I want to express my gratitude to Prof. Dr. Sara Van de Geer of the Seminar for Statistics
and to Dr. Michel Baes of the Insitute for Operational Research for taking the time to
answer my questions. Also, I thank David Lamparter for many interesting discussions on
the Lasso.
Finally, writing this thesis would not have been possible without the friendly atmosphere
at the Seminar for Statistics. I would like to thank all staff members and the assistants for
helping me with mathematical or editing issues. To Sarah Gerster and Jürg Schelldorfer,
a huge thanks for the hospitality in the office.
iii

iv
Abstract

Abstract
We investigate possibilities offered by subsampling to etimate the distribution of the Lasso
estimator and construct confidence intervals/hypothesis tests. Despite being inferior to
the bootstrap in terms of higher-order accuracy in situations where the later is consistent,
subsampling offers the advantage to work under very weak assumptions. Thus, building
upon Knight and Fu (2000), we first study the asymptotics of the Lasso estimator in a
low dimensional setting and prove that under an orthogonal design assumption, the finite
sample component distributions converge to a limit in a mode allowing for consistency of
subsampling confidence intervals. We give hints that this result holds in greater generality.
In a high dimensional setting, we study the adaptive Lasso under assumption of partial
orthogonality introduced by Huang, Ma, and Zhang (2008) and use the partial oracle result
in distribution to argue that subsampling should provide valid confidence intervals for
nonzero parameters. Simulations studies confirm the validity of subsampling to construct
confidence intervals, tests for null hypotheses ansd control the FWER through subsampled
p-values in a low dimensional setting. In the high dimensional setting, confidence intervals
for nonzero coefficients are slightly anticonservative and false positive rates are shown to
be conservative.
v

vi
CONTENTS
Contents
Notation
viii
1 Introduction 1
2 Minimizers of convex processes 3
2.1 Convergence in probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Convergence in distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Weak convergence in metric spaces . . . . . . . . . . . . . . . . . . . 7
2.2.2 Bounded and locally bounded functions . . . . . . . . . . . . . . . . 10
2.3 A continous mapping theorem for argmin functionals . . . . . . . . . . . . . 15
3 Application to the Lasso estimator 21
3.1 Limit in probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Limit in distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2.1 Limiting distribution of components . . . . . . . . . . . . . . . . . . 24
3.2.2 Uniform convergence in the orthogonal case . . . . . . . . . . . . . . 27
4 The adaptive Lasso in a high dimensional setting 31
4.1 Variable selection consistency . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Partial asymptotic normality . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3 Marginal regressors as initial estimates . . . . . . . . . . . . . . . . . . . . . 41
5 Subsampling 43
5.1 Pointwise consistency for distribution estimation . . . . . . . . . . . . . . . 43
5.2 Uniform consistency for quantiles appproximation . . . . . . . . . . . . . . . 46
5.2.1 Statement of the general result . . . . . . . . . . . . . . . . . . . . . 47
5.2.2 Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6 Numerical results 55
6.1 Low dimensinal setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.1.1 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.1.2 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.1.3 F.W.E.R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
6.2 High dimensional setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
7 Concluding remarks 71
Bibliography 72
A R Codes 75
A.1 Simulation code for the Lasso in a low dimensional setting . . . . . . . . . . 75
A.2 Simulation code for the adaptive Lasso in a high dimensional setting . . . . 79

LIST OF FIGURES
vii
List of Figures
3.1 Monte Carlo estimates of the distribution of the root √ n( ˆβ j − β j ), j =
7, . . . , 10, with penalization parameter λ n = 2 √ n. . . . . . . . . . . . . . . . 29
5.1 In red, the subsampling distribution estimates for the roots √ n( ˆβ j − β j ),
j = 1, . . . , 9. n = 10000, b = n 0.65 ≈ 400, B = 2000 with penalization
parameter λ n = 2 √ n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.1 Subsampling confidence intervals for single scenarios of the models A and
A’ (n= 250). Red triangles stand for the true parameters. . . . . . . . . . . 58
6.2 Subsampling confidence intervals for single scenarios of the models B and
B’ (n= 250). Red triangles stand for the true parameters. . . . . . . . . . . 59
6.3 Subsampling confidence intervals for single scenarios of the models C and
C’ (n= 250). Red triangles stand for the true parameters. . . . . . . . . . . 60
6.4 Coverage rates of the two sided confidence intervals I 2 for the adaptive
Lasso in high dimension. Green triangles correspond to relevant variables.
Black dots correspond to noise variables. . . . . . . . . . . . . . . . . . . . . 69

viii
Notation
List of Tables
6.1 Model A and A’. Empirical coverage/false positive rates for the two sided
interval I 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2 Model B nd B’. Empirical coverage/false positive rates for the two sided
interval I 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.3 Model C and C’. Empirical coverage/false positive rates for the two sided
interval I 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.4 Model A and A’. FWER and empirical power for nonzero coefficients. . . . 67
6.5 Model B and B’. FWER and empirical power for nonzero coefficients. . . . 68
6.6 Model C and C’. FWER and empirical power for nonzero coefficients. . . . 68

Notation
Spaces and sets
A
A σ-algebra.
B o The Borel σ-algebra on [0, 1].
B loc (R p ) Set of locally bounded functions on R p .
B(x, ε) Open ball of radius ε around a point x in a metric space.
C b (D) Bounded continous functions on a metric space D.
C min (R p ) Set of continuous functions uniquely minimized on R p .
K δ
δ-enlargement of a compact set K in a metric space.
l ∞ (T ) Space of uniformly bounded functions on a set T .
l ∞ (T 2 , T 2 , . . . ) Space of functions uniformly bounded on each T i , i ∈ N.
R
The extended real line.
Ω
Sample space.
Modes of convergence
→
→ P
→ as∗
→ au
Deterministic or almost sure convergence.
Convergence in probability.
Outer almost sure convergence.
Almost uniform convergence.
Weak convergence.
O notation
X n = o P (a n ) Convergence in probability of {X n /a n } n to zero.
X n = O P (a n ) Uniform tightness of {X n /a n } n .
Further notation
∆J(x) Jump height at x of a right continous function J with left limit.
E, E ∗ , E ∗ Expectation, outer expectation and inner expection, respectively.
f |T Restriction of a function f on a subset T .
P, P ∗ , P ∗ Probability, outer probability and inner probability, respectively.
T ∗ Minimal measurable majorant of a random map T .
T ∗ Maximal measurable minorant of a random map T .
‖X‖ ψd Orlicz norm of a random variable X.
ix

Chapter 1
Introduction
In this thesis we consider the linear regression model
Y i = x ′ iβ + ε i , i = 1, . . . , n
and focus on the Least Absolute Shrinkage and Selection Operator, or Lasso, as estimation
method for the parameter β. The Lasso, defined as
arg min
φ∈R p
∑ n p∑
(x ′ iφ − Y i ) 2 + λ |φ| j ,
i=1
j=1
was introduced by Tibshirani (1996) and has become very popular over the last years.
There are two main reasons for this gain in popularity. The first being, in a context of
high dimensional data analysis, where the dimension of the parameter is very large, by introducing
an l 1 -norm penalty in addition to the squared loss, the Lasso can induce sparsity
on the estimated model while maintaining its prediction ability. That is, many coefficients
will be set to zero, which is a desirable property if one believes that only few parameters
are relevant. The second reason is that the solution can be computed efficiently by convex
optimization. Efron, Hastie, Johnstone, and Tibshirani (2004) introduced the LARS
algorithm which computes the entire solution path, that is, estimates corresponding to all
values λ > 0 with the cost of a single least square regression.
For the sole purpose of prediction, the Lasso proves to be very efficient. Several authors
showed under various assumptions that it enjoys a so-called oracle property, see for instance
Zou (2006), Bunea, Tsybakov, and Wegkamp (2007) or Van De Geer and Bühlmann
(2009). Roughly speaking, an oracle result states that prediction based on the Lasso solution
will be as accurate as if the true model was known. Variable selection properties
of the Lasso are by now also quite well understood and it was shown that consistency in
this mode can be characterized by a so called irrerepresentable condition, see for instance
Zhao and Yu (2006) or Meinshausen and Bühlmann (2006).
Nevertheless, the Lasso theory still needs to bridge a major gap with traditional statistical
inference. Indeed, it is to this date difficult to assign a measure of incertainty to its estimates.
One consequence is that the data analyst who goes beyond the goal of prediction
and is interrested in selecting the exact generating model will typically face many noise
variables for which no statistical tests are available. Also, there are no confidence intervals
for the estimated coeffficents.
Assigning confidence intervals to an estimator or testing for a null hypothesis typically
involves understanding its distributional properties. In the situation where an analytic
1

2 Introduction
solution to this problem doesn’t seem to be sight, as is the case when the estimator results
from numerical optimization, one usually resorts to resampling techniques, of which Efron’s
bootstrap is a prime example. However, proving the consistency (or the inconsistency)
of the bootstrap often proves to be a hard task. Meanwhile, subsampling, or resampling
without replacement, will yield consistency under very weak assumptions. Roughly speaking,
the estimator, or an associated root needs to have a limiting distribution only. In this
thesis we investigate this direction.
Chapter 2 reviews the asymptotic theory for minimizers of convex processes. In Chapter
3, the results obtained are applied to the Lasso estimator in a low dimensional setting in
the vein of Knight and Fu (2000) and continuity propperties of the limiting distribution
are outlined. In Chapter 4, following Huang et al. (2008) the adaptive Lasso is studied
in a sparse high dimensional setting and a partial oracle property (in distribution) is presented.
In Chapter 5, the theory of subsampling is introduced and consistency results
are presented, following the expositions of Politis, Romano, and Wolf (1999) and Romano
and Shaikh (2010). Finally, a simulation study is conducted in Chapter 6 to acess the
performance of subsampling applied to the aforementioned problems.

Chapter 2
Minimizers of convex processes
This chapter is devoted to the asymptotic theory of minimizers of convex processes. The
general message we hope to convey is that both in probability and in distribution, the
determination of the limit for such minimizers can be reduced to the determination of the
pointwise (or marginal) limit of the processes, provided that some regularity conditions
are met and that this pointwise limiting process is uniquely minimized.
2.1 Convergence in probability
We first show that, in probability, pointwise convergence of convex processes on a dense
subset implies uniform convergence on compacts. This result will be derived from the
following determistic version after a subsequencing argument.
Theorem 2.1.0.1. (Rockafellar, 1970, Theorem 10.8) Let C ⊆ R p be open and let {f n :
C → R} n be a sequence of finite convex functions on C. Suppose that {f n } n converges
pointwise on a dense subset of C 0 of C, that is
lim f n(x)
n→∞
exists for every x ∈ C 0 . Then, {f n } n converges pointwise on the whole set C and the limit
function
f(x) := lim
n→∞ f n(x)
is finite and convex on C. Moreover, {f n } n∈N converges uniformly to f on every compact
subset K ⊂ C, that is
sup |f n (x) − f(x)| = 0.
x∈K
Theorem 2.1.0.2. (Andersen and Gill, 1982, Theorem 2.1) Let C ⊆ R p be open and let
{f n } n be a sequence of random convex functions on C such that for every x ∈ C,
f n (x) → P f(x),
where f is a real (random) function on C. Then f is convex and for every compact subset
K ⊂ C, it holds that
sup |f n (x) − f(x)| → P 0.
x∈K
3

4 Minimizers of convex processes
Proof. Recall that convergence in probability implies almost sure convergence along some
subsequence. Consider a dense sequence {x i } i ⊂ C. Since f n (x i ) → P f(x i ) for every
i ∈ N, there exists a subsequence {j 1 n} n for which
f j 1 n
(x 1 ) → f(x 1 ) (2.1.0.1)
almost everywhere.
almost everywhere,
This subsequence contains a further subsequence {j 2 n} n for which,
f j 2 n
(x 2 ) → f(x 2 )
holds in addition to 2.1.0.1.
subsequence {jn} k n satisfying
Repeating this argument, we obtain for every k ∈ N a
f j k n
(x i ) → f(x i ) for i = 1, . . . , k,
almost everywhere. Then for the sequence {j ∗ n} n of diagonal members, i.e. j ∗ n = j n n, we
have
f j ∗ n
(x i ) → f(x i ) for every i ∈ N
almost surely. Next, Theorem 2.1.0.1 implies that f is almost everywhere convex on C
and that
∣
sup ∣f j ∗ n
(x) − f(x) ∣ → 0
x∈K
almost surely for every compact K ⊂ C. We have shown more generally that every
subsequence of {sup x∈K |f n (x) − f(x)|} n has a further subsequence which converges almost
surely to zero. This implies convergence in probability to zero along the whole sequence.
Indeed, suppose that this is not the case, then there exists a subsequence {j n } n and
costants M, ε > 0 such that P (sup x∈K |f jn (x) − f(x)| > M) > ε for all n ∈ N. For every
further subsequence {j n} ′ n , it holds that
(
∣
P ∣f j ′ n
(x) − f(x) ∣ ) ⎛ {
∞⋃ ⋂ ∣
→ 0 ≤ P ⎝
∣f j ′ n
(x) − f(x) ∣ ≤ M} ⎞ ⎠
sup
x∈K
N=1 n≥N
sup
x∈K
(
)
≤ lim P ∣
sup ∣f
n→∞ j ′
N
(x) − f(x) ∣ ≤ M
≤ 1 − ε
which stands in contradiction with the previous conclusion. This argument is known as
the subsequence criterion((Kallenberg, 2002, Lemma 3.2)).
Remark. Note that a more direct and elementary proof of the previous result was given
by (Pollard, 1991, Convexity Lemma).
x∈K
Next, we use this result to infer on the minimizers themselves.
Corollary 2.1.0.3. Let C ⊆ R be open and let {f n } n be a sequence of random convex
functions on C such that
f n (x) → P f(x)

2.1 Convergence in probability 5
for every x ∈ C, where f is a real (random) variable on C. Further, assume that f is
uniquely minimized at α ∈ C and let {α n } n be a sequence of minimizers of {f n } n , i.e.
α n ∈ arg min
x∈C
f n (·), n ∈ N.
Then,
α n → P α.
Proof. The proof rests on (Hjort and Pollard, 1993, Lemma 2) and subsequent remarks.
For arbitrary δ > 0, define
and
∆ n (δ) =
h(δ) =
sup |f n (x) − f(x)|
‖x−α‖≤δ
inf f n(x) − f n (α)
‖x−α‖=δ
First, we show that
P (‖α n − α‖ ≥ δ) ≤ P
(∆ n (δ) ≥ 1 2 h(δ) )
. (2.1.0.2)
For every x ∈ R p with ‖x − α‖ > δ there exist some c with ‖c − α‖ = δ and t ∈ [0, 1] such
that c = tx + (1 − t)α. By convexity of the functions f n , we have
f n (c) ≤ tf n (x) + (1 − t)f n (α).
for every n ∈ N. Writing r n (x) = f(x) − f n (x), we obtain
t(f n (x) − f n (α)) ≥ f n (c) − f n (α)
= f(c) − f(α) + r n (c) − r n (α)
≥ h(δ) − 2∆ n (δ)
In particular,
{∆ n (δ) < 1 }
2 h(δ) ⊆ {f n (x) > f n (α)}
for every x with ‖x − α‖ > δ and every n ∈ N. Since α minimizes f n , this implies that
{‖α n − α‖} ⊆
{∆ n (δ) ≥ 1 }
2 h(δ)
and 2.1.0.2 follows.
Next, we show that
∆ n (δ) = o P (1) (2.1.0.3)

6 Minimizers of convex processes
Let ε > 0, for arbitrary M > 0 we have
P (∆ n (δ) > ε) = P (∆ n (δ) > ε; ‖α‖ ≤ M) + P (∆ n (δ) > ε; ‖α‖ > M)
(
)
≤ P
sup |f n (x) − f(x)| > ε
‖x‖≤δ+M
= o(1) + P (‖α‖ > M)
+ P (‖α‖ > M)
by Theorem 2.1.0.2. Letting M tend to infitnity completes the argument.
Now, we can prove
α n → P α.
Let δ > 0, for fxed arbitrary M > 0, we have
P (‖α n −α‖ > δ) ≤ P
(∆ n (δ) ≥ 1 )
2 h(δ) (
= P ∆ n (δ) ≥ 1 ) (
2 h(δ); 1
h(δ) ≤ M + P ∆ n (δ) ≥ 1 )
2 h(δ); 1
h(δ) > M (
≤ P ∆ n (δ) ≥ 1 ) ( ) 1
+ P
2M h(δ) > M
For fixed M, the first term tends to zero as n tends to infinity by 2.1.0.3. Then, the
second term tends to zero as M tends to infinity. Indeed, h(δ) −1 is almost surely finite by
uniqueness of α. This completes the proof.
2.2 Convergence in distribution
The goal of this section is to derive conditions under which a sequence of minimizers of
convex objective functions converge in distribution and to provide means to determine this
limit. At first, the concept of weak convergence must be revisited to encompass not necessarily
measurable maps defined on probability spaces, this is a feature typically exhibited
by argmin functionals. This wider concept of weak convergence was originally introduced
by Hofmann-Jørgensen but first exposited in Dudley (1985) and Pollard (1990). However,
in the present section we follow the more mature exposition of Van der Vaart and Wellner
(1996). Indeed they showed that even in this general setting, most important results
from weak convergence theory, going from the portmanteau theorem to the almost sure
representation theorem, through Prohorov’s theorem, remain valid, provided one makes
necessary, but essentially minor, modifications. This section is ended with the argmin continuous
mapping theorem which will be applied to the Lasso estimator in the next chapter.
Definition 2.2.0.4. Let (Ω, A, P ) be a probability space and T : Ω → R be an arbitrary
map.
(i) The outer expection and the inner expectation of T with respect to P are defined
as
E ∗ (T ) = inf {E(U)|U ≥ T, U : Ω → R measurable and E(U) exists} (2.2.0.4)

2.2 Convergence in distribution 7
and
E ∗ (T ) = sup {E(U)|U ≤ T, U : Ω → R measurable and E(U) exists} (2.2.0.5)
respectively.
(ii) The outer probability and the innner probability of an arbitrary set B ⊂ Ω are
defined as
and
respectively.
P ∗ (B) = inf {P (A)|A ⊃ B, A ∈ A} (2.2.0.6)
P ∗ (B) = sup {P (A)|A ⊂ B, A ∈ A} (2.2.0.7)
It turns out that outer/inner integrals and outer/inner probabilities are indeed attained
at measurable maps and sets, respectively, as stated in
Lemma 2.2.0.5. (Van der Vaart and Wellner, 1996, Lemma 1.2.1) Let (Ω, A, P ) be a
probability space. For an arbitrary map T : Ω → R, there exist measurable functions
T ∗ , T ∗ : Ω → R with
(i) T ∗ ≥ T ,
(ii) T ∗ ≤ U a.s. for every measurable U : Ω → R with U ≥ T a.s.,
(iii) T ∗ ≤ T ;
(iv) T ∗ ≥ U a.s. for every measurable U : Ω → R with U ≤ T a.s.
For every such T ∗ and T ∗ , it holds that E ∗ (T ) = E(T ∗ ) and E ∗ (T ) = E(T ∗ ), respectively,
provided that E(T ∗ ), respectively E(T ∗ ) exists.
We call T ∗ and T ∗ minimal measurable majorant and maximal measurable minorant
of T , respectively.
2.2.1 Weak convergence in metric spaces
In the remaining, let (D, d) nad (E, e) denote metric spaces.
Definition 2.2.1.1. (Weak convergence) Let (Ω n , A, P n ) , n ∈ N, be probability spaces
and let {X n : Ω n → D} n be a sequence of arbitrary random maps. {X n } n converges weakly
to a Borel measure L if
This is denoted by X n L.
∫
E ∗ (f(X n )) → fdL,
for every f ∈ C b (D).

8 Minimizers of convex processes
Remark. Weak convergence is also defined for convergence toward a Borel measurable
map X defined on a probability space (Ω, A, P ) by taking L := P ◦ X −1 , this is denoted
by X n X. Note that, while measurability of the maps X n has been dropped, it remains
mandatory for the limit X.
The Portmanteau theorem and the continuous mapping are essential tools in asymptotic
statistics. They remain valid in this general framework. Their proof are omitted but can
be found in the references.
Theorem 2.2.1.2. (Van der Vaart and Wellner, 1996, Theorem 1.3.4) Let {X n } n be a
sequence of arbitrary random maps and let L be a Borel measure. The following statements
are equivalent :
(i) X n L;
(ii) lim inf P ∗ (X n ∈ G) ≥ L(G) for every open G;
(iii) lim sup P ∗ (X n ∈ F ) ≤ L(F ) for every closed F;
(iv) lim P ∗ (X n ∈ B) = lim P ∗ (X n ∈ B) = L(B) for every Borel L-continuous set B, that
is B with L(∂B) = 0 ;
(v) lim inf E ∗ f(X n ) ≥ ∫ fdL for every bonded, Lipschitz continuous, nonnegative function
f.
Theorem 2.2.1.3. (Van der Vaart and Wellner, 1996, Theorem 1.3.6) Let g : D → E be
continous at every point in a set D 0 ⊂ D. If X n X and X takes its values in D 0 , then
g(X n ) g(X).
Definition 2.2.1.4. A Borel probability measure L is called tight if for every ε > 0, there
exists a compact set K ⊂ D with L(K) ≥ 1 − ε. A Borel measurable map X : Ω → D is
called tight if its law P ◦ X −1 is tight.
Next, we want to state Prohorov’s theorem which is a fundamental result in weak convergence
theory, it associates sequential compactness (with respect to weak convergence) to
the concept of tightness, hence giving conditions for the existence of a weak limit. In the
present setting, the statement of the result requires introducing the concept of asymptotic
measurability first.
Definition 2.2.1.5. (Asymptotic measurability and tightness) A sequence of arbitrary
random maps {X n : Ω n → D} n is asymptotically measurable if and only if
E ∗ f(X n ) − E ∗ f(X n ) → 0,
for every f ∈ C b (D).
The sequence {X n } n is called asymptotically tight if for every ε > 0, there exists a compact
set K such that
lim inf
n→∞ P ∗(X n ∈ K δ ) ≥ 1 − ε, for every δ > 0.
Here, K δ = {y ∈ D | d(x, K) < δ} is the δ-enlargement around K, an open set.
Remark. One can show that for a sequence {X n } n of Borel measurable maps, asymptotic
tightness and the usual uniform tighness, that is, the existence for every ε > 0 of a compact
set K ⊂ D satisfying P (X n ∈ K) ≥ 1 − ε for every n ∈ N, are equivalent.

2.2 Convergence in distribution 9
Lemma 2.2.1.6. (Van der Vaart and Wellner, 1996, Lemma 1.3.8)
(i) If X n X, then X n is asymptotically measurable.
(ii) If X n X, then X n is tight if and only if X n is tight.
Definition 2.2.1.7.
(i) A vector lattice F ⊂ C b (D) is a vector space that is closed under taking positive parts,
that is, if f ∈ F then f + = f ∨ 0 ∈ F.
(ii) An algebra F ⊂ C b (D) is a vector space that is closed under taking products, i.e. if
f, g ∈ F, then fg : x ↦→ f(x)g(x) ∈ F.
(iii) A set of functions on F on D is said to separate points in D if, for every pair x ≠ y,
there is a f ∈ F with f(x) ≠ f(y).
Lemma 2.2.1.8. (Van der Vaart and Wellner, 1996, Lemma 1.3.12)
(i) Let L 1 and L 2 be finite measurable measures on D. If ∫ fdL 1 = ∫ fdL 2 for every
f ∈ C b (D), then L 1 = L 2 .
(ii) Let L 1 and L 2 be tight Borel probability measures on D. If ∫ fdL 1 = ∫ fdL 2 for every
f in a vector laticce F ⊂ C b (D) that contains the constant functions and separates
points of D, then L 1 = L 2 .
Lemma 2.2.1.9. (Van der Vaart and Wellner, 1996, Lemma 1.3.13) Let {X n : Ω → D} n
be a sequence of arbitrary maps. Suppose that {X n } n is asymptotically tight and that
E ∗ f (X n ) − E ∗ f (X n ) → 0 (2.2.1.1)
for every f in a subalgebra F ⊂ C b (D) that separates points of D. Then {X n } n
is asymptotically
measurable.
Lemma 2.2.1.10. (Van der Vaart and Wellner, 1996, Lemma 1.4.3) Sequences {X n :
Ω → D} n and {Y n : Ω n → E} of arbitrary random maps are asymptotically tight if and
only if the same is true for (X n , Y n ) : Ω n → D × E.
Lemma 2.2.1.11. (Van der Vaart and Wellner, 1996, Lemma 1.4.4) Asymptotically tight
sequences {X n : Ω → D} n and {Y n : Ω n → E} are asymptotically measurable if and only if
the same is true for (X n , Y n ) : Ω n → D × E.
Remark. Both previous results remain true for finitely many or countably many sequences
of maps.
Theorem 2.2.1.12. (Van der Vaart and Wellner, 1996, Theorem 1.3.9)(Prohorov’s
theorem) Let {X n : Ω n → D} be a sequence of arbitrary random maps. If {X n } n is
asymptotically tight and asymptotically measurable, then it has a subsequence {X jn } n that
converges weakly to a tight Borel law.

10 Minimizers of convex processes
2.2.2 Bounded and locally bounded functions
Uniformly bounded functions
We are ultimatelly interrested in convex functions which are locally bounded, so from now
on we will first focus on
(D, d) = (l ∞ (T ), ‖·‖ T )
That is ,the space of bounded functions z : T → R on an arbitrary set T with the supremum
norm ‖z‖ T = sup t∈T |z(t)|. Later, we will see that asymptotic tightness of locally
bounded processes can be characterized in terms of asymptotic tightness of restrictions on
compacta.
The next result reformulates in this context the abstract characterization of asymptotic
measurability and uniqueness of Borel measures given in Lemma 2.2.1.9 and 2.2.1.8. They
give a first hint at the important role played by marginals distributions.
Lemma 2.2.2.1. (Van der Vaart and Wellner, 1996, Lemma 1.5.2 and Lemma 1.5.3)
(i) Let {X n : Ω n → l ∞ (T )} n be asymptotically tight. Then it is asymptotically measurable
if and only if {X n (t)} n is asymptotically measurable for every t ∈ T .
(ii) Let X and Y be tight Borel measurable maps into l ∞ (T ). Then X and Y are equal
in Borel law if and only if all corresponding marginals of X and Y are equal in law.
Proof. One easily verifies that the set F ⊂ C b (l ∞ (T )) of continuous functions f of the
form f(z) = g(z(t 1 ), . . . , z(t k )) forms an algebra and a vector lattice, that it contains the
constant functions and that it separates points of T .
(i) Note that for M > 0, {|X n (t)| ≤ M} ⊂ {‖X n ‖ |T ≤ M} n for every t ∈ T . In particular
assymptotic tightness of a sequence {X n : Ω n → l ∞ (T )} implies asymptotic
tightness of {X n (t)} n for every t ∈ T .
Hence, the necessity of assymptotic tightness for {X n (t)} n for every t ∈ T follows
directly from the fact that coordinate maps {π t : l ∞ → R}, t ∈ T are continuous.
For the sufficiency of this assymptotic tightness, note first that given the
assumption and the fact that F defined above forms a vector laticce containing constants
and separating points, it follows from Lemma 2.2.1.11 that every sequence
{(X n (t 1 ), . . . , X n (t k )) : Ω n → R k } n is asymptotically measurable. The claim is then
a direct consequence of Lemma 2.2.1.9 with F given above.
(ii) If X and Y have equal Borel laws on l ∞ (T ), then their marginals are also equal
in law since coordinates maps π t1 ,...,t k
: l ∞ (T ) → R k are Borel measurables. The
converse statement follows from Lemma 2.2.1.8.
Theorem 2.2.2.2. (Van der Vaart and Wellner, 1996, Lemma 1.5.4) Let {X n : Ω n →
l ∞ (T )} n be a sequence of arbitrary random maps. Then {X n } n converges weakly to a
tight limit if and only if {X n } is asymptotically tight and for every finite subset t 1 , . . . , t k

2.2 Convergence in distribution 11
of T , the sequence of marginals {(X n (t 1 ), . . . , X n (t k ))} n converges weakly to a tight limit. If
{X n } n is asymptotically tight and its marginals converge to the marginals ((X(t 1 ), . . . , X(t k ))
of a stochastic process X, then there is a version of X with uniformly bounded sample paths
and X n X.
Proof. Coordinates maps π t1 ,...,t k
are continuous so it follows from the continuous mapping
Theorem 2.2.1.3 that for every subsequence and every finite subset t 1 , . . . , t k of T {j n } n ,
X jn L jn implies (X jn (t 1 ), . . . , X jn (t k )) L jn ◦ π −1
(t 1 ,...,t k ). In particular, if the whole
sequence converges weakly to a tight limit, the same is true for every sequence of marginals;
asymptotic tightness follows from Lemma 2.2.1.6.
Conversely, suppose that {X n } n is asymptotically tight and that the marginals converge,
then it follows from Lemma 2.2.2.1, trough Lemma 2.2.1.11, that {X n } n is asymptotically
measurable. Asymptotic measurability and tightness are kept under subsequencing, hence,
by Prohorov’s theorem 2.2.1.12, every subsequence {X jn } n converges weakly to a tight
limit, say L jn . It remains to show that the limits are the same. Invoke the continuous
mapping theorem again to show that for every subsequence {j n } n , X jn L jn implies
(X jn (t 1 ), . . . , X jn (t k )) L jn ◦ π −1
(t 1 ,...,t k ) . By assumption, marginals (X j n
(t 1 ), . . . , X jn (t k ))
share the same weak limit, namely the weak limit of (X n (t 1 ), . . . , X n (t k )), uniqueness
now follows from the second part of lemma 2.2.2.1 applied to the measures L jn . Finally,
if {X n } n is asymptotically tight and its marginals converge to the marginals of some
stochastic process X, then {X n } n converge to some Borel measurable process Y : Ω Y →
l ∞ (T ) which has the same marginal distribution as X, again by the second part of lemma
2.2.2.1. This completes the proof.
Definition 2.2.2.3. A sequence {X n : Ω → l ∞ (T )} of arbitrary random maps is asymptotically
uniformly ρ-equicontinuous if for every ɛ, η > 0 there exists a δ > 0 sucht that
)
(
lim sup P ∗ sup |X n (s) − X n (t)|
n→∞ ρ(s,t)
Theorem 2.2.2.4. (Van der Vaart and Wellner, 1996, Theorem 1.5.7)
(i) A sequence {X n : Ω n → l ∞ (T )} n of arbitrary random maps is asymptotically tight
if and only if {X n (t)} n is asymptotically tight in R for every t and there exists a
semimetric ρ on T such that (T, ρ) is totally bounded and {X n } n is asymptotically
uniformly ρ-equicontinous in probability.
(ii) If, moreover, X n X, then almost all paths t ↦→ X(t, ω) are uniformly ρ-continuous;
and the semimetric ρ can without loss of generality be taken equal to any semimetric
ρ for which this is true and (T, ρ) is totally bounded.
Proof. We prove sufficiency of the conditions in i, see the reference for the other direction
and the second part. Suppose that {X n (t)} n is asymptotically tight in R for every t ∈
T and that there exists a semimetric ρ on T such that (T, ρ) is totally bounded and
that {X n } n is asymptotically uniformly ρ-equicontinuous in probability. Using totally
boundedness of (T, ρ), then disjointification of sets we conclude that for every ε, η > 0,
there is a finite partition T = ∪ k i=1 T i such that
)
lim sup
n→∞
P ∗ (sup
i
sup |X n (s) − X n (t)| > ε
s,t∈T i
> η (2.2.2.1)

12 Minimizers of convex processes
For such a partition and an arbitrary choice of points t i ∈ T i , i = 1, . . . , k, we have
(
)
(
)
lim inf P ∗ ‖X n ‖
n→∞ |T ≤ max |X n(t i )| + ε ≥ lim inf P ∗ sup sup |X n (s) − X n (t)| ≤ ε
i=1,...,k n→∞ i s,t∈T i
≥ 1 − η.
The maximum of finitely many asymptotically tight sequences of real valued maps is
asymptotically tight (consider the finite union of compacts), so it follows that {‖X n ‖} n is
asymptotically tight in R.
Fix ζ > 0 and consider a sequence ε m ↘ 0. Take a constant M > 0 with
(
lim sup P ∗ ‖X n ‖ |T > M
n→∞
)
< ζ
by invoking asymptotic tightness, and for each ε = ε m and η = 2 −m ζ, take a partition
T = ∪ km
1=1 t i satisfying 2.2.2.1. Fix m for the moment and consider the set
Z m =
⋂ {
}
z ∈ l ∞ (T ) | z |Ti ≡ jε m for some j ∈ {−⌊M/ε⌋, . . . , ⌊M/ε⌋} .
i=1,...,k m
Z m is obviously finite, denote its elements by z 1 , . . . , z p and define
K m =
⋃
B(z i , ε m ).
Then the two conditions
i=1,...,p
‖X n ‖ |T ≤ M and sup
i
sup |X n (s) − X n (t)| ≤ ε m
s,t∈T i
imply that X n ∈ K m . An explicit z ∈ Z m with ‖X n − z‖ T ≤ ε m is given by setting
z |Ti ≡ c for some c ∈ ( inf s∈Ti |X n (s)|, sup s∈Ti |X n (s)| ) ∩ {0, ±ε m ± ⌊M/ε m ⌋ε m }. This is
true for every m. Let K = ∩ ∞ i=1 K m, K is closed, it is totally bounded since ε m ↘ 0; we
are working in a metric space so it follows that K is compact. Now we show that for every
δ > 0, there is a m with K δ ⊃ ∩ m i=1 K i. Indeed, if this is not true then there exists a
sequence {z m } m not contained in K δ but with z m ∈ ∩ m 1=1 for every m. This sequence has
a first subsequence contained in one of the balls making up K 1 , then a further subsequence
contained in one of the balls making ip K 2 . By the usual diagonal argument it follows
that there is a subsequence which is contained in a single ball of radius ε m for every m,
hence Cauchy. Since K is closed its limit is contained in K, which stands in contradiction
to the fact that {z m } m ⊂ l ∞ \ K δ . It follows that
{
}
m⋃
{X n /∈ K δ } ⊂ X n /∈ K m
for some fixed m. For this m, we have
lim sup
n→∞
( P ∗ X n /∈ K δ) ≤ lim sup P
(X ∗ n /∈
n→∞
which concludes the proof.
≤ lim sup P ∗ (‖X n ‖ > M) +
n→∞
m∑
≤ ζ + ζ2 −m < 2ζ,
i=1
i=1
)
m⋂
K i
i=1
m∑
i=1
P ∗ (sup
i
sup |X n (s) − X n (t)| > ε
s,t∈T i
)

2.2 Convergence in distribution 13
Locally bounded functions
Next, for an arbitrary index set T and a covering sequence T 1 ⊂ T 2 ⊂ . . . of subsets
(T = ∪ ∞ i=1 T i), denote by l ∞ (T 1 , T 2 , . . . ) the set of functions z uniformly bounded on each
T i and equip it with the metric
∞∑
d(z 1 , z 2 ) = (‖z 1 − z 2 ‖ Ti ∧ 1) 2 −1 .
i=1
It turns out that weak convergence with respect to this topology can be characterized in
terms of weak convergence of the restrictions on sets T i .
Remark. When T = R p and the sets T i are chosen equal to closed balls B(0, i) or cubes
[−i, i] p , ( i ∈ N), d induces the topology of uniform convergence on compacta. We will
consider this case at the end of this chapter.
Theorem 2.2.2.5. (Van der Vaart and Wellner, 1996, Theorem 1.6.1) Let {X n : Ω n →
l ∞ (T 1 , T 2 , . . . )} n be a sequence of arbitrary maps. {X n } n converges weakly to a tight limit
if and only if each of the sequences of restrictions {X |Ti : Ω n → l ∞ (T i )} n (i ∈ N) converges
weakly to a tight limit.
Proof. The restriction maps
l ∞ (T 1 , T 2 , . . . ) → l ∞ (T i )
z ↦→ z |Ti
i ∈ N, are continuous, so the necessity of convergence for all sequences of restrictions
follows from the continuous mapping Theorem.
Conversely, suppose that for every i ∈ N, {X |Ti : Ω n → l ∞ (T i )} n converges weakly to a
tight limit. Recall that weak convergence implies asymptotic tightness. ( For fixed ) ε > 0,
find for every i a compact set K i ⊂ l ∞ (T i ) such that lim sup n→∞ P ∗ X |Ti /∈ Ki
δ < ε for
every δ > 0. Then
(i) Define
K =
{
z : T → R | z |(Ti \T i−1 ) ∈ (K i ) |Ti \T i−1
}
for every i ∈ N ⊂ l ∞ (T 1 , T 2 , . . . ).
K is compact. Indeed, let {z n } n be a sequence in K, for every i ∈ N there is
a sequence {z n (i) } n in K i such that (z n ) |Ti \T i−1
= (z n (i) ) |Ti \T i−1
for all n ∈ N. By
compactness of K i , every subsequence of {z n (i) } n has a convergent subsequence. First,
find a convergent subsequence {jn} 1 n of {z n
(1) } n , then extract a further subsequence
{jn} 2 n for which {z n (2) } n also converges. Proceed further so with the remaining i ≥ 2.
We verify that the diagonal subsequence {z j n } n is convergent. To do so, first note
that for deterministic functions {z n } n ⊆ l ∞ (T 1 , T 2 , . . . ), the convergence of every
sequence of restrictions in l ∞ (T i ) implies the convergence in l ∞ (T 1 , T 2 , . . . ). Indeed,
for n, m, i ∈ N it holds that
i∑
‖z j n − z j m ‖ Ti = ‖(z j n − z j m ) |T1 ‖ T1 + ‖(z j n − z j m ) |Tl \T l−1
‖ |Tl \T l−1
l=2
i∑
= ‖(z (1)
jn
− z(1) jm ) |T 1
‖ T1 + ‖(z (l)
jn
− z(l) jm ) |T l \T l−1
‖ |Tl \T l−1
l=2

14 Minimizers of convex processes
Since every sequence {z (l)
j n n } n is Cauchy in l ∞ (T l ) by construction and
‖(z (l)
j n n
− z(l) jm ) |T l \T l−1
‖ |Tl \T l−1
≤ ‖z (l)
jn
− z(l) jm ‖ |T l
for l = 1, . . . , i, it follows that {(z j n n
) |Ti } n is Cauchy in l ∞ (T i ).
(ii) Using compactness of the sets K i one easily verifies that for arbitrary δ > 0, it holds
that
{
}
z ∈ l ∞ (T 1 , T 2 , . . . ) | z |Ti ∈ Ki
δ for every i ∈ N ⊆ K δ .
This implies that
(
lim sup P X n /∈ K δ) ≤
n→∞
∞∑
P
i=1
(
X n|Ti
)
/∈ Ki
δ < ε
for every δ > 0, that is, {X n } n
is asymptotically tight in l ∞ (T 1 , T 2 , . . . ).
Then, consider the set F of continuous functions f : l ∞ (T 1 , T 2 , . . . ) → R of the form
f(z) = g(z(t 1 ), . . . , z(t k )) with g ∈ C b (R), t 1 , . . . , t k ∈ T and k ∈ N. Obbviously
F is an algebra that separates points in l ∞ (T 1 , T 2 , . . . ). Lemma 2.2.1.9 now implies
that {X n } n is asymptotically measurable in l ∞ (t 1 , T 2 , . . . ).
(iii) Finally Prohorov’s theorem 2.2.1.12 asserts that every subsequence of {X n } n has a
further subsequence which converges weakly to a tight limit. This limit is unique in
viwe of Lemma 2.2.1.8
Corollary 2.2.2.6. (Weak convergence of convex processes) Let {X n } n be a sequence
of stochastic processes indexed by a convex, open subset C of R p such that every
sample path t ↦→ X n (ω, t) is convex on C. If {X n } n converges marginally in distribution
to a limit, then it converges in distribution to a tight limit in the space l ∞ (K 1 , K 2 , . . . , )
for any sequence of compacts sets K 1 ⊂ K 2 ⊂ · · · ⊂ C. In particular, it converges to a
tight limit with respect to the topology of uniform convergence on compacta.
Proof.
(i) We show that for every compact K ⊂ C, there exists an ε > 0 such that K ε ⊂ K 2ε ⊂
C. In particular, ‖f‖ K ε = sup x∈K ε |f(x)| is well defined for every convex function
f : C → R.
Indeed, for each x ∈ K one can find an ε x such that B(x, 2ε x ) ⊂ C. Let ∪ i∈I B(x i , 2ε xi )
be a finite covering of K with such balls. Set ε = min i∈I ε xi . One easily verifies that
K 2ε ⊂ ∪ i∈I B(x i , 4ε) then holds.
(ii) Next, we first show that a sequence {x n : C → R} n of deterministic convex functions
such that {x n (t)} n is bounded for every t is automatically uniformly bounded over
K ε , i.e. sup n ‖x n ‖ K ε < ∞
For such functions t ↦→ sup n |x n (t)| is finite and convex on C, hence uniformly
continuous on every compact subset of C. This implies that for arbitrary ε > 0
there exists some δ > 0 such that for every t 0 ∈ K ε , sup n |x n (·)| ≤ sup n |x n (t 0 )| +
ε on B(t 0 , δ). Cover K ε by finitely many balls B(t i , δ), i = 1, . . . , N. Then it holds
that sup n ‖x n ‖ K ε ≤ max i sup n |x n (t i )| + ε.
We can then conclude that the sequence {‖X n ‖ K ε} n is asynmptotically tight.

2.3 A continous mapping theorem for argmin functionals 15
(iii) A bounded convex function x on K ε is automatically Lipschitz on K with Lipschitz
constant (2/ε)‖x‖ K ε.
Indeed, for arbitrary y 1 , y 2 ∈ K, set z = y 1 + η(y 1 − y 2 )/‖y 1 − y 2 ‖ with η < ε. Then
z ∈ K ε and y 1 = λz + (1 − λ)y 2 , λ = ‖y 1 − y 2 ‖/(‖y 1 − y 2 ‖ + η) ∈ [0, 1]. By convexity
of x, we have x(y 1 ) − x(y 2 ) ≤ λ(f(z) − f(y 2 )) ≤ 2λ‖x‖ K ε. Since z ∈ K ε and x ∈ K
it follows from the triangle inequality that λ ≤ 1/ε.
(iv) We can now show that {X n } n is asymptotically equicontinuous in l ∞ (K).
For arbitrary ε equi > 0, η equi > 0, choose M > 0 with
lim sup P (‖X n ‖ K ε > M) < η equi
n→∞
by invoking asymptotical tightness of ‖X n ‖ K ε. On {‖X n ‖ K ε ≤ M} it holds that
|X n (t) − X n (s)| ≤ (2/ε)M‖t − s‖. Set δ = ε equi ((2/ε)M) −1 . Then,
{
}
sup |X n (t) − X n (s)| > ε equi ⊆ {‖X n ‖ K ε > M} .
‖t−s‖

16 Minimizers of convex processes
(ii) X n converges almost uniformly to X if for every ε > 0, there exists a measurable set
A with P (A) ≥ 1 − ε and d(X n , X) → 0 uniformly on A; this is denoted X n → au X.
Almost uniform convergence is more convenient to prove, so we will make use of
Lemma 2.3.0.9. (Van der Vaart and Wellner, 1996, Lemma 1.9.2)Let X be Borel measurable.
Then X n → au X if and only if X n → as∗ X.
Theorem 2.3.0.10. (Dudley’s almost sure representation theorem)
For probability spaces (Ω n , A n , P n ), n ∈ N ∪ {∞}, let X n : Ω n → D be arbitrary maps and
let X ∞ : Ω → D be Borel measurable and separable. If X n X ∞ , then there exists a
probability space (˜Ω, Ã, ˜P ) and maps X n : ˜Ω → D, n ∈ N ∪ {∞} with
(i) ˜X n → as∗ ˜X∞
(
(ii) E ∗ f( ˜X
)
n ) = E ∗ (f(X n )), for every bounded f : D → R and every n ∈ N ⋃ {∞}.
Moreover each ˜X n can be chosen as ˜X n = X n ◦ φ n with φ n measurable and perfect, and
P n = ˜P ◦ φ −1
n .
Proof. Call a set B ⊂ D a continuity set if P (X ∞ ∈ ∂B) = 0.
(i) We first show that for every ε > 0, there exists a partition of D into finitely many
disjoint continuity sets B (ε)
0 , B(ε) 1 , . . . , B(ε) k ε
satisfying
(
P X ∞ ∈ B (ε) )
0 < ε 2 (2.3.0.2)
and
diam(B (ε)
i ) < ε (2.3.0.3)
for i = 1, . . . , k ε . Let C ⊂ D be a separable subset for which P (X ∞ ∈ C) = 1 and let
{s i } i be a dense sequence in C. For each i ∈ N, there are at most countably many
values r > 0 for which the open ball B(s i , r) is a discontinuity set (every σ−finite
measure space has at most countably many disjoint sets with positive measure). So,
choose ε/3 < r i < ε/2 such that B(s i , r i ) is a continuity set. Then for every i, set
B (ε)
i = B(s i , r i )\ ⋃ j 1 − ε 2 , so set B (ε)
0 = D \ ⋃ i≤k ε
B (ε)
i .
(ii) There is a sequence ε n → 0 taking values 1/m (m ∈ N) only, such that
(
P n∗ X n ∈ B (εn) )
(
≥ (1 − ε n ) P ∞ X ∞ ∈ B (εn) )
, i = 1, . . . , k εn . (2.3.0.4)
i
By the ( Portmanteau theorem, for fixed ε > 0 and i = 1, . . . , k ε , it holds that
P n∗ X n ∈ B (εn) ) (
i → P ∞ X ∞ ∈ B (εn) )
i . So, for m ∈ N choose n(m) be such that for
i = 1, . . . , k 1/m and n ≥ n(m)
(
P n∗ X n ∈ B (1/m) )
(
≥ (1 − 1/m) P ∞ X ∞ ∈ B (1/m) )
, (2.3.0.5)
i
i
i

2.3 A continous mapping theorem for argmin functionals 17
Next, assume without loss of generality that 1 = n(1) < n(2) < . . . and set γ(n) =
max{k : n(k) ≤ n} and define ε n = 1/γ(n). The sequence ε n then satisfies 2.3.0.4
since γ(n) → ∞ and n(γ(n)) ≤ n by definition.
(iii) For i = 1, . . . , k εn , let A n i
and set
⊆ {X n ∈ B (εn)
i } be measurable with
(
P n (A n i ) = P n∗ X n ∈ B (εn) )
A n 0 = Ω n \
k εn
⋃
i=1
A n i .
For each n ∈ N define a probability measure µ n on (Ω n , A n ) as
µ n (A) = 1
k
∑ εn
P n (A|A n i ) ( P n (A n i ) − (1 − ε n ) P ∞ (X ∞ ∈ Bi εn )) .
ε n
i=1
where P n (·|A n i ) is the P n-conditional measure given A n i . Finally, define the probability
space (˜Ω, Ã, ˜P ) as follows :
⎛
⎞
˜Ω = Ω ∞ × ∏ k
∏ εn
⎝Ω n × A n ⎠
i × [0, 1],
n
i=0
⎛
⎞
à = A ∞ × ∏ k
∏ εn
⎝A × A n ∩ A n ⎠
i × B o ,
n i=0
⎛
⎞
˜P = P ∞ × ∏ k
∏ εn
⎝µ n × P n (·|A n i ) ⎠ × λ.
n
Here, B o is the Borel σ-algebra and λ is the Lebesgue measure on [0, 1], respectively.
i=0
(iv) We now define the maps φ n and verify that ˜P ◦ φ −1
n
Define
For A ∈ Ã, we have:
˜ω = ( ω ∞ , . . . , ω, ω n0 , . . . , ω kɛn n, . . . , ξ ) .
i
= P n . Write elements ˜ω of ˜Ω as
φ ∞ = ω ∞
{
ωn , if ξ > 1 − ε n ,
φ n =
ω ni , if ξ ≤ 1 − ε and X ∞ (ω ∞ ) ∈ B (ε)
i .
˜P (φ n ∈ A) = ˜P (φ n ∈ A; ξ > 1 − ε n ) + ˜P (φ n ∈ A; ξ ≤ 1 − ε n )
k εn
= ˜P
∑ (
(ω n ∈ A; ξ > 1 − ε n ) + ˜P ω ni ∈ A; ξ ≤ 1 − ε n ; X ∞ ∈ B (εn) )
i
i=0
k
∑ εn
(
= µ n (A) ε n + P n (A|A n i ) (1 − ε n )P ∞ X ∞ ∈ B (εn) )
i
i=0
k εn
∑
= P n (A|A n i ) P n (A n i )
i=0
= P n (A)

18 Minimizers of convex processes
(v) We show that ˜X n → au ˜X∞ . It follows from the construction of the maps ˜X n =
X n ◦ φ n and of the balls B (εn)
i
that d( ˜X n , ˜X ∞ ) on {˜ω : X ∞ /∈ B o
(εn) , ξ ≤ 1 − ε}. Set
A k = ∪ m≥k {˜ω : X ∞ ∈ B 1/m
0 or ξ > 1 − 1/k}. Note that ˜P (A k ) ≤ 1/k, so for every
ε > 0 there exists some k with ˜P (A k ) ≤ ε. For ˜ω ∈ ˜Ω\A k and n such that ε n ≤ 1/k,
it holds that d( ˜X n , ˜X ∞ ) ≤ ε n . This completes the argument.
(vi) Let T : Ω n → R be bounded, and let T ∗ be its minimal measurable cover with respect
to P n . Write
k
∑ εn
T ◦ φ n = 1 πξ ≤1−ε n
T |A n
i
i=0
◦ π n,i 1 X∞◦π ∞∈B (εn)
i
+ 1 πξ >1−ε n
T ◦ π n .
The coordinate projections are perfect so that the minimal measurable cover of (T ◦
φ n ) with respect to ˜P can be computed as follows
k
(T ◦ φ n ) ∗ ∑ εn
= 1 πξ ≤1−ε n
(T |A n
i
◦ π n,i ) ∗ 1 X∞◦π∞∈B (εn) + 1 πξ >1−ε n
(T ◦ π n ) ∗
i=0
k
∑ εn
= 1 πξ ≤1−ε n
(T |A n
i
) ∗Pn[·|An i ] ◦ π n,i 1 X∞◦π∞∈B (εn) + 1 πξ >1−ε n
(T ) ∗µn ◦ π n
i=0
Now note that P n and P n (·|A n i ) are equivalent on (An i , A n∩A n i ), P n and µ n are equivalent
on (Ω n , A n ), hence the measurable covers in the last formula can be computed
under P n . It follows that (T ◦ φ n ) ∗ = T ∗ ◦ φ n . This completes the proof.
i
i
In the following, denote by B loc (R p ) the space of locally bounded functions on R p equipped
with the topology of uniform convergence on compacta; denote by C min (R p ) the separable
subset of continuous functions z(·) with minimum achieved at a unique point in R p and
satisfying z(t) → ∞ as |t| → ∞.
Dudley’s Almost sure representation theorem is used to prove
Theorem 2.3.0.11. (Argmin continuous mapping theorem)
Let {X n : Ω n → B loc (R p )} n and {t n : Ω n → R p } n be sequences of random maps. If
(i) X n X for X Borel measureable and concentrated on C min (R p );
(ii) t n = O P (1);
(iii) X n (t n ) ≤ inf t∈R p X n (t) + α n for random variables {α n } n of order o P (1);
then t n arg min(X).
Proof. Invoke Dudley’s almost sure representation Theorem 2.3.0.10 to find a probability
space (˜Ω, Ã, ˜P ) and maps ˜X n : ˜Ω → B loc (R p ), ˜X : ˜Ω → Bloc (R p ) satisfying
˜X n → as∗ ˜X (2.3.0.6)

2.3 A continous mapping theorem for argmin functionals 19
and
E ∗ (f( ˜X n )) = E ∗ (f(X n )) (2.3.0.7)
for every f ∈ C b (B loc (R p )) and every n ∈ N. According to the same theorem, we can
assume that the maps ˜X n take the form
˜X n = X n ◦ φ n (2.3.0.8)
for measurable perfect maps φ n : ˜Ω → Ω satisfying P n = ˜P ◦ φ −1
n .
Next, consider the maps ˜t n , ˜t and ˜α n obtained after composittion with φ n . Denote by ˜t
the unique minimizer of ˜X, using a density argument, one can show that ˜t is measurable,
hence has the same law as arg min(X). We need to prove that
E ∗ f(t n ) − Ẽf(˜t) → 0 (2.3.0.9)
for every f ∈ C b (R p ). By perfectness of the maps φ n , the difference is equal to
Ẽ ∗ ( f(˜t n ) − f(˜t) ) ≤ Ẽ∗ (∣∣ f( ˜ t n ) − f(˜t) ∣ ∣ ) .
To prove 2.3.0.9, first define for abitrary δ > 0
{ }
˜∆(δ) = inf ˜X(t) | ‖t − ˜t‖ > δ − ˜X(˜t).
Then, for an arbitrary ε > 0, use tightness of {˜t n } n and uniqueness of ˜t to find R > 0,
η > 0 satisfying :
lim sup
n→∞
˜P ∗ ( ‖˜t n ‖ > R ) < ε
and
˜P
(
‖˜t‖ > R or ˜∆(δ)
)
< η < ε.
The restriction of f on the closed ball B(0; R) is uniformly continuous by Heine-Borel’s
Theorem. Since
E ∗ (f(˜t n ) − f(˜t)) ≤ E ∗ (f(˜t n ) − f(˜t))1 { ‖˜t n ‖, ‖˜t‖ ≤ R } + 2 · 2‖f‖ |B(0;R) ε
for n sufficiently large, it is sufficent to show that
˜P ∗ ( ‖˜t n − ˜t‖ > δ; ‖˜t n ‖, ‖˜t‖ ≤ R ) → 0
for each δ > 0.
According to the Representation theorem, it holds that
2 −R sup
‖t‖≤R
(
‖ ˜X n (t) − ˜X(t)‖
) ∗ (
≤ d( ˜X ˜X)) ∗
n , ≤ ε,
for n sufficently large. Furthermore, it follows from
˜X n (˜t) = ˜X n (t) + ˜X(t) − ˜X n (t)
+ ˜X(˜t) − ˜X n (˜t)
+ ˜X(˜t) − ˜X(t)

20 Minimizers of convex processes
that
{
‖˜t‖, ‖t‖ ≤ R; ‖t − ˜t‖ > δ; ˜∆(δ)
} {
≤ η ⊆ ˜Xn (˜t) ≤ ˜X
}
n (t) + 2 R+1 ε − η . (2.3.0.10)
The event
{
}
η > ˜α n + 2 R+1 ε
has probability tending to one as n → ∞, since ˜α n = o ˜P
(1). Note that by perfectness of
the maps φ n , one also has
˜X n (˜t n ) ≤ inf
t
˜X n (t) + ˜α n
for every n ∈ N. Thus, 2.3.0.10 forces ˜t n , either to be at a distance greater than R from
zero, or to stay within a δ neighborhood of ˜t. We finally obtain
˜P ∗ ( ‖˜t n − ˜t‖ > δ; ‖˜t n ‖, ‖˜t‖ ≤ R ) ≤ 3ε
for n large enough. This completes the proof.

Chapter 3
Application to the Lasso estimator
In this chapter, we first apply Theorem 2.1.0.2 and 2.3.0.11 to determine the limit in
probability and in distribution of the Lasso estimator, defined as ˆβ n = arg min φ∈R p Z n (φ)
for
for a linear regression model
Z n (φ) = 1 n∑
(Y i − x ′
n
iφ) 2 + λ n
p∑
|φ j |. (3.0.0.1)
n
i=1
j=1
Y i = x ′ iβ + ε i , i = 1, . . . , n. (3.0.0.2)
Then some continuity properties of the limit distributions are derived and it is shown
under assumption of an orthogonal design that the convergence in distribution is actually
uniform.
Throughout this chapter we make the following assumption on the model 3.0.0.2.
and
C n = 1 n X′ nX n → C (3.0.0.3)
1
n max
1≤i≤n x′ ix i → 0 (3.0.0.4)
3.1 Limit in probability
To determine the limit in probability, the following law of large numbers will be needed.
Theorem 3.1.0.12. (Kallenberg, 2002, Corollary 4.22) Let ξ 1 , ξ 2 , . . . be independent random
variables with mean 0 satisfying
∞∑
n −2c E(ξi 2 ) < ∞
i=1
21

22 Application to the Lasso estimator
for some c > 0.Then
almost surely.
1
n
n∑
ξ i → 0
i=1
Theorem 3.1.0.13. (Convergence in probability) If C is nonsingular and λ n /n →
λ 0 ≥ 0, then
where
ˆβ n → P arg min(Z)
p∑
Z(φ) = (φ − β) ′ C(φ − β) ′ + λ 0 |φ j |.
j=1
Thus if λ n = o(n), arg min(Z) = β and so ˆβ n is consistent.
Proof. Maps Z n defined in 3.0.0.1 have convex sample paths and are minimized at ˆβ n .
Sample paths of Z are strictly convex since C is nonsingular, hence have unique minimizers.
Following Corollary 2.1.0.3, it is sufficient to show that Z n (φ) → P Z(φ) + σ 2 for every
point φ ∈ R p .
We have
Set ξ i = ε i (β − φ) ′ x i with
Z n (φ) = 1 n∑
(Y i − x ′
n
iφ) 2 + λ n
p∑
|φ j |
n
i=1
j=1
= 1 n∑
(x ′
n
iβ + ε i − x ′ iφ) 2 + λ n
p∑
|φ j |
n
i=1
j=1
( )
1
n∑
= (β − φ) ′ x i x ′ i (β − φ) + 1 n∑
ε i
n
n
i=1
i=1
+ 2 n∑
ε i (β − φ) ′ x i + λ n
p∑
|φ j |
n
n
i=1
j=1
i=1
.
E(ξ 2 i ) = σ 2 |〈β − φ, x i 〉| 2
≤ σ 2 ‖(β − φ)‖ 2 ‖x i ‖ 2 .
Under assumption 3.0.0.4 i −1 ‖x i ‖ 2 is asymptotically bounded above by i −δ for some δ > 0,
thus we obtain
∞∑ 1
∞
i 2 E(ξ2 i ) ≤ σ 2 ‖(β − φ)‖ 2 ∑
( ) 1 2
i ‖x i‖ < ∞.
Now it follows from theorem 3.1.0.12 that
2
n
i=1
n∑
ε i (β − φ) ′ x i → 0
i=1

3.2 Limit in distribution 23
almost surely. Finally we have
Z n (φ) → P (φ − β) ′ C(φ − β) ′ + σ 2 + λ 0
as claimed.
p∑
|φ j |.
j=1
3.2 Limit in distribution
Asymptotic normality of independent, but not necessarily identically distributed random
variables is usually proved by verifying the so called Lindeberg-Feller condition.
Theorem 3.2.0.14. (Van der Vaart, 2000, Proposition 2.27) For each n ∈ N, let
ξ n,1 , . . . , ξ n,kn
be independent random vectors with finite variances Σ n,1 , . . . , Σ n,kn
such that
k n ∑
i=1
(
)
E ‖ξ n,i ‖ 2 1 {‖ξn,i ‖>ε} → 0 for every ε > 0
and
k n ∑
i=1
Σ n,i → Σ
Then
k n ∑
i=1
(
)
ξ n,i − E(ξ n,i ) N (0, Σ)
The Lindeberg-Feller condition will also be used later to prove a partial asymptotic normality
result for the adaptive Lasso.
Theorem 3.2.0.15. (Convergence in distribution) If λ n / √ n → λ 0 ≥ 0 and C is
nonsingular then
√ n( ˆβ − β) → arg min(V )
where
V (u) = −2u ′ W + u ′ Cu + λ 0
for W ∼ N (0, σ 2 C).
p∑
(sgn(β j )I(β j ≠ 0) + |u j |I(β j = 0))
j=1
Proof. Define V n : R p → R as
V n (u) =
n∑
i=1
(
(ε 2 i − u ′ x i / √ ) p∑
n) 2 − ε 2 (
i + λ n |βj + u j / √ n| − |β j | ) ,
j=1

24 Application to the Lasso estimator
a convex function minimized at √ ( )
n ˆβ n − β . We have
n∑
i=1
(
(ε 2 i − u ′ x i / √ )
n) 2 − ε 2 i =
n∑
( 1
n u′ x i x ′ iu − 2ε i u ′ x i / √ n)
i=1
= 1 ( n
)
∑
n∑
n u′ x i x ′ i u − −2ε i u ′ x i / √ n
i=1
i=1
Set ξ i = ε i x i / √ n, under the assumption 3.0.0.4, it holds that
n∑
i=1
(
)
E ‖ξ i ‖ 2 1 {‖ξi ‖>ε} =
=
≤
n∑
i=1
n∑
i=1
n∑
i=1
(
)
E ‖ξ i ‖ 2 1{‖ξ i ‖ > ε}
1 (
n ‖x i‖ 2 E |ε i | 2 1{|x i ‖|ε i |/ √ )
n > ε}
1
n ‖x i‖ 2 E
= tr(C n ) E
(
|ε i | 2 1{|ε i | max ‖x i‖/ √ )
n > ε}
1≤i≤n
(
|ε 1 | 2 1{|ε 1 | max
1≤i≤n ‖x i||/ √ n > ε}
This converges to zero by Lebesgue’s dominated convergence theorem since tr(C n ) = p.
The Lindeberg-Feller theorem 3.2.0.14 now implies that
n∑
i=1
(
(ε 2 i − u ′ x i / √ )
n) 2 − ε 2 i −2u ′ Wu + u ′ Cu
with W ∼ N (0, C) for every u ∈ R p .
On the other hand we have
p∑ ( |βj + u j / √ n| − |β j | ) p∑
→ λ 0 (sgn(β j )I(β j ≠ 0) + |u j |I(β j = 0)) .
λ n
j=1
j=1
Now weak convergence of the marginals of V n to the marginals of V , both viewed as random
maps into R p follows directly from the Cramer-Wold device. Since C is nonsingular the
process V is strictly convex, hence takes values in C min (R p ). In view of the Argmin
continuous mapping theorem 2.3.0.11, this completes the proof.
)
.
3.2.1 Limiting distribution of components
In this section, we further investigate the asymptotic behavior in distribution of single
components the root √ n( ˆβ n − β), our goal being the construction of confidence intervals
and hypothesis tests for individual coefficients.
In the next proposition, we show that, in the limit, the subvector of the root corresponding
to zero parameters can take value zero with positive probability, this probability is
quantified. Also, we show that the limiting distribution of each component has at most a
discontinuity at the point zero, that it is otherwise Gaussian.
For notational convenience, in the remaining, we assume without loss of generality that
β 1 , . . . , β r are nonzero and that β r+1 = · · · = β p = 0, this yields
⎛
⎞
r∑
p∑
V (u) = −2u ′ W + 2u ′ Cu + λ 0 ⎝ sgn(β j )u j + |u j | ⎠ .
j=1
j=r+1

3.2 Limit in distribution 25
We also adopt the following partitioning of C, W, and u:
( )
C11 C
C =
12
,
C 21 C 22
where C 11 is a r × r matrix, C 22 is a (p − r) × (p − r) matrix and C 12 = C ′ 21 ;
W =
( )
W1
;
W 2
u =
(
u1
)
,
u 2
where W 1 and u 1 are r-vectors. Finally, denote û = arg min u V (u).
Proposition 3.2.1.1. If λ n / √ n → λ 0 ≥ 0 and C is nonsingular then
(i)
P (û 2 = 0) = P
(
− λ 0
2 1 ≤ C 21C −1
11 (W 1 − λ o sgn(β))/2 − W 2 ≤ λ 0
2 1 )
where W 1 ∼ N (0, σ 2 C 11 ), W 2 ∼ N (0, σ 2 C 22 ) and the inequality is to be interpreted
componentwise.
(ii) û 1 has a Gaussian distribution and the distribution of every component of û 2 is the
mixture of a Gaussian distribution with a point mass at zero.
Proof. The subdifferential ∂V (û) at û contains 0, i.e.
− 2
= 0
( ) ( ) ( ) ⎛
W1 C11 C
+ 2
12 û1
+ λ
W 2 C 21 C 22 û 0 ⎝
2
sgn(β)
(sgn(û j 2 )I(ûj 2 ) ≠ 0 + e jI(û j 2 ) ) p−r
for some values e j ∈ [0, 1], j = 1, . . . , p − r. In particular, the following relations between
C, W, û 1 and û 2 always hold :
û 1 = C −1
11 (W 1 − C 12 û 2 − λ 0 sgn(β)/2)
j=1
⎞
⎠
Together, they imply
− λ 0
2 1 ≤ −W 2 + C 21 û 1 + C 22 û 2 ≤ λ 0
2 1.
− λ (
)
0
2 1 ≤ −W 2 + C 21 C −1
11 (W 1 − λ 0 sgn(β)) + C 22 − C 21 C −1
11 C 12 û 2 ≤ λ 0
2 1 (3.2.1.1)
Next, suppose that û 2 = 0. Then it follows readily from 3.2.1.1 that
− λ 0
2 1 ≤ −W 2 + C 21 C −1
11 (W 1 − λ 0 sgn(β)) ≤ λ 0
2 1 (3.2.1.2)

26 Application to the Lasso estimator
Conversely, suppose that 3.2.1.2 holds, we show that the minimizer necessarily satisfies
û 2 = 0. Indeed, if û 2 ≠ 0 then the equality is attained in 3.2.1.2 at every line corresponding
to nonzero components of û 2 , that is,
⎧
(
(
) ) ⎪⎨
−W 2 + C 21 C −1
11 (W 1 − λ 0 sgn(β)) + C 22 − C 21 C −1
11 C 12 û 2 = − λ 0
2 , if ûj 2 > 0
j ⎪ ⎩
λ 0
2 , if ûj 2 < 0
In particular, it follows from 3.2.1.2 that
{
((
) )
C 22 − C 21 C −1
11 C 12 û 2 ∈ [−λ0 , 0], if û j 2 > 0
j [0, λ 0 ] if û j 2 < 0 (3.2.1.3)
Note that (C 22 − C 21 C −1
11 C 12), the Schur complement of C, is SPD since C is. Let
D be the matrix which results from (C 22 − C 21 C −1
11 C 12) after removal of the lines and
columns corresponding to the zero components of û 2 , D is also SPD. Let û≠0 2 be the vector
which results from û 2 after removal of the zero components. Then 3.2.1.3 implies that
û≠0 T
2 Dû≠0 2 ≤ 0 which contradicts the fact that D is SPD, this complete the argument.
Finally, we show that every component of the limit distribution has a discontinuity at the
point zero only, and is otherwise Gaussian. Consider after an eventual permutation of the
last p − r covariates a further partitioning of W 2 , u 2 and C of the form
( )
W2,1
W 2 =
W 2,0
u 2 =
(
u2,1
u 2,0
)
,
and
C =
⎛
⎜
⎝
⎞
C 11 C 12 C 13
⎟
C 21 C 22 C 23 ⎠
C 31 C 32 C 33
respectively, where W 2,1 and u 2,1 are r ′ -vectors for some 0 < r ′ < p − r. For notational
convenience, denote
( )
C11 C ˜C = 12
.
C 21 C 22
Next, consider an arbitrary sign pattern s ′ ∈ {−1, +1} r′ . Then, given the event
{ sgn(u2,1 ) = s ′ ; u 2,0 = 0 } ,
(u 1 , u 2,1 ) ′ has the following distribution :
( ) (
( ))
u1
∼ N σ 2 λ 0 ˜C, −
u 2,1 2 ˜C −1 sgn(β)
s ′
This completes the proof.

3.2 Limit in distribution 27
3.2.2 Uniform convergence in the orthogonal case
More can be said about the convergence in distribution of single components of the Lasso
estimator when the matrix X n is orthogonal. We show that in this case, under the assumptions
of Theorem 3.2.0.15, the finite sample distribution of each component converges
uniformly either to a normal distribution or to the mixture of a normal ditribution with a
point mass at zero, depending thereon, whether the corresponding parameter is equal to
zero or not.
Assumption A. X n is an othogonal matrix for every n ∈ N. In particular C n = 1 n X′ nX n
is diagonal.
The following generalization of Polya’s theorem will be needed, it allows for discontinuities
in the limiting distribution, provided that jump heights in the finite sample distribution
converge precisely to the jump height in the limit,
Theorem 3.2.2.1. (Chow and Teicher, 1978, Chapter 8, Lemma 3)
Let {F n } n , F be probability distribution functions. If F n F and ∆F n (x 0 ) → ∆F (x 0 ) at
every discontinuity point x 0 of F , then F n onverges uniformly to F , that is
sup |F n (x) − F (x)| → 0
x∈R
In the following, let J n,j (·) denote the probability distribution function of the j-th component
of
√ n( ˆβ n − β)
and let J j (·) denote the probability distribution function of the j-th component of
V (·) as in theorem 3.2.0.15.
arg min V (u),
u∈R p
Theorem 3.2.2.2. Assume that assumption A holds. Then
for every j ∈ {1, . . . , p}.
sup |J n,j (x) − J j (x)| → 0
x∈R
Proof. Proposition 3.2.1.1 establishes the continuity of J j (·) for every j with β j ≠ 0, it
also states that for j with β j ≠ 0, there is a discontinuity at the point zero only. Hence,
in view of Polya’s theorem 3.2.2.1, it is sufficient to show that for such j
holds.
Define
∆J n,j (0) → ∆J j (0) (3.2.2.1)
û n = √ n( ˆβ − β) = arg min V n (û).
u∈R p

28 Application to the Lasso estimator
From subdifferential calculus, it follows that the subdifferential ∂V n (û n ) of V n at û n contains
the point zero, that is,
− 2
n∑ (
εi − û ′ nx i / √ n ) x i / √ n
i=1
(
+ λ n sgn(βj + û j / √ n)1/ √ n1(β j + û j / √ n ≠ 0) + e j / √ n1(β j + û j / √ n = 0) ) n
n∑
= −2 ε i x i / √ ( )
1
n∑
n + 2 x i x ′ i û n
n
i=1
i=1
(
+ λ n sgn(βj + û j / √ n)1/ √ n1(β j + û j / √ n ≠ 0) + e j / √ n1(β j + û j / √ n = 0) ) n
n∑
= −2 ε i x i / √ n + 2 C n û n
i=1
+ λ n
( sgn(βj + û j / √ n)1/ √ n1(β j + û j / √ n ≠ 0) + e j / √ n1(β j + û j / √ n = 0) ) n
j=1
= 0
for some values e j ∈ [−1, 1], j = 1, . . . , p. Assumption A allows for decoupling of the
components of û n , so writing w n,j = β j + û j / √ n we obtain :
(
û j n = (C n ) −1 ∑ n
jj ε i x ij / √ n − λ n
2 √ n (sgn(w n,j)/ √ n1(w n,j ≠ 0) + e j / √ )
n1(w n,j = 0)) .
i=1
In particular,
where
− λ n
2 √ n ≤ (C n) −1
jj û j n −
n∑
ε i x ij / √ n ≤
i=1
n∑
ε i x ij / √ n N (0, σ 2 C jj ).
i=1
λ n
2 √ n
One easily verifies that for j ∈ {r + 1, . . . , p}, û j n = 0 if and only if
− λ n
n
2 √ n ≤ ∑
ε i x ij / √ n ≤
i=1
λ n
2 √ n
j=1
j=1
(3.2.2.2)
since an equality is attained in the inequality 3.2.2.2 if û j n is non-zero and (C n ) jj is
positive. Since the distribution of the errors is assumed continuous, the Portmanteau
theorem implies that
∆J n,j (0) → ∆J j (0)
for j ∈ {r + 1, . . . , p}. This completes the proof.
Remark. This result on uniform convergence toward the limit will be key to justify the use
of subsampling to construct asymptotically valid confidence interval. Altough we proved
it under the orthogonality assumption A only, we conjecture that it holds for a larger
class of symmetric positive definite matrices, if not for all such matrices. However, the
argument seems to be tedious in greater generality. This fact can be illustrated by mean
of Monte-Carlo simulations as shown in Figure 3.1 for a linear model with parameter β =
(1.5, −1.5, 0.75, −1.5, 1.5, −3, 0) ′ ∈ R 20 , error ε i = N (0, √ 20) and covariates multivariate
normal with mean zero and Toeplitz covariance matrix with parameter ρ = 0.99.

3.2 Limit in distribution 29
Figure 3.1: Monte Carlo estimates of the distribution of the root √ n( ˆβ j −β j ), j = 7, . . . , 10,
with penalization parameter λ n = 2 √ n.

30 Application to the Lasso estimator

Chapter 4
The adaptive Lasso in a high
dimensional setting
In a high dimensional setting, where the use of the Lasso is most justified, due to its
sparsity inducing property, there are to this date no asymptotic results similar to those of
Knight and Fu (2000). Hence we turn to a variant, the adaptive Lasso and presents the
results of Huang et al. (2008) who studied it under saprsity asssumptions and a further a
partial orthogonality assumption between relevant and noise covariates. Their approach
offers the advantage to provide an asymptotic normality result for estimates corresponding
to nonzero-coefficients.
One typically resorts to a triangular array to model high dimensionality in linear models,
that is, one assumes that
Y i = x ′ iβ n0 + ε i , i = 1, . . . , n (4.0.2.1)
where the parameter β n0 has a dimension p n allowed to grow faster than n. For observations
(Y i , x i ), i = 1, . . . , n drawn from 4.0.2.1, the adaptive Lasso is defined as
arg min
φ∈R p
L n ( φ) = arg min
φ∈R p
n ∑
i=1
(Y i − x ′ iφ) 2 ∑p n
+ λ n w nj |φ j | (4.0.2.2)
where λ n is the usual penalty parameter and are {w nj } j are in general strictly positive
weights. However, here we make the assumption that they are computed using an initial
estimator ˜β nj of β nj , that is :
j=1
w nj = | ˜β nj | −1 , (4.0.2.3)
for j = 1, · · · , n. For notational convenience, in the remaining, we drop the subscript n
for β n0 , yet dependence on n is implicitely assumed.
Next, we introduce further notation.
First, assume without loss of generality that β 0 takes the form
β 0 = (β ′ 1, β ′ 2) ′
31

32 The adaptive Lasso in a high dimensional setting
where β 1 is a k n × 1 vector, has each of its components different from zero and β 2 is a
m n × 1 vector equal to zero. Let x i = (x i1 , . . . , x ipn ) ′ be the covariate vector of the i-th
observation, denote by u i its first part, corresponding to non-zero coefficients, and by z i
the part corresponding to zero coefficients, i.e.
The empirical covariance matrix
x i = (u ′ i, z ′ i) ′ .
C n = 1 n X′ nX n
is also partitioned according to zero and non-zero coefficients. For
X ′ n1 = (u 1 , · · · , u kn ) ′ ,
set
C n1 = 1 n X′ n1X n1 ,
a k n ×k n matrix. Finally let ρ n1 and τ n1 be the smallest respectively the largest eigenvalue
of τ n1 of C n1 .
Results on the variable selection consistency and the partial normality of the adaptive
Lasso will be derived under the following assumptions:
Assumption B.
B.1 {ε i } i is a sequence of i.i.d. random variables with mean zero andc there are some
constants 1 ≤ d ≤ 2, C > 0 and K > 0, such that the tail probability satisfies
for every x ≥ 0.
P (|ε i | > x) ≤ K exp
(−Cx d)
B.2 The initial estimators ˜β nj are r n -consistent for the estimation of unknown constants
η nj depending on β, that is
r n
The constants η nj satisfy
∣ ∣ ∣∣ ∣∣
max ˜βnj − η nj = OP (1), r n → ∞.
1≤j≤p n
max
k n

33
B.3 (Adaptive irrepresentable condition) For s n1 defined as
s n1 =
(
) ′
|η n1 | −1 sgn(β 01 ), . . . , |η nkn | −1 sgn(β 0kn )
and some constant κ < 1, it holds that
∣
∣x ′ j X n1C −1
n
n1 s n1∣
≤
κ
|η nj | ,
for every j ∈ {k n + 1, . . . , p n }.
B.4 The constants k n , m n , λ n , M n1 , M n2 and b n1 satisfy
(
log(n) 1{d=1} n −1/2 (log(k n)) 1/d
(
+ n 1/2 λ −1
n (log(m n )) 1/d M n2 + 1 ) ) + M n1λ n
b n1
r n b n1 n → 0
B.5 There is a constant τ 1 > 0 such that τ n1 ≥ τ 1 for all n.
B.6
n −1/2 max
1≤i≤n u′ iu i → 0
Next, we introduce the Orlicz norm and some of its properties which be used later to
bound tail probabilities.
Definition 4.0.2.3. (Orlicz norm) Let ψ d = exp(x d )−1 for d ≥ 1. The ψ d -Orlicz norm
||X|| ψd of a random variable X is defined as
{
( ( ))
∣ |X|
||X|| ψd = inf C > 0∣E
ψ d
C
}
≤ 1
Lemma 4.0.2.4. (Van der Vaart and Wellner, 1996, Lemma 2.2.1) Let X be a random
variable with P (|X| > x) ≤ K exp(−Cx d ) for every x, for constants K and C and for
d ≥ 1. Then its Orlicz norm sasitfies
||X|| ψd ≤ ((1 + K)/C) 1/d
In the next Lemma, let ‖X‖ P,d denote the d-moment of a random variable X and S n the
(partial) sum of the first n random variables in a sequence.
Lemma 4.0.2.5. (Van der Vaart and Wellner, 1996, Proposition A.1.6) Let X 1 , . . . , X n
be independent, mean zero random variables indexed by an arbitrary index set T . Then
(i)
(ii)
‖S n ‖ P,d ≤ K
d
[
∥ ]
‖S n ‖ P,1 +
log(d)
∥ max ∥ ∥ ∥∥∥P,d
∥X i , (d > 1).
1≤i≤n
∥ ]
‖S n ‖ ψd ≤ K p
[‖S n ‖ P,1 +
∥ max ∥ ∥ ∥∥∥ψd
∥X i , (0 < d ≤ 1).
1≤i≤n

34 The adaptive Lasso in a high dimensional setting
(iii)
‖S n ‖ ψ,d ≤ K d
⎡
⎣‖S n ‖ P,1 +
( n ∑
i=1
∥ ∥ ) ⎤ 1/d ′
∥∥ ∥X i d ′
⎦ , (1 < d ≤ 2).
ψ d
Here, 1/d + 1/d ′ = 1, K is a universal constant and K d is a constant depending on d only.
Lemma 4.0.2.6. (Van der Vaart and Wellner, 1996, Lemma 2.2.2) Let ψ be a convex,
nondecreasing, nonzero function with ψ(0) = 0 and
lim sup ψ(x)ψ(y)/ψ(cxy) < ∞ (4.0.2.4)
x,y→∞
for some constant c. Then for arbitrary random variables X 1 , . . . , X m , it holds that
for a constant K depending on ψ only.
∥ max X i
1≤i≤m ∥ ≤ Kψ −1 (m) max ‖X i‖
ψ 1≤i≤m
Lemma 4.0.2.7. Let {ε i } i be a sequence of i.i.d. random variables with mean zero and
finite variance. Suppose that their tail probability satisfies
P (|ε i > x|) ≤ K exp(−Cx d ),
i ∈ N
for constants C and K, and for 1 ≤ d ≤ 2. Then for all constants a i with ∑ n
i=1 a 2 i
holds that
∥ n∑ ∥∥∥∥ψd a
∥ i ε i ≤
i=1
{ (
Kd σ + (1 + K) 1/d C −1/d) , 1 < d ≤ 2,
K 1 (σ + (1 + K)C log(n)) , d = 1
= 1, it
where K d is a constant depending on d only. Consequently,
q ∗ n(t) =
sup P
a 2 1 +···+a2 n =1
(
∑ n
)
⎧
⎪⎨
a i ε i > t ≤
i=1
⎪⎩
( )
exp − td
, 1 < d ≤ 2
M
(
)
t
exp −
, d = 1,
M(1 + log(n))
for some constant M depending on d, K and C only.
Proof. {ε i } i satisfies P (|ε i | > x) ≤ K exp(−Cx d ), so it follows from 4.0.2.4 that
‖ε i ‖ ψd ≤ [(1 + K)/C] 1/d .
Let d ′ be the conjugate of d, that is 1/d + 1/d ′ = 1. Setting X i = a i ε i , by 4.0.2.7, there

35
exists a constant K d depending on d only such that
∥ ⎡ (∣ ∥∥∥∥ψd ∣∣∣∣ ∑
a i ε i ≤ K d
⎣E
n [
∑ n
n∑
∥
i=1
= K d
⎡
⎣E
≤ K d
⎡
⎢
≤ K d
⎡
⎣E
∣) ∣∣∣∣
a i ε i +
i=1
(∣( ∣∣∣∣ ∑ n
)
a i ε i · 1
∣
i=1
⎛( ∑ n
⎣E ⎝
(( n ∑
) ⎞ 2
a i ε i ⎠
i=1
a 2 i ε 2 i
i=1
] ⎤ 1/d ′
‖a i ε i ‖ d′ ⎦
ψ d
i=1
) [
∣ ∑ n
1/2
+
] ⎤ 1/d ′
|a i | d′ ‖ε i ‖ d′ ⎦
ψ d
i=1
[ n
] ⎤
+ (1 + K) 1/d C −1/d ∑ 1/d ′
|a i | d′ ⎥ ⎦
i=1
)) 1/2
+ (1 + K) 1/d C −1/d [ n ∑
⎡
[ n
] ⎤
=≤ K d
⎣σ + (1 + K) 1/d C −1/d ∑ 1/d ′
|a i | d′ ⎦
i=1
] ⎤ 1/d ′
|a i | d′ ⎦
i=1
Here, Hölder’s inequality and Lemma 4.0.2.4 have been used in the second inequality. For
1 < d ≤ 2, d ′ = d/(d − 1) ≥ 2, which implies
It follows that
(
n∑
n
)
∑ d ′ /2
|a i | d′ ≤ |a i | 2 = 1
i=1
i=1
∥ n∑ ∥∥∥∥ψd (
a
∥ i ε i ≤ K d σ + (1 + K) 1/d C −1/d)
i=1
For d = 1, by Lemma 4.0.2.7, there exists some constant K 1 such that
∥ (∣ n∑ ∥∥∥∥ψ1 ∣∣∣∣ a
∥ i ε i ≤ K 1
[E
n ∣) ]
∑ ∣∣∣∣
a i ε i + ‖ max |a iε i |‖ ψ1
1≤i≤n
i=1
i=1
]
≤ K 1
[σ + K ′ log(n) max ‖a iε i ‖ ψ1
1≤i≤n
]
≤ K 1
[σ + K ′ (1 + K)C −1 log(n) max |a i|
1≤i≤n
]
≤ K 1
[σ + K ′ (1 + K)C −1 log(n)
where Hölder’s inequality and Lemma 4.0.2.6 have been used in the second inequality.
Finally, note that an arbitrary random variable X, the following holds
P (X > t‖X‖ ψd ) ≤ (1 + ψ d (t)) −1 (1 + E (ψ d (|X|/‖X‖ ψd )))
≤ 2 exp(−t d )
for all t > 0, by Markov’s inequality and by definition of the ψ d -Orlicz norm.
(
Lemma 4.0.2.8. Let ˜s n1 = | ˜β
′
nj | −1 sgn(β 0j ))
and s n1 = ( | η˜
nj | −1 sgn(β 0j ) ) ′
j∈J j∈J n1
.
n1
Suppose that assumption B.2 holds. Then,
‖˜s n1 ‖ = (1 + o P (1)) M n1

36 The adaptive Lasso in a high dimensional setting
and
∥ ∥ ∥∥| ∥∥
max ˜βnj |˜s n1 − |η nj |s n1 = oP (1)
j /∈J n1
Proof. By assumption B.2, we have
∣∣ ∣ ∣∣∣∣ ∣∣∣∣ ˜βnj ∣∣∣∣ max − 1∣
1≤j≤k n η nj
∣ =
∣ ∣ ∣∣∣∣ max 1 ∣∣∣∣ ∣ ∣∣| ˜βnj | − |η nj | ∣ ≤ O P (1/r n )M n1 = o P (1)
1≤j≤k n η nj
∣∣ ∣ ∣∣∣∣ ∣∣∣∣ η ∣∣∣∣ nj
We also have max 1≤j≤kn − 1
˜β nj
∣ = o P (1). Indeed, note that
1
| ˜β nj | = 1
| ˜β nj | − |η nj | + |η nj | ≤ 1
|η nj |
1
(
∣
∣∣
∣1 + ˜βnj /η nj − 1)∣
≤ M n1
1
|1 + o P (1)| = M n1O P (1) = o P (r n ),
for every 1 ≤ j ≤ k n , it follows that
∣ ∣∣∣∣ η nj
max − 1
1≤j≤k n ˜β nj
∣ = max 1/| ˜β ∣ nj | ∣|η nj | − | ˜β nj | ∣
1≤j≤k n
( ∣ ∣∣|ηnj
≤ o P (r n ) max | − | ˜β )
nj | ∣
1≤j≤k n
≤ o P (r n )o P (1/r n ) = o P (1)
Now we can prove the first part of the claim,
‖˜s n1 ‖ = √
=
≤
k n ∑
1
j=1
˜β nj
2 ∑k n
√
j=1
∑k n
√
j=1
( (
1
1 + |η )) 2
nj|
|η nj | |β nj | − 1
( ) 2
1
|η nj | (1 + o P (1))
≤ (1 + o P (1)) √
k n ∑
j=1
≤ (1 + o P (1)) M n1
1
|η nj | 2

4.1 Variable selection consistency 37
For the second part of the claim, first note that
max ‖|η nj |˜s n1 − |η nj |s n1 ‖ 2 ∑k n ∣
= max ∣|η nj |/| ˜β ni | − |η nj |/|η ni | ∣ 2 ∑k n ≤ M 2 |η nj | − | ˜β ni |
n2
j>k n j>k n
∣
i=1
i=1
| ˜β nj ||η ni | ∣
(
) 2
≤ max || η nj | − | ˜β
k n 2
∑
nj ||
M n2
1≤j≤k n ∣|η ni |
(1 2 + | ˜β
)
ni |/|η ni | − 1 ∣
(
=
max || η nj | − | ˜β nj ||
1≤j≤k n
i=1
) 2 ∑k n ∣
i=1
= o P (1/r 2 n) O P (1) 2 M 2 n1 = o P (1).
M n2
|η ni | 2 (1 + o P (1)) ∣
Next, we have
∣ ∣∣| max ˜βnj | − |η nj | ∣ ‖˜s n1 ‖ = o P (1/r n )(1 + o P (1))M n1 = o P (1)
j>k n
The second part of the claim now follows from the triangle inequality.
2
2
4.1 Variable selection consistency
Theorem 4.1.0.9. (Huang et al., 2008, Theorem 1)(Variable selection consistency
of the adaptive Lasso) Suppose that assumptions B.1 to B.5 are valid, then
( )
P ˆβn = s β 0 → 1.
Proof. From subdifferential calculus, we know that a vector ˆβ n ∈ R pn is a solution to the
adaptive Lasso problem if the the subdifferential ∂L n ( ˆβ n ) of the objective function at ˆβ n
contains the point zero, that is
{ x
′ j (y − X ˆβ n ) = λ n w nj sgn( ˆβ nj ), if ˆβ nj ≠ 0
∣
∣x ′ j (y − X) ˆβ
∣ ∣∣
n < λn w nj , if ˆβ (4.1.0.5)
nj = 0
Furthermore, if the vector family
solution. Define
˜s n1 =
{
x j , ˆβ
}
nj ≠ 0 is linearly independent, then ˆβ n is unique
(
| ˜β n1 | −1 sgn(β 01 ), . . . , | ˜β
) ′
nkn | −1 sgn(β 0kn )
and
ˆβ n1 = ( X ′ n1X n1
) −1 ( X
′
n1 y − λ n˜s n1
)
= β 01 + 1 (
n C−1 n1 X
′ )
1 ε − λ n˜s n1
If ˆβ n1 = s β 01 then 4.1 holds for ˆβ
(
n = ˆβ′ n1, 0 ′) ′
. For this particular ˆβn we have X ˆβ n =
X 1 ˆβ n1 . The family {x j , 1 ≤ j ≤ k n } is then linearly independent, hence X ′ n1 X n1 regular
and we obtain that ˆβ n = s β 0 if
{
∣ ˆβn1 = s β 01
∣x ′ j (y − X n1 ˆβ (4.1.0.6)
n1 ) ∣ < λ n w nj , k n < j ≤ p n

38 The adaptive Lasso in a high dimensional setting
Now, let H n = I n − 1 n X n1C −1
n1 X′ n1 be the projection matrix on the nullspace of X′ 1 . By
definition of ˆβ n1 , we have
(
y − X n1 ˆβn1 = ε + X 1 β 01 − ˆβ
)
n1
= ε + 1 n X n1C −1 (
n1 X
′ )
1 ε − λ n˜s n1
= H n ε + λ n
n X n1C −1
n1 ˜s n1
Following condition 4.1.0.6 we have that ˆβ n = s β 0 if
⎧
⎨ sgn (β 0j )
(β 0j − ˆβ
)
nj < |β 0j | 1 ≤ j ≤ k n
(
)∣
⎩ ∣
∣x ′ j H n ε + λn n X n1C −1
n1 ˜s ∣∣ (4.1.0.7)
n1 < λn w nj , k n < j ≤ p n
Next, denote by e j , the j-th standard unit vector and choose an arbitrary 0 < κ < κ + δ <
1. One verifies that according to condition 4.1.0.7, ˆβn ≠ s β 0 implies the realization of
either of the following events :
B n1 =
B n2 =
B n3 =
B n4 =
k n ⋃
j=1
k n ⋃
j=1
⋃p n
j=k n+1
⋃p n
j=k n+1
{
}
n −1 |e ′ jC −1
n1 X 1ε| ≥ |β 0j |/2
{
}
|e j C −1
n1 ˜s n1| ≥ |β 0j |/2
{∣ ∣∣x ′
j H n ε∣
∣ ≥ (1 − κ − δ)λ n w nj
}
{
n −1 ∣ ∣ ∣x
′
j X 1 C −1
n1 ˜s ∣
n1
}
∣ ≥ (κ + δ) .
We show that they each have probability tending to zero as n → ∞. For B n1 , first note
that the matrix C n1 has root n 1/2 X n1 (X ′ n1 X n1) −1 . Thus we obtain
∥ ∥∥∥ ( ) ∥
n −1 e ′ jCn1 −1 ′ ∥∥∥ X′ 1 = n −1/2 ‖n −1/2 X n1 C −1
n1 e j‖
≤ n −1/2 ‖C −1/2
n1 ‖ ‖e j ‖
≤ (nτ n1 ) −1/2
by the spectral decomposition theorem. It follow that
⎛
⋃k n {
} ⎞
P (B n1 ) = P ⎝ ‖e ′ jC −1
n1 X′ 1ε‖/n ≥ |β 0j |/2 ⎠
≤ k n q ∗ n
1=1
( √ ) bn1 τn1 n
2
with q n (·) given in Lemma 4.0.2.7. Now, assumptions B.1, B.4 and B.5 imply that
P (B n1 ) → 0.
For the set B n2 , note that Lemma 4.0.2.6, then assumptions B.4 and B.5 imply that
λ n ∣
∣e j C −1
n1
n ˜s n1∣ ≤ λ ( )
n‖˜s n1 ‖ λn M n1
= O P = o P (b n1 ).
nτ n1
nτ n1

4.2 Partial asymptotic normality 39
Finally assumption B.5 yields P (B n2 ) = o(1).
For B n3 , first note that
1
w nj
= | ˜β nj | ≤ M n2 + O P (1/r n ), j = k n + 1, . . . , p n
Furthermore, ‖(x j H n ) ′ ‖ ≤ √ n, so it follows that
⎛
⋃p n {
P (B n3 ) ≤ P ⎝ |x ′ jH n ε| ≥ (1 − κ − δ)λ n C −1 (M n2 + 1/r n ) −1}⎞ ⎠ + o P (1)
≤ m n q ∗ n
j=k n+1
((1 − κ − δ)λ n n −1/2 C −1 (M n2 + 1/r n ) −1)
for C large enough. Lemma 4.0.2.7 and assumption B.4 now imply that P (B n3 ) → 0.
Finally, for the set B n4 , recall that ‖x j ‖/n = 1, so Lemma 4.0.2.6 and assumption B.5
together imply
{∣ ∣∣x ′
max j X 1 C −1
k nk n
≤ τ −1/2
n1 o P (1) = o P (1).
∣
Now, by assumption B.3, it holds that ∣η nj x ′ j X 1C −1
0. Thich completes the proof.
∣
∣ /(nw nj ) − ∣η nj x ′ jX 1 C −1
) ∥ ′ ∥∥∥ ∥ }
∥∥| n1 ˜βnj˜s n1 − η nj s n1 | ∥
n1 s ∣
n1
}
n1 s n1∣
∣ ≤ κ, so we indeed obtain P (B n4 ) →
4.2 Partial asymptotic normality
The proof of the following result on partial asymptotic normality builds upon the fact that
estimates of relevant coefficients stay away from zero with high probability as n tends to
infinity. This is indeed a conclusion of Theorem 4.1.0.9 which asserts variable selection
consistency. Note however that this result is too strong for this purpose and a consistency
result for an l q -norm, q > 1 would actually be sufficient.
Theorem 4.2.0.10. (Huang et al., 2008, Theorem 2)(Asymptotic normality for nonzero
coefficients) Suppose that assumptions B.1 to B.6 are valid. For an arbitrary k n ×1-vector
α n with ‖α n ‖ ≤ 1, let
If M n1 λ n n −1/2 → 0, then
)
n 1/2 s −1
n α ′ n
(ˆβn − β 0 = n −1/2 s −1
n
s 2 n = σ 2 α ′ nC −1
n1 α′ n
n∑
ε i α ′ nC ′ n1u i + o P (1) N (0, 1).
i=1
where o P (1) is a term that converges to zero in probability uniformly with respect to α n .
Proof. Under assumptions B.1 to B.5, one has variable selection consistency according
to Theorem 4.1.0.9, in particular ˆβ n1 has no zero component on a set with probability
converging to one. On this set, one has ∂/∂β 1 L n ( ˆβ n1 , ˆβ n2 ) = 0, that is,
−2
n∑
i=1
(y i − u i ˆβn1 − z i ˆβn2
)
u i + 2λ n ψ n = 0 (4.2.0.8)

40 The adaptive Lasso in a high dimensional setting
where ψ n =
written as
(
w nj sgn( ˆβ
)
n1j ) . Since β n2 = 0 and ε i = Y i − u i β 10 , this can be
1≤j≤k n
n∑
i=1
(ε i − u i ( ˆβ n1 − β 01 ) − z i ˆβn2
)
u i + λ n ψ n = 0
Using C n1 = n −1 X ′ n1 X n1, we first obtain
then
n 1/2 α ′ n
( )
C n1 ˆβn1 − β 10 = 1 n
( ˆβ n1 − β 10
)
= n −1/2 n ∑
i=1
n∑
i=1
ε i u i − λ n
n ψ n − 1 n
n∑
z ′ ˆβ i n2 w i ,
i=1
ε i α ′ nC −1
n1 u i − n −1/2 λ n α ′ nC −1
n1 ψ n − n −1/2
n ∑
i=1
z ′ i ˆβ n2 u i .
By Theorem 4.1.0.9, the set { ˆβ n2 = 0} has probability tending to one, thus the last term
converges to zero in probability. By the spectral decomposition Theorem and assumption
B.5 one has
‖C −1
n1 ‖ = τ n1
−1 ≤ τ 1 −1 .
Cauchy-Schwarz inequality then yields
∣
∣n −1/2 λ n α ′ nC −1
∣ ≤ n −1/2 λ n ‖α ′ n‖ ‖C −1
n1 ‖ ‖˜s n1‖
n1 ψ n
≤ n −1/2 λ n τ −1
1 (1 + o P (1))M n1
= o P (1)
Here we used Lemma 4.0.2.8 in the first inequality and the assumption n −1/2 λ n M n1 → 0
in the equality. So we obtain as claimed
n 1/2 s −1
n
= n −1/2 s −1
n
n∑
i=1
ε i α ′ nC −1
n1 u i + o P (1).
To prove asymptotic normality, we verify that the Lindeberg-Feller condition holds. Set
and
v i = n −1/2 s −1
n α n C −1
n1 x n1
ξ i = ε i v i .
Then we have
var
( n ∑
)
ξ i
i=1
= σ 2 s −2
n α nC ′ −1
n1 α n = 1.
For arbitrary δ > 0,
n∑
i=1
(
E ξi 2 1{|ξ| > δ}
)
= σ 2 n ∑
i=1
(
)
vi 2 E ε 2 i 1{|ε i v i | > δ}

4.3 Marginal regressors as initial estimates 41
holds, and one easily verifies that σ 2 ∑ n
i=1 vi 2 = 1. So by the Lebesgue’s dominated convergence
theorem it is sufficient to show that
max |v i| = max
1≤i≤n i≤i≤n n−1/2 s −1 ∣
n ∣α nC ′ −1
n1 u i∣ → 0.
This claim follows from assumptions B.5 and B.6 as
max
i≤i≤n n−1/2 s −1 ∣
n
∣α ′ nC −1
n1 u ∣
i
∣ ≤ max
1≤i≤n n−1/2 s −1
n
≤= σ −1 n −1/2 max
1≤i≤n
( )
α ′ nC −1 1/2 ( )
n1 α′ n u ′ iC −1 1/2
n1 u i
( )
u ′ iC −1 1/2
n1 u i
≤ n −1/2 σ −1 ‖C −1
n1 ‖ max
1≤i≤n (u′ iu i ) 1/2
≤ σ −1 τ −1
1 n−1/2 max
1≤i≤n (u′ iu i ) 1/2 → 0.
4.3 Marginal regressors as initial estimates
The validity of both Theorem 4.1.0.9 and 4.2.0.10 requires the existence of an initial
estimator ˜β n which satisfies assumption B.2, that is, is r n -consistent for the estimation
of a proxy η n of the true parameters β 0 . In this section we show that under a weak
correlation assumption between relavant and noise variables, this assumption is satisfied
by marginal regressors which also offer the advantage to be computationally attrative.
The marginal regressor ˜β n is defined as
and the proxies η nj are chosen as
˜β nj = x′ j Y
n , j = 1, . . . , p n (4.3.0.9)
η nj = E( ˜β nj ) = x′ j Xβ 0
n
(4.3.0.10)
The weak correlation assumption is formally stated as:
Assumption C.
C.1 Condition B.1 holds.
C.2 For 1 ≤ j ≤ k n and k n < k ≤ p n , it holds that
∣ 1
n∑ ∣∣∣∣ ∣ x
∣ ij x ik =
1 ∣∣∣
n
∣n x jx k ≤ ρ n , 1 ≤ j ≤ k n , k n < k ≤ p n
i=1
with ρ n satisfying
for some 0 < κ < 1.
(
) ⎛
c n = max |η nj | ⎝
k n

42 The adaptive Lasso in a high dimensional setting
C.3 The minimum ˜b n1 = min 1≤j≤kn |η nj | satisfies
for
r n =
kn
1/2 (1 + c n )˜b n1 rn −1 → 0
n 1/2
log(m n ) 1/d log(n) 1{d=1}
Theorem 4.3.0.11. (Huang et al., 2008, Theorem 3) Suppose that assumptions C.1 to
C.3 hold. Then β nj defined in 4.3.0.9 is r n consistent for η nj defined in 4.3.0.10 and the
adaptive irrepresentable condition holds.
Proof. Let
Then
µ 0 = E(Y) =
˜β nj = x′ j Y
n
p n
∑
j=1
x j β 0j .
= η nj + x′ j ε
n
where η nj = x ′ j µ 0/n. The covariates are assumed to be standardized, that is ‖x j ‖/n = 1,
so for arbitrary δ > 0, Lemma 4.0.2.7 implies that
( {
P r n max | ˜β
} ) ( { } )
nj − η nj | > δ = P r n max |x ′ jε|/n > δ
1≤j≤k n 1≤j≤k n
( )
≤ p n qn
∗ n 1/2 rn
−1 δ = o P (1).
Here, r n log(p n ) log(n) 1{d=1} n 1/2 = o P (1) have been used in the last equality. This proves
the first part of assumption B.2. The second part follows from assumption C.3:
(
∑k n 1
j=1
For asumption B.3, note that
η 2 nj
+ M 2 n2
η 4 nj
)
≤ k n
˜b2 nj
(1 + c 2 n) = o(r 2 n).
and
∥
∥X ′ ∥
n1x j 2 ∑k n
( = x
′ ) 2
i x j ≤ kn n 2 ρ 2 n
i=1
|η nj |‖s n1 ‖ ≤ kn
1/2 c n
for every k n < j ≤ p n . Now, assumption C.2 yields
n −1 ∣
|η nj | ∣x ′ jX n1 C −1
n1 s n1∣ ≤ τn1 −1 c nk n ρ n .
for such j. This completes the proof.
The general message of Theorem 4.3.0.11 is that marginals regressors can be used as initial
estimates if correlations between relevant and noise variables are believed to be weak.

Chapter 5
Subsampling
In his seminal paper already, Tibshirani (1996) noted that bootstrap variance estimates
for the lasso could take value zero. Knight and Fu (2000) showed by heuristical means
that residual bootstrap estimates of the lasso distribution are inconsistent, formal arguments
on this inconsistency were finally given by Chatterjee and Lahiri (2010) who proved
that the conditional residual boostrap distribution given the data converges to a random
measure. In the subsequent paper Chatterjee and Lahiri (2011), they propose a consistent
modification of the Lasso which also allows for consistency of residual bootstrap estimates.
However, their modification, which consists in setting smallish estimates to zero according
to a threshold value, can be problematic in finite samples in the presence of small parameters,
furthermore it involves choosing the value of the threshold parameter. In general,
proving consistency of the bootstrap is a difficult task. It typically involves proving that
the distribution of the root of interest is continuous in the sampling distribution, usually
the empirical distribution. In this chapter we introduce subsampling, a resampling
method without replacement which offers the advantage to be consistent under very weak
aasumptions.
In the remaining we will denote by X (n) a sample of n i.i.d. random variables drawn
from some distribution P belonging to a family P. For a real valued parameter function
θ : P → R and a root R n (X (n) , θ(P )), denote the distribution of R n under P by J n (·, P ).
5.1 Pointwise consistency for distribution estimation
In this section, we restrict ourselves to the case
R n (X (n) , θ(P )) = τ n
(ˆθn (X (n)) − θ(P ))
where ˆθ n is an estimator of the parameter θ and τ n is a scaling factor tending to infinity.
For a sample X (n) of size n and a value b = b(n) < n, denote by {X n,b.i } i the set of all
N n := ( n
b) subsamples of size b. The idea is to approximate the distribution of τb (ˆθ b −θ(P ))
instead, using the subsamples X n,b,i at hand (which are true samples of size b drawn from
P ) and replacing θ by ˆθ n . This results in the following estimator
ˆL n,b (x) := 1 ∑N n
1{τ b (ˆθ n,b,i −
N ˆθ n ) ≤ x} (5.1.0.1)
n
i=1
43

44 Subsampling
The consistency of ˆL n,b as an estimator to J n (·, P ) is derived under
Assumption D.
J(P ) as n → ∞.
There exists a limit law J(P ) such that J n (P ) converges weakly to
Theorem 5.1.0.12. (Politis et al., 1999, Theorem 2.2.1) Assume that assumption D
holds. Also assume τ b /τ n → n, b → ∞, and b/n → 0 as n → ∞. Then the following are
true:
(i) If x is a continuity point of J(·, P ), then
ˆL n,b (x) → P J(x, P )
(ii) If J(·, P ) is continuous, then
∣
sup ∣ˆL n,b (x) − J n (x, P ) ∣ → P 0
(iii) Let
Correspondingly, define
x∈R
c n,b (1 − α) = inf{x ∈ R|ˆL n,b (x) ≥ 1 − α}.
c(1 − α, P ) = inf{x ∈ R|J(x, P ) ≥ 1 − α}.
Then
(
1 − α − ∆J(c(1 − α, P )) ≤ lim inf P n→∞
≤ lim sup P
n→∞
≤ 1 − α.
)
τ n (ˆθ n − θ(P ) ≤ c n,b (1 − α))
)
(
τ n (ˆθ n − θ(P )) ≤ c n,b (1 − α)
If J(·, P ) is continuous at c(1 − α, P ), then
(
)
lim P τ n (ˆθ n − θ(P )) ≤ c n,b (1 − α)
n→∞
(iv) Assume that τ b (ˆθ n − θ(P )) → 0 almost surely and that
∞∑
exp{−d(n/b)} < ∞,
n=1
= 1 − α
for every d > 0, then the convergence in i and ii hold with probability one.
Proof. Define
U n (x) = U n (x, P ) = N −1
n
N n
∑
i=1
{
) }
1 τ b
(ˆθn,b,i − θ(P ) ≤ x
To prove i, we show that U n (x) converges in probablity to J(x, P ) for every continuity
point x of J(x, P ). Note that
ˆL n,b (x) = N −1
n
N n
∑
i=1
{
1 τ b (ˆθ n,b,i − θ(P )) + τ b (θ(P ) − ˆθ
}
n ) .

5.1 Pointwise consistency for distribution estimation 45
For arbitrary ε > 0, set
E n (ε) =
{
τ b |θ(P ) − ˆθ
}
n | ≤ ε .
One then verifies that
U n (x − ε) ≤ 1 {E n (ε)} ≤ ˆL n,b (x)1{E n (ε)} ≤ U n (x + ε)
holds. Further, note that the assumption τ b /τ b → 0 implies P (E n (ε)) → 0, so for fixed ε,
with probaility tending to one, we obtain
For U n (x ± ε) → P J(x ± ε), we obtain
U n (x − ε) ≤ ˆL n,b (x) ≤ U n (x + ε)
J(x − ε, P ) ≤ ˆL n,b (x) ≤ J(x + ε, P ) + ε
with probability tending to one. Hence, taking a sequence ε n → 0, such that x ± ε n are
continuity points of J(·, P ) yields ˆL n,b (x) → P J(x, P ). Thus, it is sufficient to show that
U n (x) → P J(x, P ) for every continuity point x of J(·, P ).
For every 1 ≤ i ≤ N n , ˆθ n,b,i is a statistic based on a sample of size b drawn from the
distribution P , hence U n (x) is a U-statistic of degree b with
0 ≤ U n (x) ≤ 1 and E (U n (x)) = J b (x, P )
By Hoeffding’s ineqality ((Serfling, 1980, Theorem A, Section 5.6)), it follows that
P (U n (x) − J b (x, P ) ≥ t) ≤ exp
(−2⌊n/b⌋t 2)
for every t > 0. A similar inequality is obtained for t < 0 by considering −U n (x). So, it
follows that U n (x) → P J b (x, P ), for continuity points x of J(·, P ), this yields U n (x) → P
J(x, P ) since for such x, J b (x, P ) → J(x, P ) by the portmanteau theorem.
To prove ii, we use the subsequence criterion. Following i, given an arbitrary subsequence
{j n } n , for every continuity point x of J(·, P ), one can extract a further subsequence {k jn } n
such that L kjn (x) → J(x, P ) almost surely. By a diagonal argument, one can assume that
L kjn (x) → J(x, P ) almost surely for every x in a countable subset of the real line. So, we
obtain L kjn J(·, P ). By continuity of J(·, P ), it follows from Polya’s theorem that
∣
sup ∣L kjn (x) − J(x, P ) ∣ → 0
x∈R
almost surely, which completes the argument.
To prove iii, for α ∈ (0, 1), define
and
c L (1 − α) = inf {x ∈ R|J(x, P ) ≥ 1 − α}
c U (1 − α) = sup {x ∈ R|J(x, P ) ≤ 1 − α}
Then choose ε > 0 such that c L (1 − α) − ε and c U (1 − α) + ε are both continuity points
of J(·, P ). Following i, we have both
ˆL n,b (c L (1 − α) − ε) → P J(c L (1 − α) − ε, P )

46 Subsampling
and
ˆL n,b (c U (1 − α) + ε) → P J(c U (1 − α) + ε, P ).
Hence, the sets
{ˆLn,b (c L (1 − α) − ε) < 1 − α ≤ ˆL n,b (c U (1 − α) + ε)}
⊆
{
}
−1
c L (1 − α) − ε < ˆL
n,b (1 − α) ≤ c U(1 − α) + ε
have probability tending to one as n → ∞. It follows that
(
)
P τ n (ˆθ n − θ(P )) ≤ ĉ n,b (1 − α) ≤ J n (c U (1 − α) + ε, P ) + o(1)
and
P
(
)
τ n (ˆθ n − θ(P )) ≤ ĉ n,b (1 − α) ≥ J n (c L (1 − α) − ε, P ) + o(1).
Letting n tend to infinity first, then ε tend to zero yields, together with the Portmanteau
Theorem, the inequalities.
Finally iv can be proved similarly to i and ii using Borel-Cantelli Lemma.
Remark.
(i) Note that point iii also holds for the root U n,b . Indeed, the proof for ˆL n,b (·) solely
rests on the convergence in probability of ˆL n,b (x) to J(x, P ) for every continuity
point x of J(·, P ). As seen in the proof of i, this is a property shared by U n,b (·) as
well, this without even requiring τ b /τ n → 0, the assumption b/n → 0 being sufficient.
Obviously the price to pay are larger confidence intervals.
(ii) The conclusion of point iii can also be stated for two-sided confidence intervals with
obvious changes in the assumptions.
In the regular situation where τ n = √ n, the choice b = n δ for some 0 < δ < 1 satisfies the
conditions of Theorem 5.1.0.12.
In view of our goal, constructing confidence intervals for Lasso estimates, the message
conveyed by Theorem 5.1.0.12 is that in the situation where the 1 − α quantile happens
to be a discontinuity point, and this can indeed happen if the corresponding parameter is
equal to zero (cf. Theorem 3.2.1.1), the subsampling confidence interval assymptotically
carries an error which is in the worst case equal to the jump height at the quantile.
However, as we will see in the next section, this conclusion is too pessimistic and it turns
out that some form of uniform convergence is what we need to achieve consistency.
5.2 Uniform consistency for quantiles appproximation
The present section focuses on the use of subsampling for the construction of confidence
intervals only, in contrast to the previous one where the estimation of the distribution
function in a uniform sense was also considered. We will see that achieving asymptotic
valid or conservative confidence intervals is possible if the distribution functions satisfy
some uniformity or monoticity condition in the limit.
All results of this section, appart the Dvoretzky-Kiefer-Wolfowitz inequality, are due to
Romano and Shaikh (2010) who stated their results in a uniform sense for a family of
probability measures, we follow their exposition.

5.2 Uniform consistency for quantiles appproximation 47
5.2.1 Statement of the general result
Theorem 5.2.1.1. (Uniform asymptotic validity of subsampling) Let X (n) =
(X 1 , . . . , X n ) be an i.i.d. sequence of random variables with distribution P ∈ P. Denote
by J n (x, P ) the distribution of a real-valued root R n = R n (X (n) , P ) under P . Let
b = b n < n be a sequence of positive integers tending to infinity, but satisfying b/n → 0.
Let N n = ( b
n
) and
L n (x, P ) = 1
N n
∑
1≤i≤N n
1{R b (X n,(b),i , P ) ≤ x}, (5.2.1.1)
where X n,(b),i denotes the i-th subset of data of size b. Then, the following statements are
true for every α ∈ (0, 1) :
(i) If lim sup n→∞ sup P ∈P sup x∈R {J b (x, P ) − J n (x, P )} ≤ 0, then
lim inf
n→∞
(
)
inf P R n ≤ L −1
n (1 − α, P ) ≥ 1 − α (5.2.1.2)
P ∈P
(ii) If lim sup n→∞ sup P ∈P sup x∈R {J n (x, P ) − J b (x, P )} ≤ 0, then
lim inf
n→∞
(
)
inf P R n ≥ L −1
n (α, P ) ≥ 1 − α (5.2.1.3)
P ∈P
(iii) If lim sup n→∞ sup P ∈P sup x∈R |J b (x, P ) − J n (x, P )| = 0, then 5.2.1.2 and 5.2.1.3 hold
with the lim inf n→∞ and ≥ replaced by lim n→∞ and =, respectively. Moreover,
(
)
lim inf P L −1
n→∞
n (α, P ) ≤ R n ≤ L −1
n (1 − α, P ) = 1 − 2α (5.2.1.4)
P ∈P
Remark. Consider again the root R n of the previous section:
R n (X (n) , P ) = τ n (ˆθ n − θ(P )) (5.2.1.5)
where ˆθ n = ˆθ n (X (n) ) is an estimate of a real-valued parameter θ(P ) and tau n > 0 is
a normalizing sequence tending to infinity. For the feasible estimate ˆL n (x) of J n (x, P )
defined as
we have
L −1
n (1 − α, P ) = inf
ˆL n (x) = 1
N n
⎧
⎨
1
x∈R ⎩N n
⎧
⎨ 1
= inf
x∈R ⎩N n
=
∑
1≤i≤N n
1
(
τ b (X n,(b),i − ˆθ
)
n )
⎫
∑
)
⎬
1{τ b
(ˆθb (X n,(b),i ) − θ(P ) ≤ x} ≥ 1 − α
⎭
1≤i≤N n
⎫
∑
1{τ b
(ˆθb (X n,(b),i ) − ˆθ
)
⎬
n ≤ x − τ b (ˆθ n − θ(P ))}
⎭
1≤i≤N n
ˆL
−1
n (1 − α) + τ b (ˆθ n − θ(P ))
Hence, under the conditions of Theorem 5.2.1.1, we obtain for this particular root
(i) lim inf n→∞ inf P ∈P P
(
)
(τ n − τ b )(ˆθ n − θ(P )) ≤ ˆL
−1
n (1 − α) ≥ 1 − α

48 Subsampling
(ii) lim inf n→∞ inf P ∈P P
(iii) lim n→∞ inf P ∈P P
(
(τ n − τ b )(ˆθ n − θ(P )) ≥
) ˆL
−1
n (α) ≥ 1 − α
(ˆL−1
)
n (α) ≤ (τ n − τ b )(ˆθ n − θ(P )) ≤ ˆL
−1
n (1 − α) = 1 − 2α.
Remark. Following the previous remark, under the additional assumption τ n /τ b → 0,
conclusions i, ii and iii are also valid for a scaling factor τ n in lieu of τ n − τ b .
5.2.2 Proof
Lemma 5.2.2.1. Massart (1990)(Dvoretzky-Kiefer-Wolfowitz inequality) For n ∈
N, let X 1 , . . . , X n be i.i.d. random variables with probability distribution function F . Let
ˆF n be the empirical distribution function, that is
Then for every n ∈ N and ε > 0,
P
(
ˆF n (x) = 1 n∑
1{X i ≤ x}.
n
i=1
sup | ˆF n (x) − F n (x)|
x∈R
〉
(
ε) ≤ 2 exp −2nε 2) (5.2.2.1)
The following inequalities will be useful. Their verification is elementary, hence we omit
the proof.
Lemma 5.2.2.2. (Romano and Shaikh, 2010, Lemma 4.1) Let G and F be (nonrandom)
distribution functions on R, then the following are true :
(i) If sup x∈R {G(x) − F (x) ≤ ε}, then
G −1 (1 − α) ≥ F −1 (1 − (α + ε)) (5.2.2.2)
(ii) If sup x∈R {F (x) − G(x) ≤ ε}, then
G −1 (α) ≥ F −1 ((α + ε)) (5.2.2.3)
Furthermore for a random variable X with probability distribution function F , it holds :
(i) If sup x∈R {G(x) − F (x) ≤ ε}, then
P
(
)
X ≤ G −1 (1 − α) ≥ 1 − (α + ε) (5.2.2.4)
(ii) If sup x∈R {F (x) − G(x) ≤ ε}, then
P
(
)
X ≥ G −1 (α) ≥ 1 − (α + ε) (5.2.2.5)
(iii) If sup x∈R |G(x) − F (x)| ≤ ε, then
P
(
)
G −1 (α) ≤ X ≥ G −1 (1 − α) ≥ 1 − 2(α + ε). (5.2.2.6)

5.2 Uniform consistency for quantiles appproximation 49
If Ĝ is a random function R and δ > 0, then we further have that:
)
(i) If P
(sup {Ĝ(x) − F (x) ≤ ε} x∈R ≥ 1 − δ, then
)
(ii) If P
(sup x∈R {F (x) − Ĝ(x) ≤ ε} ≥ 1 − δ, then
)
(iii) If P
(sup |Ĝ(x) − F (x)| ≤ ε x∈R ≥ 1 − δ, then
P
P
(
)
X ≤ Ĝ−1 (1 − α) ≥ 1 − (α + ε + δ) (5.2.2.7)
P
(
)
X ≥ G −1 (α) ≥ 1 − (α + ε + δ) (5.2.2.8)
(
)
G −1 (α) ≤ X ≥ G −1 (1 − α) ≥ 1 − 2(α + ε) (5.2.2.9)
Lemma 5.2.2.3. Let X (n) = (X 1 , . . . , X n ) be an i.i.d. sequence of random variables with
distribution P . Denote by J n (x, P ) the probability distribution function of a real valued
root R n = R n (X (n) , P ) under P . Let N n = ( b
n) , kn = ⌊n/b⌋ and define L n (x, P ) according
to 5.2.2.5. Then, for every ε > 0 and every 0 < δ < 1, we have :
i.)
ii.)
P
(
P
(
sup |L n (x, P ) − J b (x, P )| > ε
x∈R
sup |L n (x, P ) − J b (x, P )| > ε
x∈R
)
)
≤ 1 ε
√
2π
k n
(5.2.2.10)
≤ δ ε + 2 ε exp(−2k nδ 2 ) (5.2.2.11)
Proof. Define
S n (x, P ; X 1 , . . . , X n ) = 1
k n
∑
1
1≤ik n
{
}
R((X b(i−1) , . . . , X bi ), P ) ≤ x − J b (x, P )
Denote by S n the symmetric group on a set of cardinality n. Note that {X n,(b),i } 1≤i≤Nn =
⋃
π∈S n
{(X π(b(i−1)+1) , . . . , X π(bi) )} 1≤i≤kn , This allows us to express
as
Then we have
1
N n
∑
1≤i≤N n
1{R b (X n,(b),i , P ≤ x)} − J b (x, P )
Z n (x, P ; X 1 , . . . , X n ) = 1 n!
sup
x∈R
∑ (
)
S n x, P ; X π(1) , . . . , X π(n) .
π∈S n
|Z n (x, P ; X 1 , . . . , X n )| ≤ 1 n! sup |S n (x, P ; X π(1) , . . . , X π(n) )|
x∈R

50 Subsampling
where the left hand side is the sum of n! identically distributed random ) variables, indeed
given some π ∈ S n , for every 1 ≤ i ≤ k n ,
(X π(b(i−1)+1) , . . . , X π(bi) is a sample of size b
from the distribution P . For an arbitrary ε > 0 we have
(
)
)
P
sup |Z n (x, P ; X 1 , . . . , X n )| > ε
x∈R
(
1
≤ P
n! sup |S n (x, P ; X π(1) , . . . , X π(n) )| > ε
x∈R
)
≤ 1 (sup
ε E |S n (x, P ; X 1 , . . . , X n )|
≤ 1 ε
x∈R
∫ ( 1
0
P
sup |S n (x, P ; X 1 , . . . , X n ) > u|
x∈R
where Markov’s inequality have been used in the second inequality and Fubini’s theorem
in the third one. Applying the Dvoretzky-Kiefer-Wolfowitz inequality 5.2.2.1 to the root
R b (X n,(b),i , P ), we obtain
P
(
sup |Z n (x, P ; X 1 , . . . , X n )| > ε
x∈R
)
≤ 1 ε
≤ 2 ε
∫ 1
0
2 exp(−2k n u 2 )du
√
2π
k n
(
Φ(2 √ k n − 1 2 ) )
≤ 1 ε
√
2πkn
where Φ(·) is the standard normal probability distribution function, this proves the first
inequality of the claim. Further, for arbitrary 0 < δ < 1, following previous arguments,
and after partitioning the unit interval according to δ, we obtain
(
)
≤ 1 (
)
ε E
P
sup |Z n (x, P ; X 1 , . . . , X n )| > ε
x∈R
sup |S n (x, P ; X 1 , . . . , X n )|
x∈R
≤ δ ε + 1 (sup
ε P |S n (x, P ; X 1 , . . . , X n )| > δ
x∈R
Applying the Dvoretzky-Kiefer-Wolfowitz inequality 5.2.2.1 to bound the second term on
the right hand side yields the second part of the claim.
Lemma 5.2.2.4. Let X (n) = (X 1 , . . . , X n ) be an i.i.d. sequence of random variables with
distribution P . Denote by J n (x, P ) the distribution of a real-valued root R n = R n (X (n) , P )
under P . Let k n = ⌊n/b⌋ and define L n (x, P ) according to 5.2.2.5. Then, for every ε > 0
and every 0 < γ < 1, the following hold:
(i) P (sup x∈R {L n (x, P ) − J n (x, P )} > ε) ≤
√ {
}
1 2π
+ 1 sup{J b (x, P ) − J n (x, P )} > (1 − γ)ε
γε k n x∈R
(ii) P (sup x∈R {J n (x, P ) − L n (x, P )} > ε) ≤
√ {
}
1 2π
+ 1 sup{J n (x, P ) − J b (x, P )} > (1 − γ)ε
γε k n x∈R
)
du
)

5.2 Uniform consistency for quantiles appproximation 51
(iii) P (sup x∈R |L n (x, P ) − J n (x, P )| > ε) ≤
√ {
}
1 2π
+ 1 sup |J b (x, P ) − J n (x, P )| > (1 − γ)ε
γε k n x∈R
Proof. We prove (iii), (i) and (ii) can be proved by similar arguments. For arbitrary ε > 0
and 0 < γ < 1, denote by A n (ε, γ) the event {sup x∈R |J b (x, P ) − J n (x, P )| ≤ (1 − γ)ε}, we
then have :
P
(
≤ P
≤ P
sup |L n (x, P ) − J n (x, P )| > ε
x∈R
(
+ ≤ P
≤ P
)
sup{|L n (x, P ) − J b (x, P )| + sup |J b (x, P ) − J n (x, P )|} > ε
x∈R
(
(
x∈R
sup{|L n (x, P ) − J b (x, P )| + |J b (x, P ) − J n (x, P )|} > ε; A n (ε, γ)
x∈R
(
)
sup{|L n (x, P ) − J b (x, P )| + |J b (x, P ) − J n (x, P )|} > ε; A n (ε, γ) c
x∈R
) {
sup |L n (x, P ) − J n (x, P )| > γε
x∈R
+ 1
sup |J b (x, P ) − J n (x, P )| > (1 − γ)ε
x∈R
Applying Lemma 5.2.2.3 to the first term on the right hand side of the last inequality
completes the argument.
Lemma 5.2.2.5. Let X (n) = (X 1 , . . . , X n ) be an i.i.d. sequence of random variables
with distribution P ∈ P. Denote by J n (x, P ) the distribution of a real valued root R n =
R n (X (n) , P ) under P . Let k n = ⌊n/b⌋ and define L n (x, P ) according to . For arbitrary
ε > 0 and 0 < γ < 1, define
δ 1,n (ε, γ, P ) = 1
√ {
}
2π
+ 1 sup{J b (x, P ) − J n (x, P )} > (1 − γ)ε
γε k n x∈R
δ 2,n (ε, γ, P ) = 1
√ {
}
2π
+ 1 sup{J n (x, P ) − J b (x, P )} > (1 − γ)ε
γε k n x∈R
δ 3,n (ε, γ, P ) = 1
√ {
}
2π
+ 1 sup{|J b (x, P ) − J n (x, P )|} > (1 − γ)ε
γε k n x∈R
)
)
}
.
Then we have
(i) P ( R n ≤ L −1
n (1 − α) ) ≥ 1 − (α + ε + δ 1,n (ε, γ, P ))
(ii) P ( R n ≥ L −1
n (α) ) ≥ 1 − (α + ε + δ 2,n (ε, γ, P ))
(
(iii) P
L −1
n
( α 2 ) ≤ R n ≤ L −1
n (1 − α 2 ) )
≥ 1 − (α + ε + δ 3,n (ε, γ, P ))

52 Subsampling
Proof. We prove (iii), (i) and (ii) are established by similar arguments. By part (iii) of
Lemma 5.2.2.4, for arbitrary ε > 0 and 0 < γ < 1, we have
(
)
P
sup |L n (x, P ) − J n (x, P )}| ≤ ε
x∈R
≥ 1 − δ 3,n (ε, γ, P )
The claim directly follows from the last claim of Lemma 5.2.2.2 with Ĝ(x) = L n(x, P ) and
X = R n (X (n) , P ) with probability distribution function F (x) = J n (x, P ).
We are now able to prove Theorem 5.2.1.1.
Proof. We prove (iii), the arguments for (i) and (ii) go along the same lines. For arbitrary
ε > 0 and 0 < γ < 1, define
δ 3,n (ε, γ, P ) = 1
√ {
}
2π
+ 1 sup{|J b (x, P ) − J n (x, P )|} > (1 − γ)ε
γε k n x∈R
By Lemma 5.2.2.5, we have
(
)
L −1
n (α) ≤ R n ≤ L −1
n (1 − α) ≥ 1 −
(
inf P
P ∈P
α + ε + sup{δ 3,n (ε, γ, P )}
P ∈P
Under the assumption lim n→∞ sup P ∈P sup x∈R |J n (x, P ) − J b (x, P )| = 0, for ε fixed we
have sup P ∈P {δ 3,n (ε, γ, P )} → 0. By a diagonal argument, one argues that there exists a
sequence ε n ↘ 0 such that sup P ∈P {δ 3,n (ε n , γ, P )} → 0. Applying Lemma 5.2.2.5 to the
sequence {ε n } n∈N yields
lim inf
n→∞
(
inf P
P ∈P
L −1
n
)
(α) ≤ R n ≤ L −1
n (1 − α) ≥ 1 − 2α
For the other inequality we show that lim sup n→∞ inf P ∈P P ( L −1
n (α) ≤ R n ≤ L −1
n (1 − α) ) ≤
1 − 2α. Consider an arbitrary P 0 ∈ P. Then for every ε > 0 and 0 < γ < 1,
(
)
P 0 L −1
n (α) ≤ R n L −1
n (1 − α)
(
)
= P 0 L −1
n (α) ≤ R n ≤ L −1
n (1 − α); sup |J n (x) − L n (x)| ≤ ε
x∈R
)
+ P 0
(
≤ P 0
(
L −1
n
L −1
n
(α) ≤ R n ≤ L −1
n
(α) ≤ R n ≤ L −1
n
+ P 0
(sup |J n (x) − L n (x)| > ε
x∈R
(
)
≤ P 0 Jn
−1 (α) ≤ R n ≤ Jn −1 (1 − α) + δ 3,n (ε, γ, P 0 )
= 1 − 2α + δ 3,n (ε, γ, P 0 )
(1 − α); sup |J n (x) − L n (x)| > ε
x∈R
)
(1 − α); Jn
−1 (α − ε) ≤ L −1
n (α); L −1
n (1 − α) ≤ Jn −1 (1 − α + ε)
)
where Lemma 5.2.2.3 have been used in the first inequality. Further, under the assumption
lim n→∞ sup P ∈P sup x∈R |J n (x, P ) − J b (x, P )| = 0, we have
This completes the proof.
1 − 2α + δ 3,n (ε, γ, P 0 ) → 1 − 2α.
)

5.2 Uniform consistency for quantiles appproximation 53
Figure 5.1: In red, the subsampling distribution estimates for the roots √ n( ˆβ j − β j ), j =
1, . . . , 9. n = 10000, b = n 0.65 ≈ 400, B = 2000 with penalization parameter λ n = 2 √ n.

54 Subsampling

Chapter 6
Numerical results
Motivated by the conclusions of Proposition 3.2.1.1 and Theorem 5.2.1.1 , we conduct
numerical simulations to evaluate the finite sample performance of subsampling based
confidence intervals. Control of the type I error is adressed for zero coefficients using
the duality between confidence intervals and hypothesis tests. Also, subsampling based
p-values are constructed for the purpose of multiple testing adjustment. Finally, in the
last section, the method is applied to the adaptive Lasso in a high dimensional setting.
6.1 Low dimensinal setting
For the simulation study in the low dimensional setting we consider data sets generated
from six linears models with random predictors. The six models differ in the correlation
structure between the covariates and the noise level. More precisely, data sets are generated
from a linear model
with regression parameter
Y i = x ′ iβ + ε i , i = 1, . . . , n
β = (1.5, −1.5, 0.75, −1.5, 1.5, −3, 0) ′ ∈ R 20 .
The errors {ε i } i are i.i.d normal with mean zero and standard deviation σ = 1 for the
models A, B and B, σ = 1.5 for the models A, B’ and C’. Rows of X are normally
distributed vectors with mean zero and Toeplitz covariance matrix
The models differ in the value ρ:
C ij = ρ |i−j| .
• Model A and A’: ρ = 0 (orthogonal design)
• Model B and B’: ρ = 0.6
• Model C and B’: ρ = 0.9
55

56 Numerical results
6.1.1 Confidence intervals
We recall the definition of a confidence interval.
Definition 6.1.1.1. Let α ∈ (0, 1). An interval valued functions I (j) (Z (n) ) is called confidence
interval for the parameter β j if it satisfies
(
)
P β j ∈ I (j) (Z (n) ) ≥ (1 − α).
Estimation procedure
For a sample (Y i , x i ) n i=1 of simulated observations we proceed as follows to construct confidence
intervals for the coefficients:
1. Compute the Lasso solution path for the scaled and centered observations, that is, for
Ỹ i = Y i − n −1
˜x ij =
(
n ∑
l=1
x ij − n −1
Y l , i = 1, . . . , n
∑ n ) (
x lj n −1
l=1
n ∑
l=1
(x lj −
)
n∑ −1
x kj ) 2 , j = 1, . . . , p, i = 1, . . . , n.
2. Choose the penalization parameter by K-fold cross validation (we choose K = 10) on
the whole data set (Ỹi, ˜x i ) n i=1 . Denote it by λ n,CV .
k=1
3. Set ˆβ n as the Lasso solution to the data set (Ỹi, ˜x i ) n i=1 corresponding to the parameter
λ n,CV .
4. Repeat the following steps for m = 1, . . . , B :
(a) Generate a random subsample I m ⊂ {1, . . . , n} of size b by drawing without replacement.
(b) Compute the Lasso solution path for the scaled and centered data set with indices
i ∈ I m , that is for
= Y i − b −1 ∑
Ỹ (m)
i
˜x (m)
ij =
Y l , i ∈ I m
l∈I m
⎛
⎞ ⎛
⎝x ij − b −1 ∑
x lj
⎠ ⎝b −1 ∑
l∈I m
⎞
(x lj − ∑
x kj ) 2 ⎠
l∈I m k∈I m
−1
, j = 1, . . . , p, i ∈ I m .
(c) Set ˆβ (m)
b as the Lasso solution to the data set
√
(Ỹ (m) , ˜X (m) ) corresponding to the
rescaled penalization parameter λ b,CV = λ n,CV b/n. Set Ln,b,m and L n,r,m to
L n,b,m = √ ( )
b ˆβ (m)
b − ˆβ n
and
L n,r,m = √ ( )
r ˆβ (m)
b − ˆβ n
respectively. Here, r = b/(1 − b/n) is the finite sample corrected subsample size,
cf. (Politis et al., 1999, Section 10.3.1).

6.1 Low dimensinal setting 57
5. Determine separately for each j ∈ {1, . . . , p} the following empirical quantiles of L (j)
n,b,·
and L (j)
n,r,·, that is, L (j)
n,b,(·)
and L(j)
n,r,(·)
being the ordered statistics, set
c (j)
n,b
(1 − α) = L(j)
n,b,(⌊(1−α)·B⌋) ,
c (j)
n,b
(α/2) = L(j)
n,b,(⌊α/2·B⌋) ,
and the analogous for L (j)
n,r,(·) .
c (j)
n,b
(1 − α/2) = L(j)
n,b,(⌊(1−α/2)·B⌋) .
6. For each j ∈ {1, . . . , p}, define the confidence intervals
[
I (j)
1 =
[
I (j)
2 =
[
I (j)
3 =
n − √ 1 c (j)
n
β (j)
β (j)
)
n,r(1 − α), ∞
n − √ 1 c (j)
n
n,r(1 − α/2), β n (j) − √ 1 c (j)
n
β (j)
n
−
[
I (j)
4 = β n
(j) −
1
√ n −
√
b
c (j)
n,b (1 − α), ∞ )
1
√ √ c (j) (1 − α/2), β(j) −
n − b
n,b
n
]
n,r(α/2)
1
√ n −
√
b
c (j)
n,b (α/2) ]
Estimated confidence intervals are illustrated in figures 6.1, 6.2 and 6.3
.
6.1.2 Hypothesis testing
For j ∈ {1, . . . , p}, consider the problem of testing the null hypothesis
against the alternative
H 0,j : β j = 0
H A,j : β j ≠ 0.
Definition 6.1.2.1. Let α ∈ (0, 1). A function φ j : R n → {0, 1} said to reject H 0,j
when it takes value 1 and to accept H 0,j when it takes value zero, is called a test for the
hypothesis H 0,j to the level α if it satisfies
E βj =0
( )
φ (j) (Z (n) ) ≤ α.
According to the duality Lemma, if I (j) is a confidence interval to the level α for β j , then
a test is given by
{
φ (j) (Z (n) 0 if 0 ∈ I
) =
(j) (Z (n) )
1 else

58 Numerical results
Figure 6.1: Subsampling confidence intervals for single scenarios of the models A and A’
(n= 250). Red triangles stand for the true parameters.
Hence, for each j ∈ {1, . . . , p} we define the test φ (j)
k
, corresponding to the intervals I(j)
k
(k = 1, . . . , 4) defined in the previous section.
Rates of coverage and of false rejection based on 500 replications of the scenario are given
in tables 6.1, 6.2 and 6.3 for the two sided interval I 2 (with corresponding test φ 2 ) and a
subsample size b = n 0.85 . Results for the other intervals are omitted but it was noted that
the one sided confidence interval I 1 has rates of coverage/false rejection similar to I 2 and
that I 3 and I 4 are extremely conservative.
Fluctuations from the nominal level for the interval I 2 can be attributed to the stochastic
approximation to the true subsample quantile and to the fact that the penalization parameter
is chosen by cross-validation for each scenario. Thus the coverage/false positive
rates can be considered correct.
6.1.3 F.W.E.R
Instead of testing for individual hypotheses, one can also be interrested in testing for a
whole family of null hypotheses. The goal is then to control the probability to make

6.1 Low dimensinal setting 59
Figure 6.2: Subsampling confidence intervals for single scenarios of the models B and B’
(n= 250). Red triangles stand for the true parameters.
at least one false rejection. This is formalized by the Family Wise Error Rate (FEWR)
defined as follows
Definition 6.1.3.1. Let H 1 , · · · , H s be a family of hypotheses. The corresponding FWER
is defined as the probability to make at least one false rejection among the hypotheses H i ,
i.e.
( s
)
⋃
F W ER = P H1 ,...,H s
{H i is rejected }
i=1
(6.1.3.1)
Our goal is to test the hypotheses H 1 , . . . , H s while controling the FWER to a given level
α. That is,
F W ER ≤ α.
We focus on the Bonferonni-Holm procedure. It is a so called step-down procedure based
on p-values for individual hypotheses. We recall the definition of a p-value,

60 Numerical results
Figure 6.3: Subsampling confidence intervals for single scenarios of the models C and C’
(n= 250). Red triangles stand for the true parameters.
Definition 6.1.3.2. Suppose that for a random variable X and an hypothesis H, one has
a familiy {S α (X); α ∈ (0, 1)} of rejection regions satisfying
S α ⊂ S α ′
whenever α < α ′ . Then a p-value is defined as
ˆp = ˆp(X) = inf{α : X ∈ S α }
that is, the smallest significance level at which one would reject the hypothesis H.
Given p-values ˆp 1 , . . . , ˆp s associated to hypothesis tests H 1 , . . . , H s , the Holm procedure
consits int he following steps:
1. Consider the ordered realized p-values ˆp (1) ≤ . . . , ˆp (s) and the corresponding hypotheses
H (1) , . . . , H (s) .
2. If ˆp (1) ≥ α/s, accept H 1 , ·, H s and stop. If ˆp (1) < α/s reject H (1) and test the remaining
s − 1 hypotheses at level α/(s − 1).

6.2 High dimensional setting 61
3. If ˆp (1) < α/s but ˆp (2) ≥ α/(s − 1), accept H (2) , ·, H (s) and stop. If ˆp (1) < α/s and
ˆp (2) < α/(s − 1), reject H (2) in addition to H (1) and test the remaining s − 2 hypotheses
at level α/(s − 2).
One can show that the Bonferonni-Holm procedure controls the FWER (Lehmann and
Romano, 2005, Theorem 9.1.2).
In our case, for an individual hypothesis H 0,j : β j = 0, we propose the following p-value
based on subsampling: for a realized statistics √ n ˆβ j n computed on the whole sample,
choose as p-value the proportion among B subsamples of values √ r( ˆβ j n,b,i − ˆβ j n) lying
beyond or beneath it, depending thereon if it is positive or negative, more precisely:
ˆp j =
{
B
−1 ∑ B
i=1 1{ √ r( ˆβ j n,b,i − ˆβ j n) ≥ √ n ˆβ j n} if ˆβ j n ≥ 0
B −1 ∑ B
i=1 1{ √ r( ˆβ j n,b,i − ˆβ j n) ≤ √ n ˆβ j n} if ˆβ j n < 0
(6.1.3.2)
where r = b/(1 − b/n) is the finite sample corrected subsample size. Similar p-values
are considered in Berg, McMurry, and Politis (2010). Here, the disctintion between the
positive and the case is made to take in consideration power issues.
Powers and Family Wise Errors Rates based on 500 replications are reported in tables
6.4, 6.5 and 6.6. The procedure controls the FWER while leaving the power unaffected,
indicating that the proposed p-values are correct.
6.2 High dimensional setting
To access the performance of subsampling in a high dimensional setting, we apply it to the
adaptive Lasso presented in Chapter 3. We generate four data sets D,D’,E and E’ which
differ in the correlation structure between covariates and the dimension of the parameters.
They result from the linear model
where
Y i = x ′ iβ + ε i , i = 1, . . . , n
β = (1, −1.25, , −1.25, 1, −1.5, 0) ′
is a 400 × 1 vector in models D and D’, and a 800 × 1 vector in models E E’. The errors ε i
are taken i.i.d. standard normal. X is multivariate normal with mean zero and covariance
matrix C given by
• C i,j = 0.8 |i−j| 1{i, j ∈ {1, . . . , 5} or i, j ∈ {6, . . . , 400}} in model D,
• C i,j = 0.8 |i−j| , i, j = 1, . . . , 400 in model D’,
• C i,j = 0.8 |i−j| 1{i, j ∈ {1, . . . , 5} or i, j ∈ {6, . . . , 800}} in model E,

62 Numerical results
• C i,j = 0.8 |i−j| , i, j = 1, . . . , 800 in model E’.
Models D and E respect the partial orthogonality condition between relevant and irrelevant
parameters while D’ and E’ violate it.
Estimation procedure
For a sample (Y i , x i ) n i=1 of simulated observations, we proceed as follows to construct
confidence intervals for the coefficinents:
(i) Center and scale the observations, that is, consider
Ỹ i = Y i − n −1
⎛
n ∑
˜x ij = ⎝x ij − n −1
Y l , i = 1, . . . , n
l=1
⎞
∑ n (
⎠ n −1
n ∑
(x lj −
lj
l=1 k=1
)
n∑ −1
x kj ) 2 , j = 1, . . . , p n , i = 1, . . . , n.
(ii) Set weights w j as
w j = n −1
n ∑
i=1
Ỹ i˜x ij , j = 1, . . . , p n .
(iii) Compute the Lasso path for the data set with scaled covariates, that is, for
(Ỹi, diag(w 1 , . . . , w pn )˜x i ) n i=1.
Then choose the penalization parameter by K-fold cross validation (we choose K=10)
based on (Ỹi, diag(w 1 , . . . , w pn )x i ) n i=1 . Denote it by λ n,CV . Let ˜β n be the Lasso solution
corresponding to (Ỹi, diag(w 1 , . . . , w pn )x i ) n i=1 and to the penalization parameter
λ n,CV . Finally, set the adaptive Lasso solution to
ˆβ n = diag(w −1
1 , . . . , w−1 p n
) ˜β n .
(iv) Repeat the folowing steps for m = 1, . . . , B:
(a) Generate a random subsample I m ⊂ {1, . . . , n} of size b by drawing without
replacement.
(b) Center and scale observations with index i ∈ I m to obtain a data set (Ỹ (m)
i , x (m)
i ) i∈Im .
(m)
(c) Compute the Lasso solution path for the data set (Ỹi , diag(w1 , . . . , w pn )x (m)
i
with covariates scaled by weights obtained in step ii. Let ˜β (m)
b be the Lasso solution
corresponding to (Ỹi , diag(w1 , . . . , w pn )x (m)
i ) i∈Im and to the rescaled
(m)
penalization parameter λ b,CV = λ n,CV (b/n) 0.4 .
) i∈Im

6.2 High dimensional setting 63
(d) Set the adaptive Lasso solution to
ˆβ (m)
b
and then set L n,b,m and L n,r,m to
= diag(w1 −1 , . . . , w−1 p n
) ˜β (m)
b
L n,b,m = √ ( )
b ˆβ (m)
b − ˆβ n
and
L n,r,m = √ ( )
r ˆβ (m)
b − ˆβ n
respectively. Here, r = b/(1 − b/n) is the finite sample corrected subsample size,
cf. (Politis et al., 1999, Section 10.3.1).
(v) Determine separately for each j ∈ {1, . . . , p n } the following empirical quantiles of
L (j)
n,b,· and L(j) n,r,·, that is, L (j)
n,b,(·)
and L(j)
n,r,(·)
being the ordered statistics, set
c (j)
n,b
(1 − α) = L(j)
n,b,(⌊(1−α)·B⌋) ,
c (j)
n,b
(α/2) = L(j)
n,b,(⌊α/2·B⌋) ,
and the analogous for L (j)
n,r,(·) .
c (j)
n,b
(1 − α/2) = L(j)
n,b,(⌊(1−α/2)·B⌋) .
(vi) For each j ∈ {1, . . . , p}, define the confidence intervals
[
I (j)
1 =
[
I (j)
2 =
[
I (j)
3 =
n − √ 1 c (j)
n
β (j)
β (j)
)
n,r(1 − α), ∞
n − √ 1 c (j)
n
n,r(1 − α/2), β n (j) − √ 1 c (j)
n
β (j)
n
−
[
I (j)
4 = β n
(j) −
1
√ n −
√
b
c (j)
n,b (1 − α), ∞ )
1
√ √ c (j) (1 − α/2), β(j) −
n − b
n,b
n
]
n,r(α/2)
1
√ n −
√
b
c (j)
n,b (α/2) ]
Remark. Note that the procedure above does not follow the subsampling scheme in the
strict sense since the adaptive Lasso weights are not recomputed over subsamples, however
we could note that using weights obtained on the whole sample to compute subsamples
estimates actually yields better results.
Coverage rates of the two sided confidence interval I (·)
2 are illustrated in Figure 6.4. First,
we note that the results are quite robust to violation of the partial orthogonality condition.
Then we see that the coverage rate for relevant coefficients are slightly below the nominal
level and that the false positive rates for zero coefficients are conservative. A possible
reason for the former is the bias introduced by the Lasso. The conservative false rejection

64 Numerical results
rates can be explained by the variable selection property or may indicate that the rate of
convergence to the limit is actually slower for zero coefficients, of the order O( √ n/ log(p))
instead of the rate √ n used. However, due to time constraints, these suggestions could
not be investigated.
Finally, p-values similar to the ones proposed in 6.1.3.2 were also computed but they don’t
allow to control the FWER through a Bonferonni-Holm procedure. We suspect that the
variable selection property is again at the source of this problem; some adaptive Lasso
estimates take value zero over almost all subsamples. A possible solution would be to set
the p-value to one if a very large proportion of subsample estimates take the value zero.
This would involve the choice of a threshold parameter. Due to time constraints, this
direction could not be investigated further.

6.2 High dimensional setting 65
Model A (σ = 1) Model A’ (σ = 1.5)
n 100 250 500 1000 2000 100 250 500 1000 2000
Rates of coverage
β 1 0.95 0.96 0.94 0.93 0.95 0.96 0.95 0.94 0.93 0.95
β 2 0.94 0.94 0.92 0.95 0.95 0.95 0.96 0.92 0.95 0.95
β 3 0.95 0.95 0.95 0.95 0.93 0.91 0.96 0.96 0.94 0.93
β 4 0.96 0.95 0.96 0.95 0.95 0.96 0.95 0.96 0.96 0.95
β 5 0.96 0.94 0.96 0.95 0.94 0.95 0.95 0.96 0.95 0.93
β 6 0.95 0.94 0.94 0.95 0.95 0.94 0.95 0.97 0.95 0.95
False rejection rates
β 7 0.02 0.02 0.01 0.02 0.02 0.02 0.02 0.01 0.02 0.02
β 8 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02
β 9 0.01 0.02 0.02 0.03 0.02 0.01 0.02 0.02 0.03 0.02
β 10 0.02 0.02 0.02 0.02 0.03 0.02 0.02 0.02 0.02 0.03
β 11 0.03 0.02 0.03 0.03 0.03 0.03 0.02 0.03 0.03 0.03
β 12 0.04 0.03 0.03 0.02 0.02 0.04 0.03 0.03 0.02 0.02
β 13 0.01 0.03 0.02 0.03 0.03 0.01 0.03 0.02 0.03 0.03
β 14 0.03 0.04 0.02 0.03 0.02 0.03 0.04 0.02 0.03 0.02
β 15 0.03 0.02 0.02 0.03 0.02 0.03 0.02 0.02 0.03 0.02
β 16 0.02 0.02 0.03 0.01 0.02 0.02 0.02 0.03 0.01 0.02
β 17 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02
β 18 0.02 0.01 0.02 0.02 0.03 0.02 0.01 0.02 0.02 0.03
β 19 0.03 0.01 0.03 0.03 0.02 0.03 0.01 0.03 0.03 0.02
β 20 0.02 0.02 0.03 0.03 0.02 0.02 0.02 0.03 0.03 0.02
Table 6.1: Model A and A’. Empirical coverage/false positive rates for the two sided
interval I 2 .

66 Numerical results
Model B (σ = 1) Model B’ (σ = 1.5)
n 100 250 500 1000 2000 100 250 500 1000 2000
Rates of coverage
β 1 0.97 0.96 0.94 0.94 0.94 0.97 0.96 0.95 0.94 0.93
β 2 0.95 0.93 0.92 0.92 0.94 0.93 0.94 0.91 0.93 0.95
β 3 0.90 0.95 0.94 0.93 0.91 0.73 0.92 0.94 0.93 0.91
β 4 0.94 0.93 0.95 0.92 0.94 0.92 0.93 0.94 0.94 0.94
β 5 0.94 0.94 0.95 0.92 0.91 0.94 0.94 0.94 0.92 0.91
β 6 0.94 0.93 0.94 0.94 0.94 0.95 0.94 0.95 0.94 0.93
False rejection rates
β 7 0.02 0.02 0.02 0.03 0.02 0.03 0.02 0.02 0.03 0.02
β 8 0.02 0.02 0.04 0.03 0.04 0.02 0.02 0.03 0.03 0.04
β 9 0.01 0.02 0.03 0.04 0.03 0.02 0.02 0.03 0.04 0.03
β 10 0.02 0.03 0.02 0.03 0.03 0.02 0.03 0.02 0.02 0.03
β 11 0.01 0.02 0.03 0.05 0.02 0.02 0.02 0.04 0.05 0.02
β 12 0.03 0.03 0.04 0.03 0.02 0.03 0.03 0.03 0.03 0.02
β 13 0.02 0.03 0.05 0.04 0.03 0.01 0.03 0.04 0.04 0.03
β 14 0.03 0.03 0.03 0.04 0.03 0.02 0.04 0.03 0.04 0.03
β 15 0.03 0.03 0.04 0.03 0.03 0.02 0.03 0.04 0.03 0.03
β 16 0.03 0.02 0.05 0.03 0.03 0.02 0.02 0.03 0.03 0.03
β 17 0.02 0.03 0.03 0.03 0.03 0.02 0.03 0.02 0.03 0.03
β 18 0.03 0.03 0.03 0.02 0.02 0.03 0.03 0.03 0.02 0.02
β 19 0.03 0.03 0.03 0.04 0.03 0.03 0.03 0.03 0.04 0.03
β 20 0.02 0.02 0.04 0.04 0.03 0.03 0.02 0.03 0.04 0.03
Table 6.2: Model B nd B’. Empirical coverage/false positive rates for the two sided interval
I 2 .

6.2 High dimensional setting 67
Model C (σ = 1) Model C’ (σ = 1.5)
n 100 250 500 1000 2000 100 250 500 100 2000
Rates of coverage
β 1 0.92 0.94 0.93 0.94 0.96 0.78 0.93 0.94 0.94 0.95
β 2 0.84 0.91 0.92 0.92 0.94 0.60 0.86 0.88 0.92 0.94
β 3 0.56 0.80 0.90 0.94 0.92 0.45 0.63 0.75 0.88 0.93
β 4 0.81 0.91 0.94 0.92 0.92 0.60 0.85 0.91 0.92 0.92
β 5 0.80 0.94 0.93 0.91 0.90 0.57 0.86 0.94 0.92 0.9
β 6 0.94 0.94 0.95 0.95 0.92 0.85 0.94 0.95 0.93 0.92
False rejection rates
β 7 0.02 0.03 0.03 0.03 0.02 0.02 0.03 0.03 0.03 0.02
β 8 0.03 0.03 0.03 0.03 0.04 0.03 0.02 0.03 0.03 0.04
β 9 0.02 0.02 0.04 0.03 0.03 0.03 0.02 0.03 0.03 0.03
β 10 0.03 0.03 0.02 0.03 0.04 0.02 0.03 0.02 0.03 0.04
β 11 0.01 0.02 0.03 0.05 0.02 0.01 0.02 0.03 0.05 0.03
β 12 0.01 0.02 0.03 0.04 0.02 0.01 0.02 0.03 0.04 0.03
β 13 0.02 0.03 0.04 0.04 0.04 0.02 0.03 0.04 0.04 0.03
β 14 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.03
β 15 0.03 0.02 0.04 0.03 0.03 0.02 0.02 0.03 0.03 0.03
β 16 0.02 0.02 0.03 0.04 0.03 0.03 0.02 0.03 0.04 0.03
β 17 0.02 0.02 0.02 0.04 0.03 0.02 0.02 0.02 0.04 0.04
β 18 0.02 0.03 0.03 0.03 0.02 0.02 0.03 0.03 0.03 0.04
β 19 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.02
β 20 0.03 0.03 0.03 0.04 0.02 0.02 0.03 0.04 0.04 0.03
Table 6.3: Model C and C’. Empirical coverage/false positive rates for the two sided
interval I 2 .
Empirical power
Model A (σ = 1) Model A’ (σ = 1.5)
n 100 250 500 1000 2000 100 250 500 1000 2000
β 1 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 2 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 3 1.00 1.00 1.00 1.00 1.00 0.83 1.00 1.00 1.00 1.00
β 4 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 5 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 6 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
FWER 0.036 0.026 0.044 0.056 0.016 0.038 0.026 0.044 0.056 0.016
Table 6.4: Model A and A’. FWER and empirical power for nonzero coefficients.

68 Numerical results
Empirical power
Model B (σ = 1) Model B’ (σ = 1.5)
n 100 250 500 1000 2000 100 250 500 1000 2000
β 1 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 2 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 3 0.67 1.00 1.00 1.00 1.00 0.33 1.00 1.00 1.00 1.00
β 4 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 5 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
β 6 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
FWER 0.036 0.034 0.046 0.044 0.036 0.04 0.034 0.046 0.044 0.036
Table 6.5: Model B and B’. FWER and empirical power for nonzero coefficients.
Empirical power
Model C (σ = 1) Model C’ (σ = 1.5)
n 100 250 500 1000 2000 100 250 500 1000 2000
β 1 1.00 1.00 1.00 1.00 1.00 0.67 1.00 1.00 1.00 1.00
β 2 0.83 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 1.00
β 3 0.00 0.67 1.00 1.00 1.00 0.00 0.33 0.83 0.98 1.00
β 4 0.83 1.00 1.00 1.00 1.00 0.17 1.00 1.00 1.00 1.00
β 5 0.83 1.00 1.00 1.00 1.00 0.33 0.83 1.00 1.00 1.00
β 6 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
FWER 0.042 0.026 0.046 0.046 0.04 0.04 0.028 0.042 0.046 0.042
Table 6.6: Model C and C’. FWER and empirical power for nonzero coefficients.

6.2 High dimensional setting 69
Figure 6.4: Coverage rates of the two sided confidence intervals I 2 for the adaptive Lasso in
high dimension. Green triangles correspond to relevant variables. Black dots correspond
to noise variables.

70 Numerical results

Chapter 7
Concluding remarks
We reviewed the general theory of weak convergence with a focus on minizers of convex
processes and applied results together with tools from convex analysis to give a fairly
precise characterization of the limiting distribution of Lasso components in a low dimensional
setting, following the steps of Knight and Fu (2000). It was outlined that despite a
discontinuity at the point zero in the limiting distributions, the use of subsampling is still
justified to construct confidence intervals if the finite population distributions converge to
the limit uniformly, as pointed out recently in Romano and Shaikh (2010). We verified
this uniform convergence for an orthogonal design setting only but there are indications
that this property holds in greater generality; this remain an open problem.
In a high dimensional setting, where the use of the Lasso is most justified, it is more difficult
to study the large sample behavior in distribution. A major hurdle is that an eventual
limit to the objective function would not be uniquely minimized, thus prohibiting a direct
application of standard techniques from weak convergence theory. This explains why in
this setting, the asymptotics of the Lasso have not been adressed in the litterature in
a satisfactory way yet. Nevertheless, we investigated the adaptive Lasso under sparsity
assumptions in the vein of Huang et al. (2008) since this variant guarantees asymptotic
normality with optimal rate of convergence for estimates corresponding to nonzero coefficients
and it achieves variable selection, thus predicting assymptotic valid subsampling
confidence intervals for non-zero coefficients at least.
Two pictures emerge from the conducted simulation study. In a low dimensional setting,
the validity of subsampling was confirmed. Confidence intervals offer sastifying coverage
rates and proposed subsampled p-values allow to control the FWER through a Bonferonni-
Holm procedure. In a high dimensional setting however, coverage rates for nonzero coefficients
are slightly below the nominal level, this may be due to the bias introduced by the
Lasso. The conclusions for zero cofeffients are more interresting though; while convergence
to the nominal level was not achieved, probably due to variable selection consistency or
to a slower rate of convergence of order √ n/log(p n ), the rate of false positives is clearly
conservative, i.e. remains below the nominal level. Subsampled p-values however, seem
not to be valid in the high dimensional setting. This issue remains unsolved.
71

72 Concluding remarks

Bibliography
Andersen, P. and R. Gill (1982). Cox’s regression model for counting processes: a large
sample study. The annals of statistics 10 (4), 1100–1120.
Berg, A., T. McMurry, and D. Politis (2010). Subsampling p-values. Statistics & Probability
Letters 80 (17-18), 1358–1364.
Bunea, F., A. Tsybakov, and M. Wegkamp (2007). Sparsity oracle inequalities for the
lasso. Electronic Journal of Statistics 1, 169–194.
Chatterjee, A. and S. Lahiri (2010). Asymptotic properties of the residual bootstrap for
lasso estimators. Proceedings of the American Mathematical Society 138 (12), 4497–4509.
Chatterjee, A. and S. Lahiri (2011). Bootstrapping lasso estimators. Journal of the American
Statistical Association 106 (494), 1–18.
Chow, Y. and H. Teicher (1978). Probability Theory : independence, interchangeability,
martingales. Springer-Verlag.
Dudley, R. (1985). An extended wichura theorem, definitions of donsker class, and weighted
empirical distributions. Probability in Banach Spaces V 1153, 141–178.
Efron, B., T. Hastie, I. Johnstone, and R. Tibshirani (2004). Least angle regression. The
Annals of statistics 32 (2), 407–499.
Hjort, N. and D. Pollard (1993). Asymptotics for minimisers of convex processes. Technical
report, University of Oslo and Yale University.
Huang, J., S. Ma, and C. Zhang (2008). Adaptive lasso for sparse high-dimensional regression
models. Statistica Sinica 18 (4), 1603–1618.
Kallenberg, O. (2002). Foundations of modern probability. Springer Verlag.
Knight, K. and W. Fu (2000). Asymptotics for lasso-type estimators. Annals of Statistics
28 (5), 1356–1378.
Lehmann, E. and J. Romano (2005). Testing statistical hypotheses. Springer Verlag.
Massart, P. (1990). The tight constant in the dvoretzky-kiefer-wolfowitz inequality. The
Annals of Probability 18 (3), 1269–1283.
Meinshausen, N. and P. Bühlmann (2006). High-dimensional graphs and variable selection
with the lasso. The Annals of Statistics 34 (3), 1436–1462.
Politis, D., J. Romano, and M. Wolf (1999). Subsampling. Springer Verlag.
73

74 BIBLIOGRAPHY
Pollard, D. (1984). Convergence of stochastic processes. Springer.
Pollard, D. (1990). Empirical processes: theory and applications. In NSF-CBMS regional
conference series in probability and statistics. JSTOR.
Pollard, D. (1991). Asymptotics for least absolute deviation regression estimators. Econometric
Theory 7 (2), 186–199.
Rockafellar, R. (1970). Convex Analysis, volume 28 of Princeton Mathematics Series.
Princeton University Press.
Romano, J. and A. Shaikh (2010). On the uniform asymptotic validity of subsampling
and the bootstrap. Technical report, Mimeo, University of Chicago.
Serfling, Robert, J. (1980). Approximation theorems of mathematical statistics. Wiley.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the
Royal Statistical Society. Series B (Methodological) 58 (1), 267–288.
Van De Geer, S. and P. Bühlmann (2009). On the conditions used to prove oracle results
for the lasso. Electronic Journal of Statistics 3, 1360–1392.
Van der Vaart, A. (2000). Asymptotic statistics. Cambridge Univ Press.
Van der Vaart, A. and J. Wellner (1996).
Springer Verlag.
Weak convergence and empirical processes.
Zhao, P. and B. Yu (2006). On model selection consistency of lasso. The Journal of
Machine Learning Research 7, 2541–2563.
Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American
Statistical Association 101 (476), 1418–1429.

Appendix A
R Codes
A.1 Simulation code for the Lasso in a low dimensional setting
library(mnormt);
library(lars);
library(Matrix);
rm(list=ls(all=TRUE));
set.seed(1);
################################
###### Setting parameters ######
p

76 R Codes
bias

A.1 Simulation code for the Lasso in a low dimensional setting 77
beta_lasso[k,],+Inf);
intervals[k,2,,]

78 R Codes
-(sqrt(nobs)+sqrt(r))*beta_lasso[k,j]) >= 0)}
}
###### Holm -procedure ######
index_sorted[k,]

A.2 Simulation code for the adaptive Lasso in a high dimensional setting 79
p_value,false_rejection_multiple,
test_multiple,FWER,file= "simulation_low.RData");
A.2 Simulation code for the adaptive Lasso in a high dimensional
setting
library(mnormt);
library(lars);
library(Matrix);
rm(list=ls(all=TRUE));
set.seed(10);
#################################
###### Setting parameters ######
p

80 R Codes
#######################################################
###### Outer loop - Replications of the scenario ######
for(k in 1:K) {
x

A.2 Simulation code for the adaptive Lasso in a high dimensional setting 81
^(-1)*apply(L[k,,],2,quantile,probs=c(1-alpha)),beta_adaptivelasso[k,],+Inf);
intervals[k,4,,]

82 R Codes
while(!(stop_loop == 1)&(l= alpha/(p-(l-1))) {
stop_loop