Orthogonality Does Not Imply Asymptotic Normality DA Freedman ...

Orthogonality Does Not Imply Asymptotic Normality
DA Freedman
Statistics 215 October 2007
Consider random variables which are orthogonal, with mean 0 and variance 1, and uniformly
bounded fourth moments. The CLT need not hold—i.e., the sum need not be asymptotically
normal—because independence is not assumed. (The CLT, being a theorem, remains true.) We
present several examples, normalizing the sum S n = X 1 +···X n by √ n,orby
n∑
D n = √ Xj 2.
The examples are relevant to general forms of the OLS model Y = Mβ + ɛ that require only
cov(ɛ|M) = σ 2 I n×n rather than IID errors; normalization by D n is akin to normalizing regression
statistics by ˆσ . The punchline: the usual asymptotics need not hold for the OLS estimator ˆβ, without
the assumption of IID errors given M. (We write M for the design matrix to avoid confusion with
the random variables X j .)
(∗) Conditions. The X j have E(X j ) = 0 and E(Xj 2) = 1. Furthermore E(X4 j
) is uniformly
bounded, and E(X j X k ) = 0 for j ̸= k.
Example 1. Let Z and {Y j } be independent. Suppose the Y j are independent, E(Y j ) = 0,
E(Z 2 ) = E(Yj 2)
= 1. Finally, suppose E(Z4 ) and E(Yj 4)
are finite. Let X j = ZY j . These random
variables satisfy the conditions (∗),butS n / √ n isn’t asymptotically normal: the limiting distribution
is normal, multiplied by Z. Normalizing by D n does give asymptotic normality, because Z cancels.
Example 2. Let U j = 0or √ 2 be a sequence of random variables constructed as follows.
With probability 1/2, the sequence consists of a long block of 0’s, followed by a very long block
of √ 2’s, followed by a very very long block of 0’s, etc. With the remaining probability 1/2, the
0’s and √ 2’s are interchanged. The Y j are IID ±1 with probability 1/2, independent of {U j }. Let
X j = U j Y j . Again, these random variables satisfy the conditions (∗). Clearly, max j Uj
2 = 2 and
Dn 2 →∞, so max j≤n Uj
2 = o(D2 n ). Furthermore, var(S n|U) = Dn 2. Thus, S n/D n is asymptotically
normal. With rapidly increasing block length, D 2 n /n oscillates between 0+ and 2−. So S n/ √ n
isn’t asymptotically normal.
Our next example involves ξ j = sin(jθ), where θ is uniform on the circle [0, 2π) and j is
an integer; the main interest is j = 1, 2,.... If z j = exp(ijθ) with i = √ −1 and exp(z) = e z ,
then ξ j is the imaginary part of z j = cos(jθ) + i sin(jθ). The next lemma follows by computing
moments.
j=1
Lemma 2. The z j all have the same distribution; furthermore, for each j,asn →∞, the joint
distribution of z n and z j converges weak-star, the two variables becoming independent.
1

Lemma 3.
(i) E(ξ j ) = 0; in fact, all odd moments vanish.
(ii) E(ξj 2 ) = 1/2 and E(ξ4
j ) = 3/8.
(iii) E(ξ j ξ k ) = 0 for j ̸= k.
(iv) ∑ n
j=1 cos(jθ) is the real part of
and ∑ n
j=1 sin(jθ) is the imaginary part.
n (θ) = e(n+1)iθ − e iθ
e iθ − 1
(v) ∑ n
j=1 sin 2 (jθ) = 1 2 (n − q n) where q n = ∑ n
j=1 cos(2jθ) is the real part of n (2θ).
Example 3. Let X j = √ 2ξ j . Conditions (∗) are satisfied. However, S n converges in distribution,
and D 2 n is of order n. Whether we normalize S n by √ n or D n —or not at all—there is no
asymptotic normality.
Sourav Chatterjee suggested that examples could be based on U-statistics. For l = 1, 2,...,
let the U l be IID, with P(U l =±1) = 1/2. Let
Q n =
∑
1≤j̸=k≤n
( n∑ ) 2 ( n∑
U j U k = U l −
whose distribution is asymptotic to n(χ 2 1 − 1). Note that Q n+1 − Q n = 2 ( ∑ n
l=1 U l
)
Un+1 .
Example 4. Let T j = ∑ j
l=1 U l/ √ j and let X j+1 = T j U j+1 for j = 1, 2,....Let X 1 = U 1 .
Conditions (∗) are easily verified; for the rest, we rely on simulations. To begin with, S n is very
skewed to the right, so cannot be asymptotically normal. On the other hand, S n /D n —although
not far from normal—has a negative mean. We can replace T j by f(T j ) for suitable functions f ,
although varf(T j ) may then depend a little on j and n. Iff(x)= x 6 , then S n itself has a much longer
tail than the normal; indeed, S n is roughly like a symmetrized log normal variable. By contrast,
S n /D n is short-tailed and bimodal. (Numerator and denominator are somewhat dependent.) Neither
S n / √ n nor S n /D n is asymptotically normal.
Back-of-the-envelope arguments suggest
1
√ n
n∑
j=1
1
n D2 n = 1 n
( 1
f √j
n∑
j=1
l=1
j∑
U l
)U j+1 →
l=1
( 1
f √j
∫ 1
0
,
l=1
U 2 l
)
f(B t / √ t)dB t (1)
j∑ ) ∫ 2 1
U l → f(B t / √ t) 2 dt (2)
l=1
2
0

where B is Brownian motion. Because the summands are uncorrelated,
⎧
⎫ ⎧
⎨ n/K
1 ∑ ( 1
j∑ ) ⎬
var √ f √j U l U j+1
⎩ n ⎭ = 1 n/K
∑ ⎨ ( 1
j∑ ) ⎫ ⎬ ( ) 1
var
n ⎩ f √j U l
⎭ = O K
j=1
Likewise,
⎧
⎨ n/K
1 ∑
E
⎩n
j=1
[ ( 1
f √j
l=1
j=1
⎫ ⎧
j∑ )] 2⎬ U l
⎭ = 1 n/K
∑ ⎨[
E f
n ⎩
l=1
j=1
The singularity near 0 therefore seems unimportant.
( 1 √j
l=1
⎫
j∑ )] 2⎬ ( ) 1
U l
⎭ = O K
l=1
(3)
(4)
Example 5. (Chaterjee.) Let the summands be U j U k , with j 1 + 2.6 √ 2 } = .03,
while P { |Z| > 2.6 } = .009, where Z is N(0,1). The tail area is off by a factor of 3, and it gets
worse further out. On the other hand, annoyingly,
P { (χ1 2 − 1)/√ 2 > 2 } {
= P |Z| >
l=1
√
1 + 2 √ 2} .= P {|Z| > 1.96}
. = .05,
The first probability is one-sided:
P {(χ1 2 − 1)/√ 2 < −2} =P {Z 2 < 1 − 2 √ 2}=0,
but the symmetric tail area is very close.
Steve Evans has another construction, which gives a sequence X 1 ,X 2 ,... of uncorrelated
random variables having mean 0 and variance 1, with subsequences of
L { (X 1 +···+X n )/ √ n }
close to any distribution with mean 0 and variance 1.
For regression asymptotics assuming independent errors, see
http://www.stat.berkeley.edu/users/census/Ftest.pdf
3