Marc Raimondo - CMI

Marc Raimondo
Luminy 12/08 – p.1

Marc at A.N.U.
John Dedman Mathematical Sciences
Building [27]
John Dedman Mathematical Sciences Building, JD
This building
houses:
•
•
•
•
Luminy 12/08 – p.2

The Annals of Statistics
2004, Vol. 32, No. 5, 1781–1804
DOI 10.1214/009053604000000391
© Institute of Mathematical Statistics, 2004
PERIODIC BOXCAR DECONVOLUTION AND DIOPHANTINE
APPROXIMATION
BY IAIN M. JOHNSTONE 1 AND MARC RAIMONDO 2
Stanford University and University of Sydney
We consider the nonparametric estimation of a periodic function that is
observed in additive Gaussian white noise after convolution with a “boxcar,”
the indicator function of an interval. This is an idealized model for the
problem of recovery of noisy signals and images observed with “motion
blur.” If the length of the boxcar is rational, then certain frequencies are
irretreviably lost in the periodic model. We consider the rate of convergence
of estimators when the length of the boxcar is irrational, using classical
results on approximation of irrationals by continued fractions. A basic
question of interest is whether the minimax rate of convergence is slower
Luminy 12/08 – p.3

� Jan - Jun, 1999
� postdoc
� ’the bicycle’
� Aug, 2001
� Jan - Feb 2002
� Oct - Nov 2003
Marc’s visits to Stanford
Luminy 12/08 – p.4

Effect of Ageing on Morphologic and Clinical
Predictors of Prostate Cancer Progression
Thomas A. Stamey, MD,* Marc Raimondo, PhD, †
Cheryl M. Yemoto, BS,* John E. McNeal, MD,* and
Iain M. Johnstone, PhD †
*Department of Urology School of Medicine, Stanford University,
Stanford, California, U.S.A.
† Department of Statistics, Stanford University,
Stanford, California, U.S.A.
© 2000, Blackwell Science, Inc. 1095-5100/00/$15.00/0
The Prostate Journal, Volume 2, Number 3, 2000 157–162 157
Luminy 12/08 – p.5

J. R. Statist. Soc. B (2004)
66, Part 3, pp. 547–573
Wavelet deconvolution in a periodic setting
Iain M. Johnstone,
Stanford University, USA
Gérard Kerkyacharian,
Centre National de la Recherche Scientifique and Université de Paris X, France
Dominique Picard
Centre National de la Recherche Scientifique and Université de Paris VII, France
and Marc Raimondo
University of Sydney, Australia
[Read before The Royal Statistical Society at a meeting organized by the Research Section on
‘Statistical approaches to inverse problems’ on Wednesday, December 10th, 2003, Professor
J. T. Kent in the Chair ]
Luminy 12/08 – p.6

Marc Raimondo Memorial Session:
–
Null distributions for largest eigenvalues in
multivariate analysis
Iain Johnstone, Statistics, Stanford
imj@stanford.edu
Luminy, December 16, 2008
Luminy 12/08 – p.7

Theme
� Many problems of classical multivariate statistics involve
eigenvalues x1 > x2 > ... > xp
� Asymptotics in p ⇒ useful information, [even p small]
� Illustrate for null distributions for x1
Luminy 12/08 – p.8

Theme
� Many problems of classical multivariate statistics involve
eigenvalues x1 > x2 > ... > xp
� Asymptotics in p ⇒ useful information, [even p small]
� Illustrate for null distributions for x1
Example: (multiple) Regression.
y = X β + ǫ
n×p n×q q×p n×p
ǫ ∼ N(0,In ⊗ Ip).
Tests of H0 : β = 0 use roots of det[A + xi(A + B)] = 0.
A = SSH = ˆ β T X T X ˆ β “Hypothesis”
B = SSE = y T (I − PX)y “Error”
Luminy 12/08 – p.9

A ∼ Wp(n1,I)
B ∼ Wp(n2,I)
Double Wishart Setting
Common feature: roots := (xi) p
i=1
Single Wishart
� Principal Component analysis
� Factor analysis
� Multidimensional scaling
2 independent Wisharts, p ≤ n1,n2
“null hypothesis” setting
det[x(A + B) − A] = 0
of generalized eigenproblem:
Double Wishart
� Canonical correlation analysis
� Multivariate Analysis of Variance
(MANOVA)
� Multivarate regression analysis
� Discriminant analysis
� Tests of equality of covariance matrices
Luminy 12/08 – p.10

Joint density of eigenvalues
Single Wishart: det[A − xiI] = 0.
Double Wishart: det[A − xi(A + B)] = 0.
In each case, (Fisher, Girshick, Hsu, Mood, Roy, (1939)):
f(x1,...,xp) = c �
w 1/2 (xi) �
(xi−xj) x1 ≥ ... ≥ xp
i
i

I. Setting
Outline
1. Single & Double Wishart, examples
II. Largest Eigenvalue: Limiting Law (H0)
1. Gaussian
2. Laguerre/Single W.
3. Jacobi/Double W.
III. Largest Eigenvalue: Concentration
1. Single W.
2. Double W.
Luminy 12/08 – p.12

Symmetric Gaussian matrices
A – symmetric N × N matrix drawn from (GOE)
f(A) = c exp{− 1
2 tr(AT A)}.
[for GUE: complex Hermitian]. Largest eigenvalue:
Centering and scaling constants:
θN = λmax(A)
µN = √ 2N, σN = 2 −1/2 N −1/6
Tracy-Widom limit: TW (1994, 1996) For β = 1, 2, 4 (R, C, Q),
θN − µN
σN
D
⇒ Fβ.
Luminy 12/08 – p.13

Tracy-Widom distributions (1994,96)
-4 -2 0 2 4
F2(s) = e − R ∞
s (x−s)2 q(x)dx
F1(s) 2 = F2(s)e − R ∞
s q(x)dx .
q ′′ = sq + 2q 3
(Painlevé II)
q(s) ∼ Ai(s) as s → ∞
Right tail decay: 1 − Fβ(s) ≈ e−(2/3)s3/2 .
Rate of convergence: First order N −1/3 , Choup (2006,07)
“Second-order”: J + Ma (2008) For β = 1, 2, with µN = √ 2N+1,
|P {λmax(A) ≤ µN + σNs} − Fβ(s)| ≤ CN −2/3 e −s/2 .
Luminy 12/08 – p.14

f(x)
0.35
0.3
0.25
0.2
0.15
0.1
0.05
Approximations at N = 2
Density Plot: n =2
simulated percentiles
TW1
empirical 6
4
2
0
−2
−4
−6
Probability Plot: n =2
0
−8 −6 −4 −2 0 2 4 −66 −4 −2 0 2 4 6
x
TW percentiles
– Better approximation in right tail : location of turning point of
Airy function
Luminy 12/08 – p.15

f(x)
0.35
0.3
0.25
0.2
0.15
0.1
0.05
Approximations at N = 10
Density Plot: n =10
simulated percentiles
TW1
empirical 6
4
2
0
−2
−4
−6
Probability Plot: n =10
0
−8 −6 −4 −2 0 2 4 −66 −4 −2 0 2 4 6
x
TW percentiles
Luminy 12/08 – p.16

I. Setting
Outline
1. Single & Double Wishart, examples
II. Largest Eigenvalue: Limiting Law (H0)
1. Gaussian
2. Laguerre/Single W.
3. Jacobi/Double W.
III. Largest Eigenvalue: Concentration
1. Single W.
2. Double W.
Luminy 12/08 – p.17

Tracy Widom Limits
Theorem For {real, complex}, {single, double } Wishart matrices,
if n/p → γ, [or (n1/p,n2/p) → (γ1,γ2),] then
P {n ˆ ℓ1 ≤ µnp + σnps|H0} → Fβ(s)
– “universality” in RMT (Deift, 06 ICM)
Single Wishart: β = 2 (Johansson, 00), β = 1 (J, 01)
“Microarray case” (El Karoui, 03) β = 1, 2,
n,p → ∞, n/p → 0, ∞.
Non Gaussian: (Soshnikov, 02, 06; S. + Fyodorov, 06)
� subGaussian: n − p = O(p 1/3 ) ⇒ TW limit
� heavy tails (e.g. Cauchy) ⇒ Poisson process limit
Luminy 12/08 – p.18

Second order accuracy
For {real, complex}, {single, double } Wishart matrices, if
n/p → γ, [or (n1/p,n2/p) → (γ1,γ2),] then
|P {n ˆ ℓ1 ≤ µnp + σnps|H0} − Fβ(s)| ≤ Ce −cs p −2/3 .
Single Wishart Complex: El Karoui (2006).
Real: Ma (2008)
µnp =
σnp =
��
n−1 2 +
��
n−1 2 +
�
�
p− 1
2
p− 1
2
� 2
� � 1
√n− 1
2
+ 1
√ p− 1
2
� 1/3
.
Luminy 12/08 – p.19

Beyond the “Null Hypothesis”
Classical RMT ensembles (e.g. Wp(n,I))
↔ “null hypothesis”, symmetry, no structure.
For Wp(n, Σ): For what conditions on Σ does
Some answers:
P { ˆ ℓ1 ≤ µnp(Σ) + σnp(Σ)s} → Fβ(s) ??
� sufficiently many ℓk(Σ) accumulate near ℓ1(Σ) (C)
El Karoui, 2007
� small number of (not too!) isolated ℓi(Σ)
Baik-Ben Arous-Peché, 2005, Baik-Silverstein 2006, Paul 2007
Harding (2008, Economics Letters)
Luminy 12/08 – p.20

I. Setting
Outline
1. Single & Double Wishart, examples
II. Largest Eigenvalue: Limiting Law (H0)
1. Gaussian
2. Laguerre/Single W.
3. Jacobi/Double W.
III. Largest Eigenvalue: Concentration
1. Single W.
2. Double W.
Luminy 12/08 – p.21

Roy’s Greatest Root Test: Package Output
SAS: (From Gledhill et. al.: NOAA Fisheries Reef Fish Video Surveys.)
Table 18. Multivariate statistics and F approximations for the CANDISC
procedure ran with all mincount and habitat data.
S=5 M=2 N=48.5
Statistic Value F Value Num DF Den DF Pr > F
Wilks' Lambda 0.58109196 1.15 50 454.87 0.2325
Pillai's Trace 0.50115530 1.15 50 515 0.2342
Hotelling-Lawley Trace 0.59036501 1.15 50 305.82 0.2355
Roy's Greatest Root 0.26759820 2.76 10 103 0.0047
NOTE: F Statistic for Roy's Greatest Root is an upper bound.
R: (car, heplots packages, Fox et. al.)
Multivariate Tests:
Df test stat approx F num Df den Df Pr(>F)
Pillai 5.0000 0.417938 1.845226 15.0000 171.0000 0.0320861 *
Wilks 5.0000 0.623582 1.893613 15.0000 152.2322 0.0276949 *
Hotelling-Lawley 5.0000 0.538651 1.927175 15.0000 161.0000 0.0239619 *
Roy
---
5.0000 0.384649 4.384997 5.0000 57.0000 0.0019053 **
Signif. codes: 0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1
Luminy 12/08 – p.22

The Tracy-Widom approximation
Let W = logit(x1) = log(x1/(1 − x1)).
Result: [IMJ] As p ∝ n1,n2 → ∞, (and with O(p −2/3 ) error):
In other words:
Percentiles fα:
f.90 = 0.4501
f.95 = 0.9793
f.99 = 2.0234
W − µp
σp
D
⇒ W∞ ∼ F1.
x1 ≈ eµ+σW∞
. µ+σW∞ 1 + e
0.3
0.25
0.2
0.15
0.1
0.05
-4 -2 2 4
Luminy 12/08 – p.23

Centering and Scaling Constants
� set N = n1 + n2 − 1
� define γ, φ from
sin 2 (γ/2) = (p − 1
2 )/N, sin2 (φ/2) = (n1 − 1
2 )/N
� Then µ and σ are given by
�
φ + γ
�
µp = 2 log tan , σ
2
3 p
= 16
N 2
� (given a table of F1), straightforward to code:
papptw, qapptw, rapptw.
1
sin 2 (φ + γ) sin φ sin γ .
Luminy 12/08 – p.24

Accuracy of approximate α th percentile
Approximate percentile using Tracy-Widom percentile fα:
xα = x TW
α (p,n1,n2) = e µp+fασp /(1 + e µp+fασp ).
William Chen’s tables: (2003, 2004a, 2002, 2004b) ⇒ can compare
� ’Exact’ xα(p,n1,n2) with
� T-W approx x TW
α (p,n1,n2).
Relative error: r = (x TW
α /xα) − 1.
Tracy-Widom based on p → ∞, but try p = 2 as n1,n2 vary
Luminy 12/08 – p.25

θ α
log 10 n
1
0.5
TW
Exact
p = 2 at 95th percentile
n = 2
n = 5
n = 10
n = 20
n = 40
n = 100
n = 500
0
0 5 10 15
m
3
0.070.08
2
1
0.06
0.02
0.05
0.04
0.03
TW
Contours of relative error r = (θ /θα )−1
α
0.01
0.03
0.02
0.02
0.02
0.01 0.01
0
0 5 10 15
m
Luminy 12/08 – p.26

θ α
log 10 n
1
0.5
p = 4 at 90th percentile
n = 2
n = 5
n = 10
n = 20
n = 40
0
0 5
n = 100
n = 500
10
TW
Exact
15
m
3
2
1
0.018
0.016
TW
90th %tile, S=4, Contours of relative error r = (θ /θα )−1
α
0.01
0.012 0.012
0.014
0.012
0.008
0.006
0.004
0.01
0.01
0.008
0.008
0.008
0.006 0.006
0.004 0.004
0.002
0
0 5 10 15
m
Luminy 12/08 – p.27

Summary (for Part II)
T-W approximation to null distribution of Roy’s largest root:
� Conventional percentiles: generally accurate to < 10%
relative error
� [Rough p−value assessments: qualitatively o.k. over many
orders of magnitude]
� Similar O(N −2/3 ) approximations for Wishart and Gaussian.
Recommend: Tracy-Widom approximation replace F − lower
bound in default package printouts
Luminy 12/08 – p.28

I. Setting
Outline
1. Single & Double Wishart, examples
II. Largest Eigenvalue: Limiting Law (H0)
1. Gaussian
2. Laguerre/Single W.
3. Jacobi/Double W.
III. Largest Eigenvalue: Concentration
1. Single W.
2. Double W.
Luminy 12/08 – p.29

Wishart: Concentration for l1
A ∼ Wp(n,I), A = X T X X ∼ N(0,In ⊗ Ip)
Theorem (e.g. Davidson-Szarek, 2001) With l1(A) = σ 2 1(X),
and
P {σ1(X) > Eσ1(X) + √ nt} ≤ e −nt2 /2 ,
Eσ1(X) ≤ √ n + √ p.
from Standard Result: If X ∼ Nm(0,I) and f : R m → R is
Lipschitz-1 (implied by |∇f| ≤ 1), then
P {f(X) > Ef(X) + s} ≤ e −s2 /2 .
Luminy 12/08 – p.30

‘Small’ vs. ‘Large’ Deviations
Rescale X ∼ N(0,In ⊗ Ip)/ √ n so that Eσ1(X) ≈ 1 + √ γ.
Tracy-Widom limit suggests for t ≈ sn −2/3 ,
P {σ1 ≥ Eσ1 + ct} ≈ e −(2/3)nt3/2
.
Lipschitz concentration of measure ⇒ ∀t > 0
P {σ1 ≥ Eσ1 + t} ≤ e −nt2 /2 .
� Not optimal (though simple) for ‘small’ deviations t ≪ 1
� Effective for ’large’ deviations t > 1
(recent: Majumdar-Vargassola)
Luminy 12/08 – p.31

Motivations:
� analog of Davidson-Szarek
Double Wishart
� arises in Candes-Tao sparse linear regression, Y = Aβ + ǫ,
restricted orthogonality constants θS,S ′, and analogs γS,S ′:
〈ATv,AT ′v′ 〉 ≤ θS,S ′�v��v′ � (≤ γS,S ′�ATv��AT ′v′ �).
Use CCA form: X ∼ N(0,In ⊗ Ip) ⊥ Y ∼ N(0,In ⊗ Iq)
Rp,q;n(X,Y ) = max{Corr(Xu,Y v) : �u� = �v� = 1}
Luminy 12/08 – p.32

Rmax as singular value
Use SVD: X = UXDXV T X , Y = UY DY V T
Y
⇒ R(X,Y ) = σ1(U T XUY ), with
UX ∈ Vn,p – Stiefel manifold of orthonormal p−frames, etc
Claim (provisional!)
P {σ1 ≥ Eσ1 + t} ≤ e −(n−2)t2 /8 .
⇒ other double Wishart cases (regression, MANOVA, etc).
Luminy 12/08 – p.33

Concentration for Riemannian manifolds
Theorem (Ledoux, 1992) Assume:
(1. manifold) (X,g) compact, connected, smooth Riemannian
manifold, dimension ≥ 2
(2. volume) dµ normalized Riemannian volume element
(3. Lipschitz) F 1−Lipschitz on X, i.e. |∇F | ≤ 1
(4. curvature) c(X) = inf{Ric(∇f, ∇f) : |∇f| = 1}
Then
µ{F ≥
�
Ricci curvature
Fdµ + t} ≤ e −ct2 /2 .
Need to interpret (1) - (4) for F = σ1(U T X UY ) = R(X,Y ).
Luminy 12/08 – p.34

Marc Raimondo
The Talented Student Program:
an integrated learning solution
Hugh Miller and Marc Raimondo,
School of Mathematics and Statistics
Luminy 12/08 – p.35

References
THANK YOU!
IMJ, “High Dimensional Statistical Inference and Random Matrices”, Proc
ICM 2006, Vol 1, 307–333.
IMJ, “Multivariate Analysis and Jacobi Ensembles: Largest eigenvalue,
Tracy-Widom Limits and Rates of Convergence”, arXiv:0803.3408, Ann.
Statist. in press.
Webpage: www-stat.stanford.edu/˜imj
Luminy 12/08 – p.36

Determinant representation:
Correlation function:
SN,2(x,y) =
Unitary Case
P {max xj ≤ x0} = det(I − SN,2χ0).
N−1 �
k=0
= 1
� ∞
2
0
Hermite polynomials (normalized):
�
φk(x)φk(y), χ0 = I (x0,∞)
φ(x + z)ψ(y + z) + ψ(x + z)φ(y + z)dz.
φk(x) = cke −x2 /2 Hk(x), φ(x) = (2N) 1/4 φN(x),
φk(x)φl(x)e −x2
dx = δkl, ψ(x) = (2N) 1/4 φN−1(x).
Luminy 12/08 – p.37

Rescaling and convergence to Airy function
Centred and scaled (weighted) polynomials
φτ(s) = (2N) 1/4 φN(uN + τNs),
ψτ(s) = (2N) 1/4 φN−1(uN−1 + τNs)
both satisfy differential equation
w ′′ N(s) = s(1 + O(N −1 ))wN(s).
Liouville-Green theory shows convergence to Airy A(s), solving
with O(N −2/3 ) rate:
A ′′ (s) = sA(s),
|φτ(s) − A(s)| ≤ CN −2/3 e −s/2 .
Luminy 12/08 – p.38

After rescaling:
Sτ(s,t) = 1
2
Unitary Case - Convergence
P {λmax ≤ µ + sσ} = det(I − Sτ),
� ∞
Limit: Tracy-Widom F2:
0
φτ(s + w)ψτ(t + w) + ψτ(s + w)φτ(t + w)dw
F2(s) = det(I − SA)
SA(s,t) =
� ∞
0
A(s + w)A(t + w)dw
Bound �Sτ − SA�1 using convergence of φτ,ψτ to A at O(N −2/3 ):
|det(I − Sτ) − det(I − SA)| ≤ �Sτ − SA�1 · exp(�Sτ � + �SA� + 1)
Luminy 12/08 – p.39

Orthogonal Case
P { max
1≤j≤N+1 xj ≤ x0} = � det(I − KN+1χ0),
⎛ ⎞ ⎛ ⎞
I
KN+1(x,y) = ⎝
ǫ1
−∂2
0 0
⎠SN+1,1(x,y) − ⎝ ⎠
T
ǫ(x − y) 0
∂2 ↔ ∂/∂y
ǫ1 ↔ convolution in x with ǫ(x) = 1
2 sgn(x)
T ↔ transpose x,y
Key identity relates orthogonal at N + 1 to unitary at N :
SN+1,1(x,y) = SN,2(x,y) + 1
2 φ(x)ǫψ(y).
Luminy 12/08 – p.40

Orthogonal Case - Convergence
P { max
1≤j≤N+1 xj ≤ x0} = � det(I − KN+1χ0),
F1(s) = � ⎛
det(I − KGOE)
⎞
I
Kτ(s,t) = ⎝
ǫ1
−∂2
⎠(SN,2,τ(s,t) +
T
1
2φτ(s)ǫψτ(t)) ⎛ ⎞
⎛ ⎞
0 0
− ⎝ ⎠
ǫ(x − y) 0
I
KGOE(s,t) = ⎝
ǫ1
−∂2
⎠(SA(s,t) +
T
1
⎛ ⎞
2A(s)ǫA(t)) 0 0
− ⎝ ⎠.
ǫ(x − y) 0
where O(N −1/3 ) terms cancel:
Sτ(s,t) = SA(s,t) − ∆NA(s)A(t) + O(N −2/3 ),
1
2 φτ(s)ǫψτ(t) = 1
2 A(s)(ǫA)(t) + ∆NA(s)A(t) + O(N −2/3 ).
Luminy 12/08 – p.41

Accuracy of approximate p−values
Multivariate Tests:
Df test stat approx F num Df den Df Pr(>F)
Pillai 5.00 0.6658 3.5369 15.00 186.00 2.309e-05 ***
Wilks 5.00 0.4418 3.8118 15.00 166.03 8.275e-06 ***
Hotelling-Lawley 5.00 1.0309 4.0321 15.00 176.00 2.787e-06 ***
Roy
---
5.00 0.7574 9.3924 5.00 62.00 1.062e-06 ***
Signif. codes: 0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1
� ’Exact’ percentile: from Koev’s qmaxeigjacobi
� Tracy-Widom p−value (using table of F1):
pTW(xα) = 1 − F1((logit(xα) − µ)/σ),
� p− value derived from F distribution lower bound:
pF(xα) = 1 − Fn1,n2(n2x/(n1(1 − x))),
Luminy 12/08 – p.42

Tail p−values: s = 2 variables
F bound gives wrong order of magnitude;
Tracy-Widom gives qualitatively correct p−value assessment.
S = 2, M = −0.5, N = 2
Largest Root Exact Tracy-Widom F
0.663 0.1 0.119 0.0223
0.737 0.05 0.066 0.00933
0.850 0.01 0.0169 0.00131
0.881 0.005 0.00927 0.000573
0.931 0.001 0.00222 8.49e-005
0.968 0.0001 0.000251 5.65e-006
0.985 1e-005 2.38e-005 3.81e-007
0.993 1e-006 1.89e-006 2.58e-008
Luminy 12/08 – p.43

s = 6 variables
F bound wrong by many orders of magnitude;
Tracy-Widom gives qualitatively correct p−value assessment.
S = 6, M = 5, N = 10
Largest Root Exact Tracy-Widom F
0.757 0.1 0.108 0.000117
0.781 0.05 0.0557 3.63e-005
0.823 0.01 0.0119 2.99e-006
0.837 0.005 0.00606 1.08e-006
0.864 0.001 0.00125 1.1e-007
0.894 0.0001 0.000125 4.75e-009
0.917 1e-005 1.17e-005 2.25e-010
0.934 1e-006 1.03e-006 1.13e-011
Luminy 12/08 – p.44