Looking Elsewhere and Looking Everywhere (a.k.a. Multiple Testing ...

Looking Elsewhere and Looking Everywhere 

(a.k.a. Multiple Testing, Simultaneous Confidence Bands,...) 

Victor M. Panaretos 

Institute of Mathematics 

Swiss Federal Institute of Technology, Lausanne 

victor.panaretos@epfl.ch 

SLAC – June 2012 

Victor M. Panaretos (EPFL) Progress on Statistical Issues in Searches SLAC – June 2012 1 / 25

The Look Elsewhere Effect 

Two “definitions” found in a (admittedly limited) search: 

When searching for a new resonance somewhere in a possible mass range, 

the significance of observing a local excess of events must take into 

account the probability of observing such an excess anywhere in the range 

OR 

In experiments that are aimed at detecting astrophysical sources [...] one 

usually performs a search over a continuous parameter space [...] looking 

for the most significant deviation from the background hypothesis. Such a 

procedure inherently involves a “look elsewhere effect”, namely, the 

possibility for a signal-like fluctuation to appear anywhere within the 

search range. 


The Look Elsewhere Effect 

One possible mathematical description is as follows: 

Interested in detecting presence of a signal µ(xt), t = 1, . . . , T over a 

discretised domain, {x1, . . . , xt}, on the basis of noisy measurements 

This is to be detected against some known background, say 0. 

Say, for the moment, that we are not specifically interested in 

detecting the presence of the signal in some particular location xt, but 

in detecting whether the a signal is present anywhere in the domain. 

Formally: 

as opposed to 

Does there exist a t ∈ {1, . . . , T } such that µ(xt) = 0? 

Is µ(xt) = 0 for some specific t ∈ {1, . . . , T }? 

(but might also be interested in“for which t is µ(xt) = 0”) 


The Look Elsewhere Effect as Hypothesis Testing 

More formally: 

Observe 

with εt assumed exchangeable 1 

Yt = µ(xt) + εt, t = 1, . . . , T . 

Wish to test, at some significance level α: 

 

H0 : µ(xt) = 0 for all t ∈ {1, . . . , T }, 

HA : µ(xt) = 0 for some t ∈ {1, . . . , T }. 

(alternatively, wish to construct a confidence set for all µ(xt) 

simultaneously) 

1 i.e. their joint distribution is invariant under permutation. 


(Some) Aspects to be considered 

Approach to problem can be influenced by: 

Are the individual tests independent or dependent? 

What is the geometry of the domain of the signal? 

Is µ(·) estimated with a biased estimator (e.g. via smoothing)? 


Bonferroni Method 

Consider initially the case where no specific assumptions are made, but we 

have a means of testing each individual hypothesis at some level: 

Set H0,t to be the hypothesis µ(xt) = 0. 

Assume that we have a test for H0,t for any level αt. 

How can we construct a test for the global hypothesis H0 at level α? 

Bonferroni 

1 Test individual hypotheses separately at level αt = α/T 

2 Reject H0 if at least one of the {H0,t} T t=1 

Global level is bounded as follows: 

 

T 

P[❩ ❩H0|H0] = P {❍ 

❍ H0,t} 

 

 

 

 

t=1 

H0 

 

≤ 

T 

t=1 

is rejected 

P[❍ 

❍ H0,t|H0] = T α 

T 

= α 


WARNING: TEST STATISTICS VS P-VALUES 

In what follows, we assume that, under H0,t, the corresponding test 

statistic Wt has a distribution F , that is the same for all t 

(e.g. all individual tests are t-tests with the same degrees of freedom, 

or χ 2 tests with the same degrees of freedom) 

This is typically the case in the signal/background problem. 

Therefore, arranging these test statistics in decreasing order, will 

correspond to arranging them from most to least significant. 

If the test statistics are not identically distributed under H0, then we 

need to work with p-values, instead (arranging p-values in increasing 

order will correspond to arranging them from most to least 

significant). 


Holm-Bonferroni Method 

Advantage: Works for any (discrete domain) setup! 

Disadvantage: Too conservative when T large 

Holm’s modification increases average # of hypotheses rejected at level α 

(but does not increase power for overall rejection of H0) 

Holm’s Procedure 

1 Suppose we reject H0,t for large values of a test statistic Wt 

2 Order individual test statistics, largest to smallest: W (T ) ≥ . . . ≥ W (1) 

3 Starting from t = T and going down, reject all H 0,(t) such that W (t) 

supercritical at level α/t. Stop rejecting at first subcritical W (t). 

Genuine improvement over Bonferroni if want to detect as many signals as 

possible, not just existence of some signal 

Both Holm and Bonferroni reject the global H0 if and only if sup t Wt 

supercritical at level α/T wrt distribution of corresponding W under H0 


Taking Advantage of Structure 

Bonferroni and Holm methods have the advantage of holding for: 

Any dependence structure between test statistics (or data) 

Any signal structure: µ can be totally arbitrary 

Any geometry of the signal domain 

Indeed, both methods would work for a collection of totally unrelated 

hypotheses that need to be tested simultaneously! 

↩→ Price to pay: low rejection power of global H0 

Can we take advantage of prior knowledge of additional structure to 

increase power for H0 while maintaining level? 


Taking Advantage of Structure: Independence 

In the (special) case where individual test statistics are independent, one 

may use Sime’s (in)equality, 

 

 

α(T − j + 1) 

 

P W (j) supercritical at level , for some j = 1, ..., T 

T 

H0 

 

= α 

(strict equality requires continuous test statistics, otherwise ≤ α) 

Yields Sime’s procedure (assuming independence) 

1 Suppose we reject H0,j for large values of a test statistic Wj 

2 Order test statistics from largest to smallest: W (T ) ≥ . . . ≥ W (1) 

3 If, for some j = 1, . . . , T the statistic W (j) is supercritical at level 

α(T −j+1) 

T , then reject H0. 

Provides a test for the global hypothesis H0, but does not “localise” the 

signal at a particular xt 



One can, however, devise a sequential procedure to “localise” Sime’s 

procedure, at the expense of lower power for the global hypothesis H0: 

Hochberg’s procedure (assuming independence) 

1 Suppose we reject H0,j for large values of a test statistic Wj 

2 Order test statistics from largest to smallest: W (T ) ≥ . . . ≥ W (1) 

3 Starting from j = 1, 2, ... and going up, accept all H 0,(j) such that 

W (j) subcritical at level α/j. 

4 Stop accepting for the first j such that W (j) is supercritical at level 

α/j, and reject all the remaining ordered hypotheses past that j. 

Genuine improvement over Holm-Bonferroni both overall (H0) and in 

terms of signal localisation: 

1 Rejects “more” individual hypotheses than Holm-Bonferroni 

2 Power for overall H0 “weaker” than Sime’s (for T > 2), much 

“stronger” than Holm (for T > 1). 



Rejection Level 

0.01 0.02 0.03 0.04 0.05 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

Bonferroni, Hochberg, Simes 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● ● ● ● ● ● ● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● 

● ● 

● 

● 

● 

● ● 

5 10 15 20 25 

Minimal to Maximal Significance 

Victor M. Panaretos (EPFL) Progress on Statistical Issues in Searches SLAC – June 2012 12 / 25 

● 

● 

● 

● 

● 

● 

● 

●

Taking Advantage of Structure: Dependence 

Though independence provides some improvement over the Holm and 

Bonferroni methods, in some sense it complicates things rather than 

simplifies them. 

Dependence can complicate the analysis, but improve power considerably! 

Manifests itself in any of the two following ways (or a combination) 

1 Dependence between the test statistics for each H0,t 

↩→ Can arise if the measurement errors are correlated, or if the datasets on 

which individual test statistics Wt are based on are not mutually 

exclusive for different values of t = 1, ..., T . 

2 Logical dependence between the hypotheses H0,t 

↩→ It might be that rejecting a hypothesis H0,t1 would imply necessarily 

the rejection of another, H0,t2 , or, more generally, that not all 

combinations of rejections of individual hypotheses are feasible 

(effective number of individual tests smaller than T ). 


Exploiting Dependence Between Test Statistics 

For the moment, focus on global hypothesis only (H0) 

Under no assumptions at all, we used sup t Wt compared to the α/T 

critical value of the corresponding W under H0 

(independence means cdf of W ∗ is product of individual cdfs) 

If there is dependence, then sup t Wt should have special behaviour, 

and maybe it’s too conservative to apply a strong correction to 

compare it with the critical value of the corresponding W under H0 

Instead, perhaps we should use the distribution of sup t Wt itself? 

Under a completely general dependence structure this is unknown – 

but it can be simulated! 

How? Focus again on our prototype signal search example: 

Yt = µ(xt) + εt, t = 1, . . . , T . 

with εt assumed exchangeable, test H0 : µ(xt) = 0 ∀t ∈ {1, . . . , T }. 



Westfall-Young Single Step Permutation Procedure 

Under the complete H0, the joint distribution of (Y1, . . . , YT ) is the 

same as the distribution of g(Y1, . . . , YT ), for any permutation g, 

g(Y1, . . . , YT ) = (Y π(1), . . . , Y π(T )), 

where π(·) : {1, . . . , T } → {1, . . . , T } is a bijection. 

Therefore, under H0, the distribution of W ∗ = sup t Wt can be 

approximated, by considering all possible permutations of 

(Y1, . . . , YT ) (re-calculating the supremum of individual test statistics 

for each permutation g in the set of all T ! permutations, say G). 

This will give us approximate quantiles for the distribution of 

W ∗ = sup t Wt under H0, upon which we can base testing 

Q1−α ≈ ˆQ1−α := inf 

⎧ 

⎨ 

1 

w ∈ R : 1 

⎩ T ! 

W ∗ g ≤ w ⎫ 

⎬ 

≥ 1 − α 

⎭ 


g∈G


Westfall-Young procedure has properties: 

Maintains global level α for H0 

Can also be “localised”: simply reject all H0,j such that Wj > ˆQ1−α 

↩→ To maintain level α under any set of null hypotheses, need additional 

assumptions (e.g. “subset pivotality”). Previous procedures maintained 

this without extra assumptions. 

Has a step-down version (like Holm) which is more powerful than 

Holm (under subset pivotality). 

Asymptotic “oracle” optimality as T → ∞ if only a small proportion 

(sparsity) of individual nulls are false. 

Computationally intensive for T large. Problem: often T → ∞, e.g. 

when the discretisation approximates a continuous domain... 

What if more is known about the dependence structure of {Wt} T t=1? 

↩→ Could we hope to obtain a mathematical result on the distribution of 

W ∗ = sup t Wt under H0 (at least approximately?) 


Exploiting Dependence over “Metric” Continuous Domains 

Recall our special problem: 

 

H0 : µ(xt) = 0 for all t ∈ {1, . . . , T }, 

HA : µ(xt) = 0 for some t ∈ {1, . . . , T }. 

The T test statistics arise as through the discrete approximation of a 

signal on an “metric” continuous domain, say [0, S], 

0 = x1 < x2 < . . . < xT −1 < xT = S 

We thus expect T to become very large as we increase our resolution. 

Might be worth “imbedding” the problem in continuous domain 

Rather than a collection of T test statistics {Wt : t = 1, . . . , T }, 

consider an uncountable collection indexed by the interval [0, S]. 

Take advantage of metric structure and continuity of domain (could 

they allow more structured dependence to be further exploited?) 

View collection {W (x) : x ∈ [0, S]} as random function (stoch. process) 


Exploiting Dependence over “Metric” Continuous Domains 

If the dependence among the {W (x) : x ∈ [0, S]} is structured enough, we 

hope to be able to determine useful 2 (approximate) bounds for 

P[W ∗ > u], where W ∗ = sup 

x∈[0,S] 

W (x) 

(and hence get global critical values) without resorting to simulation. 

Suppose (for example) that: 

{W (x) : x ∈ [0, S]} is zero mean stationary Gaussian process under H0 

Structured dependence could use “metric” domain through smoothness: 

When |x − y| is small, then |W (x) − W (y)| should be small too. 

But W (x) − W (y) ∼ N(0, 2C(x − y)), C(x − y) = Cov[W (x), W (y)] 

So if C(x − y) is smooth close to 0, we expect W (x) to be smooth 

2 Not so general that they do not provide power 


Exploiting Smoothness and Geometric Domains 

Disclaimer: Will not attempt to discuss regularity conditions 

Truly remarkable result: 

Take W (x) to be Gaussian and smooth 

Let x be in a “nice” geometric domain D 

Extrema and Expected Euler Characteristics (Taylor & Adler) 

 

 

 

P 

 

 

sup W (x) ≥ u − E {ϕ (Au(W , D))} 

x∈D 

 

 

 

< O 

 

exp − Cu2 

2σ2 

Au(W , D) = {y ∈ D : W (y) > u} is the excursion set of W at level u 

ϕ(D) is the Euler characteristic of the index set D 

Characterises basic topological features of set (e.g. for polyhedron 

ϕ = [#Vertices] − [#Edges] + [#Faces]) 

Expectation computable for Gaussian and related processes (χ 2 , t, F ) 


Taking Advantage of Structure 

Are the individual tests independent or dependent? 

What is the geometry of the domain of the signal? 

Is µ(·) estimated with a biased estimator (e.g. via smoothing)? 


Tests Based on Biased Estimators 

Often the test statistics W (t) are based on a biased estimator. 

Means that W (t) may not be centred under H0 

↩→ example: nonparametric estimation of µ(x), assuming it is smooth, 

but without assuming any specific parametric form. 

e.g. ssume µ(·) : [0, 1] → R has Lipschitz second derivative, and observe 

Yt = µ(xt) + εt, t = 1, . . . , T . 

with εt assumed iid variance σ 2 . Test H0 : µ(x) = 0 ∀x ∈ [0, 1]. 

A classical estimator of µ is a kernel estimator (convolution estimator) 

ˆµλ(x) = 1 

λT 

T 

 

x − xt 

YtK 

λ 

t=1 

 

, K a centred symmetric pdf on [−1, 1] 



 

Estimator uses local weighted average to estimate µ 

Sacrifices unbiasedness to reduce overall IMSE: 

E(ˆµλ(x)−µ(x)) 2 

dx = 

(ˆµλ(x) − Eˆµλ(x)) 2 

 

bias 2 

 

dx+ 

λ chosen to “optimise” bias-variance tradeoff. 

Note that Eˆµλ(x) = Kλ ∗ µ(x) (a smeared version of µ) 

(Eˆµλ(x) − µ(x)) 2 

dx 

 

variance 

Assume using a t statistic (assuming asymptotic normality), then 

Wλ(x) = ˆµλ(x) − µ(x) 

Var[ˆµλ(x)] = ˆµλ(x) − Eˆµλ(x) 

Var[ˆµλ(x)] + Eˆµλ(x) − µ(x) 

Var[ˆµλ(x)] 



Let µ and K be “smooth” and let ˆ λ be a data-tuned balancer of bias & 

variance with (ˆλ − λ ∗ )/λ ∗ = OP(T −1/10 ) (λ ∗ the true one). Let ˆQα be 

σ K 2 (x)dx 

 

T ˆλ 

⎧ 

⎛ 

⎪⎨ 

−2 log ˆλ + log ⎝ 1 

˙K 

2π 

2 ⎞ 

(y)dy 

⎠ − log 

K 2 (y)dy 

 

−2 log ˆλ 

⎪⎩ 

− log(1−α) 

2 

⎫ 

 

Then, under {H0 : µ(x) = 0 ∀x ∈ [0, 1]}, and ˆbˆ λ (x) = ˆλ 2 1 

2 ˆµ′′ (x) K 2dy, P 

 

 

 

sup ˆµˆ λ (x) − bˆ λ(x) (x) 

x∈[0,1] 

 

 

 

> ˆQ1−α 

T →∞ 

−→ α 

⎪⎬ 

. 

ˆbˆ λ (x) is a point-wise asymptotic bias correction. 

Bonferroni would be conservative, even though failing to account for 

bias! (still, this result is asymptotic in T (# of observations)) 


⎪⎭

Final Remarks 

Many different approaches, depending on formulation and detail of 

specification of relation between hypotheses 

Statistical literature on topic is vast 

The more specific we can be, the more powerful we can be 

Can use p-values equally easily 

Methods can be adapted when logical relations exist between 

hypotheses (e.g. when invalidity of H0,1 –say– logically implies 

invalidity of H0,2) 

Focused here on strong control of family-wise error rate which has 

dominated the literature (overwhelmingly frequentist) 

However, a whole wealth of other powerful methods can be obtained 

via other control criteria are becoming very popular (and successful) 

Per comparison error rate E[Number of true hypotheses rejected]/T 

Expected error rate E[Number of wrong decisions]/T 

False discovery rate E[#(Falsely rejected hypotheses)/#(Rejected 

Hypotheses)] – see Benjamini & Hochberg (1995) 


Adler, R.J. & Taylor (2007). Random Fields and Geometry. Springer. 

Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a 

practical and powerful approach to multiple testing. J. R. Statisti. Soc. B., 57: 

289–300. 

Efron, B. (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation, 

Testing and Prediction. Institute of Mathematical Statistics Monographs. 

Eubank, R.L. & Speckman, P.L. (1993). Confidence bands in nonparametric 

regression. J. Amer. Statisti. Assoc. 88: 1287–1301. 

Hall, P. (1993). On Edgeworth expansion and bootstrap confidence bands in 

nonparametric curve estimation. J. R. Statisti. Soc. B, 55” 291–304. 

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of 

significance. Biometrika, 75: 800–802. 

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. 

Statist., 6: 65–70. 

Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of 

significance. Biometrika, 73: 751–754. 

Westfall, P.H. & Young, S.S. (1993). Resampling-Based Multiple Testing: Examples 

and Methods for p-Value Adjustment. Wiley Series in Probability and Statistics.

Looking Elsewhere and Looking Everywhere (a.k.a. Multiple Testing ...

Create successful ePaper yourself

Delete template?

Save as template?