Optimality

Massive multiple hypotheses testing 63
The factor α0 serves asymptotically as a calibrator of the adaptive significance
threshold to the Bonferroni threshold in the least favorable scenario π0 = 1, i.e., all
null hypotheses are true. Analysis of the asymptotic ERR of the HT(α ∗ cal ) procedure
suggests a few choices of α0 in practice.
4.2. Asymptotic ERR of HT(α ∗ cal )
Recall from (2.7) that
ERR(α) = � π0α � Fm(α) � Pr(P1:m≤ α).
The probability Pr(P1:m ≤ α) is not tractable in general, but an upper bound
can be obtained under a reasonable assumption on the set Pm of the m P values.
Massive multiple tests are mostly applied in exploratory studies to produce
“inference-guided discoveries” that are either subject to further confirmation and
validation, or helpful for developing new research hypotheses. For this reason often
all the alternative hypotheses are two-sided, and hence so are the tests. It is instructive
to first consider the case of m two-sample t tests. Conceptually the data
consist of n1 i.i.d. observations on R m Xi = [Xi1, Xi2, . . . , Xim], i = 1, . . . , n1 in
the first group, and n2 i.i.d. observations Yi = [Yi1, Yi2, . . . , Yim], i = 1, . . . , n2 in
the second group. The hypothesis pair (H0k, HAk) is tested by the two-sided twosample
t statistic Tk =|T(Xk,Yk, n1, n2)| based on the dataXk ={X1k, . . . , Xn1k}
andYk ={Y1k, . . . , Yn2k}. Often in biological applications that study gene signaling
pathways (see e.g., Kuo et al. [18], and the simulation model in Section
5), Xik and Xik ′ (i = 1, . . . , n1) are either positively or negatively correlated
for certain k �= k ′ , and the same holds for Yik and Yik ′ (i = 1, . . . , n2). Such
dependence in data raises positive association between the two-sided test statis-
tics Tk and Tk ′ so that Pr(Tk ≤ t|T ′ k ≤ t) ≥ Pr(Tk ≤ t), implying Pr(Tk ≤
t, Tk ′ ≤ t)≥Pr(Tk ≤ t)Pr(Tk ′ ≤ t), t≥0. Then the P values in turn satisfy
Pr(Pk > α, Pk ′ > α)≥Pr(Pk > α)Pr(Pk ′ > α), α∈[0,1]. It is straightforward to
generalize this type of dependency to more than two tests. Alternatively, a direct
model for the P values can be constructed.
Example 4.1. LetJ ⊆{1, . . . , m} be a nonempty set of indices. Assume Pj =
P Xj
0 , j∈J , where P0 follows a distribution F0 on [0, 1], and Xj’s are i.i.d. continuous
random variables following a distribution H on [0,∞), and are independent
of the P values. Assume that the Pi’s for i�∈J are either independent or related to
each other in the same fashion. This model mimics the effect of an activated gene
signaling pathway that results in gene differential expression as reflected by the P
values: the setJ represents the genes involved in the pathway, P0 represents the
underlying activation mechanism, and Xj represents the noisy response of gene j
resulting in Pj. Because Pi > α if and only if Xj < log α � log P0, direct calculations
using independence of the Xj’s show that
⎛
Pr⎝
�
⎞ ⎛
� 1
{Pj >α} ⎠= Pr⎝
�
� �
log α
Xj <
log t
⎞ �� � �� �
|J |
⎠dF0(t)=E
log α
H
,
log P0
j∈J
0
j∈J
where|J| is the cardinalityJ . Next
�
Pr(Pj > α) = �
j∈J
j∈J
� 1
0
�
H
� �� � � � ��� |J |
log α
log α
dF0(t) = E H
.
log t
log P0