Foundations of Data Science

andom variables, but also Gaussians, exponentials, Poisson, etc.
The gold standard for tail bounds is the central limit theorem for independent, identically
distributed random variables x 1 , x 2 , · · · , x n with zero mean and Var(x i ) = σ 2 that
states as n → ∞ the distribution of x = (x 1 + x 2 + · · · + x n )/ √ n tends to the Gaussian
density with zero mean and variance σ 2 . Loosely, this says that in the limit, the
tails of x = (x 1 + x 2 + · · · + x n )/ √ n are bounded by that of a Gaussian with variance
σ 2 . But this theorem is only in the limit, whereas, we prove a bound that applies for all n.
In the following theorem, x is the sum of n independent, not necessarily identically
distributed, random variables x 1 , x 2 , . . . , x n , each of zero mean and variance at most σ 2 .
By the central limit theorem, in the limit the probability density of x goes to that of
the Gaussian with variance at most nσ 2 . In a limit sense, this implies an upper bound
of ce −a2 /(2nσ 2)
for the tail probability Pr(|x| > a) for some constant c. The following
theorem assumes bounds on higher moments, and asserts a quantitative upper bound of
3e −a2 /(12nσ 2) on the tail probability, not just in the limit, but for every n. We will apply
this theorem to get tail bounds on sums of Gaussian, binomial, and power law distributed
random variables.
Theorem 12.5 Let x = x 1 + x 2 + · · · + x n , where x 1 , x 2 , . . . , x n are mutually independent
random variables with zero mean and variance at most σ 2 . Suppose a ∈ [0, √ 2nσ 2 ] and
s ≤ nσ 2 /2 is a positive even integer and |E(x r i )| ≤ σ 2 r!, for r = 3, 4, . . . , s. Then,
( ) 2snσ
2 s/2
Pr (|x 1 + x 2 + · · · x n | ≥ a) ≤
.
a 2
If further, s ≥ a 2 /(4nσ 2 ), then we also have:
Pr (|x 1 + x 2 + · · · x n | ≥ a) ≤ 3e −a2 /(12nσ 2) .
Proof: We first prove an upper bound on E(x r ) for any even positive integer r and then
use Markov’s inequality as discussed earlier. Expand (x 1 + x 2 + · · · + x n ) r .
(x 1 + x 2 + · · · + x n ) r = ∑ ( )
r
x r 1
1 x r 2
2 · · · x rn
n
r 1 , r 2 , . . . , r n
= ∑ r!
r 1 !r 2 ! · · · r n ! xr 1
1 x r 2
2 · · · x rn
n
where the r i range over all nonnegative integers summing to r. By independence
E(x r ) = ∑ r!
r 1 !r 2 ! · · · r n ! E(xr 1
1 )E(x r 2
2 ) · · · E(x rn
n ).
If in a term, any r i = 1, the term is zero since E(x i ) = 0. Assume henceforth that
(r 1 , r 2 , . . . , r n ) runs over sets of nonzero r i summing to r where each nonzero r i is at least
two. There are at most r/2 nonzero r i in each set. Since |E(x r i
i )| ≤ σ2 r i !,
∑
E(x r ) ≤ r! σ 2( number of nonzero r i in set) .
(r 1 ,r 2 ,...,r n)
402