Foundations of Data Science

12.5 Bounds on Tail Probability
After an introduction to tail inequalties, the main purpose of this section is to state
the Master Tail bounds theorem of Chapter 2 (with more detail), give a proof of it, and
derive the other tail inequalities mentioned in the table in that chapter.
Markov’s inequality bounds the tail probability of a nonnegative random variable x
based only on its expectation. For a > 0,
Pr(x > a) ≤ E(x)
a .
As a grows, the bound drops off as 1/a. Given the second moment of x, Chebyshev’s
inequality, which does not assume x is a nonnegative random variable, gives a tail bound
falling off as 1/a 2
( (x ) 2
)
E − E(x)
Pr(|x − E(x)| ≥ a) ≤
.
a 2
Higher moments yield bounds by applying either of these two theorems. For example,
if r is a nonnegative even integer, then x r is a nonnegative random variable even if x takes
on negative values. Applying Markov’s inequality to x r ,
Pr(|x| ≥ a) = Pr(x r ≥ a r ) ≤ E(xr )
a r ,
a bound that falls off as 1/a r . The larger the r, the greater the rate of fall, but a bound
on E(x r ) is needed to apply this technique.
For a random variable x that is the sum of a large number of independent random
variables, x 1 , x 2 , . . . , x n , one can derive bounds on E(x r ) for high even r. There are many
situations where the sum of a large number of independent random variables arises. For
example, x i may be the amount of a good that the i th consumer buys, the length of the i th
message sent over a network, or the indicator random variable of whether the i th record
in a large database has a certain property. Each x i is modeled by a simple probability
distribution. Gaussian, exponential probability density (at any t > 0 is e −t ), or binomial
distributions are typically used, in fact, respectively in the three examples here. If the x i
have 0-1 distributions, there are a number of theorems called Chernoff bounds, bounding
the tails of x = x 1 + x 2 + · · · + x n , typically proved by the so-called moment-generating
function method (see Section 12.4.11 of the appendix). But exponential and Gaussian random
variables are not bounded and these methods do not apply. However, good bounds
on the moments of these two distributions are known. Indeed, for any integer s > 0, the
s th moment for the unit variance Gaussian and the exponential are both at most s!.
Given bounds on the moments of individual x i the following theorem proves moment
bounds on their sum. We use this theorem to derive tail bounds not only for sums of 0-1
401