Foundations of Data Science

Example: Consider flipping a coin 100 times. Suppose 62 heads and 38 tails occur.
What is the most likely value of the probability of the coin to come down heads when the
coin is flipped? In this case, it is r = 0.62. The probability that we get 62 heads if the
unknown probability of heads in one trial is r is
( ) 100
Prob (62 heads|r) = r 62 (1 − r) 38 .
62
This quantity is maximized when r = 0.62. To see this take the logarithm, which as a
function of r is ln ( )
100
62 + 62 ln r + 38 ln(1 − r). The derivative with respect to r is zero at
r = 0.62 and the second derivative is negative indicating a maximum. Thus, r = 0.62 is
the maximum likelihood estimator of the probability of heads in a trial.
12.4.11 Tail Bounds and Chernoff inequalities
Markov’s inequality bounds the probability that a nonnegative random variable exceeds
a value a.
p(x ≥ a) ≤ E(x)
a .
or
p ( x ≥ aE(x) ) ≤ 1 a
If one also knows the variance, σ 2 , then using Chebyshev’s inequality one can bound the
probability that a random variable differs from its expected value by more than a standard
deviations.
p(|x − m| ≥ aσ) ≤ 1 a 2
If a random variable s is the sum of n independent random variables x 1 , x 2 , . . . , x n of
finite variance, then better bounds are possible. For any δ > 0,
[ ]
e δ m
Prob(s > (1 + δ)m) <
(1 + δ) (1+δ)
and for 0 < γ ≤ 1,
Prob ( s < (1 − γ)m ) [
]
e −γ m
<
< e − γ2 m
(1 + γ) (1+γ) 2
Chernoff inequalities
Chebyshev’s inequality bounds the probability that a random variable will deviate
from its mean by more than a given amount. Chebyshev’s inequality holds for any probability
distribution. For some distributions we can get much tighter bounds. For example,
the probability that a Gaussian random variable deviates from its mean falls off exponentially
with the distance from the mean. Here we shall be concerned with the situation
where we have a random variable that is the sum of n independent random variables. This
397