Foundations of Data Science

and
Prob (A|B) =
Prob (B|A) Prob (A)
Prob (B)
≈
0.99 × 0.001
0.0210
= 0.0471
Even though the test fails to detect a defective product only 1% of the time when it
is defective and claims that it is defective when it is not only 2% of the time, the test
is correct only 4.7% of the time when it says a product is defective. This comes about
because of the low frequencies of defective products.
The words prior, a posteriori, and likelihood come from Bayes theorem.
a posteriori =
likelihood × prior
normalizing constant
Prob (B|A) Prob (A)
Prob (A|B) =
Prob (B)
The a posteriori probability is the conditional probability of A given B. The likelihood
is the conditional probability Prob(B|A).
Unbiased Estimators
Consider n samples x 1 , x 2 , . . . , x n from a Gaussian distribution of mean µ and variance
σ 2 . For this distribution, m = x 1+x 2 +···+x n
is an unbiased estimator of µ, which means
n
n∑
that E(m) = µ and 1 (x
n i − µ) 2 is an unbiased estimator of σ 2 . However, if µ is not
i=1
n∑
known and is approximated by m, then 1 (x
n−1 i − m) 2 is an unbiased estimator of σ 2 .
Maximum Likelihood Estimation MLE
i=1
Suppose the probability distribution of a random variable x depends on a parameter
r. With slight abuse of notation, since r is a parameter rather than a random variable, we
denote the probability distribution of x as p (x|r) . This is the likelihood of observing x if
r was in fact the parameter value. The job of the maximum likelihood estimator, MLE,
is to find the best r after observing values of the random variable x. The likelihood of r
being the parameter value given that we have observed x is denoted L(r|x). This is again
not a probability since r is a parameter, not a random variable. However, if we were to
apply Bayes’ rule as if this was a conditional probability, we get
L(r|x) = Prob(x|r)Prob(r) .
Prob(x)
Now, assume Prob(r) is the same for all r. The denominator Prob(x) is the absolute
probability of observing x and is independent of r. So to maximize L(r|x), we just maximize
Prob(x|r). In some situations, one has a prior guess as to the distribution Prob(r).
This is then called the “prior” and in that case, we call Prob(x|r) the posterior which we
try to maximize.
396