Foundations of Data Science

The normal distribution is
1
√ e − 1 (x−m) 2
2 σ 2
2πσ
where m is the mean and σ 2 1
is the variance. The coefficient √
2πσ
makes the integral of
the distribution be one. If we measure distance in units of the standard deviation σ from
the mean, then
φ(x) = √ 1 e − 1 2 x2
2π
Standard tables give values of the integral
∫ t
0
φ(x)dx
and from these values one can compute probability integrals for a normal distribution
with mean m and variance σ 2 .
General Gaussians
So far we have seen spherical Gaussian densities in R d . The word spherical indicates
that the level curves of the density are spheres. If a random vector y in R d has a spherical
Gaussian density with zero mean, then y i and y j , i ≠ j, are independent. However, in
many situations the variables are correlated. To model these Gaussians, level curves that
are ellipsoids rather than spheres are used.
For a random vector x, the covariance of x i and x j is E((x i − µ i )(x j − µ j )). We list
the covariances in a matrix called the covariance matrix, denoted Σ. 36 Since x and µ are
column vectors, (x − µ)(x − µ) T is a d × d matrix. Expectation of a matrix or vector
means componentwise expectation.
Σ = E ( (x − µ)(x − µ) T ) .
The general Gaussian density with mean µ and positive definite covariance matrix Σ is
1
f(x) = √
(−
(2π)d det(Σ) exp 1 )
2 (x − µ)T Σ −1 (x − µ) .
To compute the covariance matrix of the Gaussian, substitute y = Σ −1/2 (x − µ). Noting
that a positive definite symmetric matrix has a square root:
E((x − µ)(x − µ) T = E(Σ 1/2 yy T Σ 1/2 )
= Σ 1/2 ( E(yy T ) ) Σ 1/2 = Σ.
36 Σ is the standard notation for the covariance matrix. We will use it sparingly so as not to confuse
with the summation sign.
392