Foundations of Data Science

Definition 3.1 If p is a probability density in d space, the best fit line for p is the line
l = {cv 1 : c ∈ R} where
[
v 1 = arg max E (v T x) 2] .
|v|=1 x∼p
For a spherical Gaussian centered at the origin, it is easy to see that any line passing
through the origin is a best fit line. Our next lemma shows that the best fit line for a
spherical Gaussian centered at µ ≠ 0 is the line passing through µ and the origin.
Lemma 3.16 Let the probability density p be a spherical Gaussian with center µ ≠ 0.
The unique best fit 1-dimensional subspace is the line passing through µ and the origin.
If µ = 0, then any line through the origin is a best-fit line.
Proof: For a randomly chosen x (according to p) and a fixed unit length vector v,
[
E (v T x) 2] [ (v
= E
T (x − µ) + v T µ ) ] 2
x∼p
x∼p
[ (v
= E
T (x − µ) ) 2 (
+ 2 v T µ ) ( v T (x − µ) ) + ( v T µ ) ] 2
x∼p
= E
x∼p
[ (v T (x − µ) ) 2 ] + 2 ( v T µ ) E [ v T (x − µ) ] + ( v T µ ) 2
= E
x∼p
[ (v T (x − µ) ) 2 ] + ( v T µ ) 2
= σ 2 + ( v T µ ) 2
where the fourth line follows from the fact that E[v T (x − µ)] = 0, and the fifth line
follows from the fact that E[(v T (x − µ)) 2 ] is the variance in the direction v. The best fit
line v maximizes E x∼p [(v T x) 2 ] and therefore maximizes ( v T µ ) 2
. This is maximized when
v is aligned with the center µ. To see uniqueness, just note that if µ ≠ 0, then v T µ is
strictly less when v is not aligned with the center.
We now extend Definition 3.1 to k-dimensional subspaces.
Definition 3.2 If p is a probability density in d-space then the best-fit k-dimensional
subspace V k is
[
V k = argmax E |proj(x, V )|
2 ] ,
V :dim(V )=k x∼p
where proj(x, V ) is the orthogonal projection of x onto V .
Lemma 3.17 For a spherical Gaussian with center µ, a k-dimensional subspace is a best
fit subspace if and only if it contains µ.
Proof: If µ = 0, then by symmetry any k−dimensional subspace is a best-fit subspace.
If µ ≠ 0, then, the best-fit line must pass through µ by Lemma (3.16). Now, as in the
greedy algorithm for finding subsequent singular vectors, we would project perpendicular
to the first singular vector. But after the projection, the mean of the Gaussian becomes
0 and any vectors will do as subsequent best-fit directions.
58