Foundations of Data Science

d
n
A
=
b
x
Figure 10.2: Ax = b has a vector space of solutions but possibly only one sparse solution.
would be a 2s-sparse solution to the homogeneous system Ax = 0. A 2s-sparse solution to
the homogeneous equation Ax = 0 requires that some 2s columns of A be linearly dependent.
Unless A has 2s linearly dependent columns there can be only one s-sparse solution.
Now suppose n is order s 2 and we pick an n × d matrix A with random independent
zero mean, unit variance Gaussian entries. Take any subset of 2s columns of A. Since
we have already seen in Chapter 2 that each of these 2s vectors is likely to be essentially
orthogonal to the space spanned by the previous vectors, the sub-matrix is unlikely to be
singular. This intuition can be made rigorous.
To find a sparse solution to Ax = b, one would like to minimize the zero norm ‖x‖ 0
over {x|Ax = b} . This is a computationally hard problem. There are techniques to minimize
a convex function over a convex set. But ||x|| 0 is not a convex function. With no
further hypotheses, it is NP-hard. With this in mind, we use the one norm as a proxy for
the zero norm and minimize the one norm ‖x‖ 1
over {x|Ax = b}. Although this problem
appears to be nonlinear, it can be solved by linear programming by writing x = u − v,
u ≥ 0, and v ≥ 0, and then minimizing the linear function ∑ u i + ∑ v i subject to Aui
i
Av=b, u ≥ 0, and v ≥ 0.
Under what conditions will minimizing ‖x‖ 1
over {x|Ax = b} recover the s-sparse
solution to Ax=b? If g(x) is a convex function, then any local minimum of g is a
global minimum. If g(x) is differentiable at its minimum, the gradient ∇g must be zero
there. However, the 1-norm is not differentiable at its minimum. Thus, we introduce the
concept of a subgradient of a convex function. Where the function is differentiable the
subgradient is the gradient. Where the function is not differentiable, the sub gradient is
any line touching the function at the point that lies totally below the function. See Figure
10.3.
334