Foundations of Data Science

A sub gradient
Figure 10.4: Illustration of a subgradient for |x| 1
at x = 0
gives −1 ≥ v 1 and choosing y 1 = −ε, gives −1 ≤ v 1 . So v 1 = −1. Similar reasoning gives
the second condition. For the third condition, choose i in I 3 and set y i = ±ε and argue
similarly.
To characterize the value of x that minimizes ‖x‖ 1
subject to Ax=b, note that at the
minimum x 0 , there can be no downhill direction consistent with the constraint Ax=b.
Thus, if the direction ∆x at x 0 is consistent with the constraint Ax=b, that is A∆x=0
so that A (x 0 + ∆x) = b, any subgradient ∇ for ‖x‖ 1
at x 0 must satisfy ∇ T ∆x = 0.
A sufficient but not necessary condition for x 0 to be a minimum is that there exists
some w such that the sub gradient at x 0 is given by ∇ = A T w. Then for any ∆x such
that A∆x = 0, ∇ T ∆x = w T A∆x = w T · 0 = 0. That is, for any direction consistent with
the constraint Ax = b, the subgradient is zero and hence x 0 is a minimum.
10.3.2 The Exact Reconstruction Property
Theorem 10.3 below gives a condition that guarantees that a solution x 0 to Ax = b is
the unique minimum 1-norm solution to Ax = b. This is a sufficient condition, but not
necessary condition.
Theorem 10.3 Suppose x 0 satisfies Ax 0 = b. If there is a subgradient ∇ to the 1-norm
function at x 0 for which there exists a w where ∇ = A T w and the columns of A corresponding
to nonzero components of x 0 are linearly independent, then x 0 minimizes ‖x‖ 1
subject to Ax=b. Furthermore, these conditions imply that x 0 is the unique minimum.
Proof: We first show that x 0 minimizes ‖x‖ 1
. Suppose y is another solution to Ax = b.
We need to show that ||y|| 1 ≥ ||x 0 || 1 . Let z = y − x 0 . Then Az = Ay − Ax 0 = 0. Hence,
∇ T z = (A T w) T z = w T Az = 0. Now, since ∇ is a subgradient of the 1-norm function at
x 0 ,
||y|| 1 = ||x 0 + z|| 1 ≥ ||x 0 || 1 + ∇ T · z = ||x 0 || 1
and so we have that ||x 0 || 1 minimizes ||x|| 1 over all solutions to Ax = b.
336