Foundations of Data Science

Suppose ˜x 0 were another minimum. Then ∇ is also a subgradient at ˜x 0 as it is at x 0 .
To see this, for ∆x such that A∆x = 0 ,
‖˜x 0 + ∆x‖ 1
=
∥ x 0 + ˜x 0 − x 0 + ∆x
} {{ }
≥ ‖x 0 ‖ 1
+ ∇ T (˜x 0 − x 0 + ∆x) .
∥
1
α
The above equation follows from the definition of ∇ being a subgradient for the one norm
function, ‖‖ 1
, at x 0 . Thus,
But
‖˜x 0 + ∆x‖ 1
≥ ‖x 0 ‖ 1
+ ∇ T (˜x 0 − x 0 ) + ∇ T ∆x.
∇ T (˜x 0 − x 0 ) = w T A (˜x 0 − x 0 ) = w T (b − b) = 0.
Hence, since ˜x 0 being a minimum means || ˜x 0 || 1 = ||x 0 || 1 ,
‖˜x 0 + ∆x‖ 1
≥ ‖x 0 ‖ 1
+ ∇ T ∆x = || ˜x 0 || 1 + ∇ T ∆x.
This implies that ∇ is a sub gradient at ˜x 0 .
Now, ∇ is a subgradient at both x 0 and ˜x 0 . By Proposition 10.2, we must have that
(∇) i = sgn((x 0 ) i ) = sgn((˜x 0 ) i ), whenever either is nonzero and |(∇) i | < 1, whenever either
is 0. It follows that x 0 and ˜x 0 have the same sparseness pattern. Since Ax 0 = b and
A˜x 0 = b and x 0 and ˜x 0 are both nonzero on the same coordinates, and by the assumption
that the columns of A corresponding to the nonzeros of x 0 and ˜x 0 are independent, it
must be that x 0 = ˜x 0 .
10.3.3 Restricted Isometry Property
Next we introduce the restricted isometry property that plays a key role in exact
reconstruction of sparse vectors. A matrix A satisfies the restricted isometry property,
RIP, if for any s-sparse x there exists a δ s such that
(1 − δ s ) |x| 2 ≤ |Ax| 2 ≤ (1 + δ s ) |x| 2 . (10.1)
Isometry is a mathematical concept; it refers to linear transformations that exactly preserve
length such as rotations. If A is an n × n isometry, all its eigenvalues are ±1 and
it represents a coordinate system. Since a pair of orthogonal vectors are orthogonal in all
coordinate system, for an isometry A and two orthogonal vectors x and y, x T A T Ay = 0.
We will prove approximate versions of these properties for matrices A satisfying the restricted
isometry property. The approximate versions will be used in the sequel.
A piece of notation will be useful. For a subset S of columns of A, let A S denote the
submatrix of A consisting of the columns of S.
Lemma 10.4 If A satisfies the restricted isometry property, then
337