Foundations of Data Science

1. For any subset S of columns with |S| = s, the singular values of A S are all between
1 − δ s and 1 + δ s .
2. For any two orthogonal vectors x and y, with supports of size s 1 and s 2 respectively,
|x T A T Ay| ≤ 5|x||y|(δ s1 + δ s2 ).
Proof: Item 1 follows from the definition. To prove the second item, assume without loss
of generality that |x| = |y| = 1. Since x and y are orthogonal, |x + y| 2 = 2. Consider
|A(x+y)| 2 . This is between 2(1−δ s1 +δ s2 ) 2 and 2(1+δ s1 +δ s2 ) 2 by the restricted isometry
property. Also |Ax| 2 is between (1 − δ s1 ) 2 and (1 + δ s1 ) 2 and |Ay| 2 is between (1 − δ s2 ) 2
and (1 + δ s2 ) 2 . Since
it follows that
2x T A T Ay = (x + y) T A T A(x + y) − x T A T Ax − y T A T Ay
= |A(x + y)| 2 − |Ax| 2 − |Ay| 2 ,
|2x T A T Ay| ≤ 2(1 + δ s1 + δ s2 ) 2 − (1 − δ s1 ) 2 − (1 − δ s2 ) 2
6(δ s1 + δ s2 ) + (δ 2 s 1
+ δ 2 s 2
+ 4δ s1 + 4δ s2 ) ≤ 9(δ s1 + δ s2 ).
Thus, for arbitrary x and y |x T A T Ay| ≤ (9/2)|x||y|(δ s1 + δ s2 ).
Theorem 10.5 Suppose A satisfies the restricted isometry property with
δ s+1 ≤ 1
10 √ s .
Suppose x 0 has at most s nonzero coordinates and satisfies Ax = b. Then a subgradient
∇||(x 0 )|| 1 for the 1-norm function exists at x 0 which satisfies the conditions of Theorem
10.3 and so x 0 is the unique minimum 1-norm solution to Ax = b.
Proof: Let
S = {i|(x 0 ) i ≠ 0}
be the support of x 0 and let ¯S = {i|(x 0 ) i = 0} be the complement set of coordinates. To
find a subgradient u at x 0 satisfying Theorem 10.3, search for a w such that u = A T w
where for coordinates in which x 0 ≠ 0, u = sgn (x 0 ) and for the remaining coordinates
the 2-norm of u is minimized. Solving for w is a least squares problem. Let z be the
vector with support S, with z i = sgn(x 0 ) on S. Consider the vector w defined by
w = A S
(
A
T
S A S
) −1
z.
This happens to be the solution of the least squares problem, but we do not need this
fact. We only state it to tell the reader how we came up with this expression. Note that
A S has independent columns from the restricted isometry property assumption, and so
338