Foundations of Data Science

A T S A S is invertible. We will prove that this w satisfies the conditions of Theorem 10.3.
First, for coordinates in S,
as required.
For coordinates in ¯S, we have
(A T w) S = (A S ) T A S (A T SA S ) −1 z = z
(A T w) ¯S = (A ¯S) T A S (A T SA S ) −1 z.
Now, the eigenvalues of A T S A S, which are the squares of the singular values of A S , are
between (1 − δ s ) 2 and (1 + δ s ) 2 . So ||(A T S A S) −1 || ≤ 1
(1−δ S
. Letting p = (A T ) 2 S A S) −1 z, we
have |p| ≤
√ s
(1−δ S
. Write A
) 2 s p as Aq, where q has all coordinates in ¯S equal to zero. Now,
for j ∈ ¯S
(A T w) j = e T j A T Aq
and part (2) of Lemma 10.4 gives |(A T w) j | ≤ 9δ s+1
√ s/(1 − δ
2
s ) ≤ 1/2 establishing the
Theorem 10.3 holds.
A Gaussian matrix is a matrix where each element is an independent Gaussian variable.
Gaussian matrices satisfy the restricted isometry property. (Exercise ??)
10.4 Applications
10.4.1 Sparse Vector in Some Coordinate Basis
Consider Ax = b where A is a square n×n matrix. The vectors x and b can be considered
as two representations of the same quantity. For example, x might be a discrete time
sequence with b the frequency spectrum of x and the matrix A the Fourier transform. The
quantity x can be represented in the time domain by x and in the frequency domain by its
Fourier transform b. In fact, any orthonormal matrix can be thought of as a transformation
and there are many important transformations other than the Fourier transformation.
Consider a transformation A and a signal x in some standard representation. Then
y = Ax transforms the signal x to another representation y. If A spreads any sparse
signal x out so that the information contained in each coordinate in the standard basis is
spread out to all coordinates in the second basis, then the two representations are said to
be incoherent. A signal and its Fourier transform are one example of incoherent vectors.
This suggests that if x is sparse, only a few randomly selected coordinates of its Fourier
transform are needed to reconstruct x. In the next section we show that a signal cannot
be too sparse in both its time domain and its frequency domain.
339