Mathematics in Independent Component Analysis

1.4. Sparseness 25
1.4 Sparseness
One of the fundamental questions in signal processing, data mining and neuroscience is how to
represent a large data set X, given in form of a (m × T )-matrix, in different ways. A simple
approach is a linear matrix factorization
X = AS, (1.12)
which is equivalent to model (1.1) after gathering the samples into corresponding data matrices
X := (x(1), . . . , x(T )) ∈ R m×T and S := (s(1), . . . , s(T )) ∈ R n×T . We speak of a complete,
overcomplete or undercomplete factorization if m = n, m < n or m > n respectively. The
unknown matrices A and S are assumed to have some specific properties, for instance:
(i) the rows si of S are assumed to be samples of a mutually independent random vector, see
section 1.2;
(ii) each sample s(t) contains as many zeros as possible—this is the sparse representation or
sparse component analysis (SCA) problem;
(iii) the elements of X, A and S are nonnegative, which results in nonnegative matrix factorization
(NMF).
There is a large amount of papers devoted to ICA problems but mostly for the (under)complete
case m ≥ n. We refer to Lee et al. (1999), Theis et al. (2004d), Zibulevsky and Pearlmutter
(2001) and references therein for work on overcomplete ICA. Here, we will discuss constraints
(ii) and (iii).
1.4.1 Sparse component analysis
We consider the blind matrix factorization problem (1.12) in the more challenging overcomplete
case, where the additional information compensating the limited number of sensors is the sparseness
of the sources. It should be noted that this problem is quite general and fundamental, since
the sources could be not necessarily sparse in time domain. It would be sufficient to find a linear
transformation (e.g. wavelet packets), in which the sources are sufficiently sparse. Applications
of the model include biomedical data analysis, where sparsely active sources are often assumed
(McKeown et al., 1998), and audio source separation (Araki et al., 2007).
In Georgiev et al. (2005c), see chapter 10, we introduced a novel measure for sparsity and
showed that based on sparsity alone, we were still able to identify both the mixing matrix and
the sources uniquely except for trivial indeterminacies. Here, a vector v ∈ R n is said to be ksparse
if v has at least k zero entries. An n × T data matrix is said to be k-sparse, if each of its
columns are k-sparse. The goal of sparse component analysis of level k (k-SCA) is to decompose
a given m-dimensional observed signal S as in equation (1.12) such that S is k-sparse. In our
work, we always assume that the sparsity level equals k = n − m + 1, which means that at any
time instant, fewer sources than given observations are active.
The following theorem shows that essentially the SCA model is unique if fewer sources than
mixtures are active i.e. if the sources are (n − m + 1)-sparse.