Mathematics in Independent Component Analysis

1.5. Machine learning for data preprocessing 37
for NGCA essentially using the idea of separated characteristic functions from the proof was
proposed in (Kawanabe and Theis, 2007).
Finally, in (Theis and Kawanabe, 2007), we presented an modification of NGCA that evaluates
the time structure of the multivariate observations instead of their higher-order statistics.
We differentiated the signal subspace from noise by searching for a subspace of non-trivially
autocorrelated data. In contrast to blind source separation approaches however, we did not
require the existence of sources, so the model is applicable to any wide-sense stationary time
series without restrictions. Moreover, since the method is based on second-order time structure,
it could be efficiently implemented even for large dimensions, which we illustrated with an
application to dimension reduction of functional MRI recordings.
1.5.3 Clustering
Clustering methods are an important tool in high-dimensional explorative data mining. They
aim at identifying samples or regions of similar characteristics, and often code them by a single
codebook vector or centroid. In this section, we review clustering algorithms, and employ these
methods to solve the blind matrix factorization problem (1.12) from above under various source
assumptions.
Clustering for solving overcomplete BSS problems
In Theis et al. (2006), see chapter 17, we discussed the blind source separation problem (1.1)
in the difficult case of overcomplete BSS, where less mixtures than sources are observed (m <
n). We focused on the usually more elaborate matrix recovery part. Assuming statistically
independent sources with existing variance and at most one Gaussian component, it is wellknown
that A is determined uniquely by the mixtures x(t) (Eriksson and Koivunen, 2003).
However, how to do this algorithmically is far from obvious, and although some algorithms have
been proposed recently (Bofill and Zibulevsky, 2001, Lee et al., 1999, O’Grady and Pearlmutter,
2004), performance is yet limited.
The most commonly used overcomplete algorithms rely on sparse sources (after possible
sparsification by preprocessing), which can be identified by clustering, usually by k-means or
some extension (Bofill and Zibulevsky, 2001, O’Grady and Pearlmutter, 2004). However apart
from the fact that theoretical justifications have not been found, mean-based clustering only
identifies the correct A if the data density approaches a delta distribution. In figure 1.18,
we illustrate the deficiency of mean-based clustering; we get an error of up to 5 ◦ per mixing
angle, which is rather substantial considering the sparse density and the simple, complete case
of m = n = 2. Moreover the figure indicates that median-based clustering performs much
better. Indeed, mean-based clustering does not possess any equivariance property, which implies
performance independent of the choice of A.
We proposed a novel overcomplete, median-based clustering method in (Theis et al., 2006),
and proved its equivariance and convergence. Simply put, we first pick 2n normalized starting
vectors w1, w ′ 1 , . . . , wn, w ′ n, and iterate the following steps until an appropriate abort condition
has been met: Choose a sample x(t) ∈ R m and normalize it y(t) := π(x(t)) = x(t)/|x(t)|. Let