Mathematics in Independent Component Analysis
Mathematics in Independent Component Analysis
Mathematics in Independent Component Analysis
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
130 Chapter 8. Proc. NIPS 2006<br />
Towards a general <strong>in</strong>dependent subspace analysis<br />
Fabian J. Theis<br />
Max Planck Institute for Dynamics and Self-Organisation &<br />
Bernste<strong>in</strong> Center for Computational Neuroscience<br />
Bunsenstr. 10, 37073 Gött<strong>in</strong>gen, Germany<br />
fabian@theis.name<br />
Abstract<br />
The <strong>in</strong>creas<strong>in</strong>gly popular <strong>in</strong>dependent component analysis (ICA) may only be applied<br />
to data follow<strong>in</strong>g the generative ICA model <strong>in</strong> order to guarantee algorithm<strong>in</strong>dependent<br />
and theoretically valid results. Subspace ICA models generalize the<br />
assumption of component <strong>in</strong>dependence to <strong>in</strong>dependence between groups of components.<br />
They are attractive candidates for dimensionality reduction methods,<br />
however are currently limited by the assumption of equal group sizes or less general<br />
semi-parametric models. By <strong>in</strong>troduc<strong>in</strong>g the concept of irreducible <strong>in</strong>dependent<br />
subspaces or components, we present a generalization to a parameter-free<br />
mixture model. Moreover, we relieve the condition of at-most-one-Gaussian by<br />
<strong>in</strong>clud<strong>in</strong>g previous results on non-Gaussian component analysis. After <strong>in</strong>troduc<strong>in</strong>g<br />
this general model, we discuss jo<strong>in</strong>t block diagonalization with unknown block<br />
sizes, on which we base a simple extension of JADE to algorithmically perform<br />
the subspace analysis. Simulations confirm the feasibility of the algorithm.<br />
1 <strong>Independent</strong> subspace analysis<br />
A random vector Y is called an <strong>in</strong>dependent component of the random vector X, if there exists<br />
an <strong>in</strong>vertible matrix A and a decomposition X = A(Y,Z) such that Y and Z are stochastically<br />
<strong>in</strong>dependent. The goal of a general <strong>in</strong>dependent subspace analysis (ISA) or multidimensional <strong>in</strong>dependent<br />
component analysis is the decomposition of an arbitrary random vector X <strong>in</strong>to <strong>in</strong>dependent<br />
components. If X is to be decomposed <strong>in</strong>to one-dimensional components, this co<strong>in</strong>cides with ord<strong>in</strong>ary<br />
<strong>in</strong>dependent component analysis (ICA). Similarly, if the <strong>in</strong>dependent components are required<br />
to be of the same dimension k, then this is denoted by multidimensional ICA of fixed group size k<br />
or simply k-ISA. So 1-ISA is equivalent to ICA.<br />
1.1 Why extend ICA?<br />
An important structural aspect <strong>in</strong> the search for decompositions is the knowledge of the number of<br />
solutions i.e. the <strong>in</strong>determ<strong>in</strong>acies of the problem. Without it, the result of any ICA or ISA algorithm<br />
cannot be compared with other solutions, so for <strong>in</strong>stance bl<strong>in</strong>d source separation (BSS) would be<br />
impossible. Clearly, given an ISA solution, <strong>in</strong>vertible transforms <strong>in</strong> each component (scal<strong>in</strong>g matrices<br />
L) as well as permutations of components of the same dimension (permutation matrices P) give<br />
aga<strong>in</strong> an ISA of X. And <strong>in</strong>deed, <strong>in</strong> the special case of ICA, scal<strong>in</strong>g and permutation are already all<br />
<strong>in</strong>determ<strong>in</strong>acies given that at most one Gaussian is conta<strong>in</strong>ed <strong>in</strong> X [6]. This is one of the key theoretical<br />
results <strong>in</strong> ICA, allow<strong>in</strong>g the usage of ICA for solv<strong>in</strong>g BSS problems and hence stimulat<strong>in</strong>g<br />
many applications. It has been shown that also for k-ISA, scal<strong>in</strong>gs and permutations as above are<br />
the only <strong>in</strong>determ<strong>in</strong>acies [11], given some additional rather weak restrictions to the model.<br />
However, a serious drawback of k-ISA (and hence of ICA) lies <strong>in</strong> the fact that the requirement<br />
fixed group-size k does not allow us to apply this analysis to an arbitrary random vector. Indeed,