Mathematics in Independent Component Analysis

1.2. Uniqueness issues in independent component analysis 7
Instead of a multivariate random process s(t), the theorem is formulated for a random vector
s, which is equivalent to assuming an i.i.d. process. Moreover, the assumption of equal source (n)
and mixture dimension (m) is made, although relaxation to the undercomplete case (1 < n < m)
is straightforward, and to the overcomplete case (n > m > 1) is possible (Eriksson and Koivunen,
2003). The assumption of at most one Gaussian component is crucial, since independence of
white, multivariate Gaussians is invariant under orthogonal transformation, so theorem 1.2.1
cannot hold in this case.
An algorithm for separation—Hessian ICA
The proof of theorem 1.2.1 is constructive, and the exception of the Gaussians comes into play
naturally as zeros of a certain differential equation. The idea, why separation is possible, becomes
quite clear now. Furthermore, an algorithm can be extracted from the pattern used in the proof:
After decorrelation, we can assume that the mixing matrix A is orthogonal. By using the
transformation properties of the Hessian matrix, we can employ the linear relationship x = As
to get
Hln px = A⊤Hln psA (1.2)
for the Hessian of the mixtures. The key idea, as we have seen in the previous section, is that
due to statistical independence, the source Hessian Hln ps is diagonal everywhere. Therefore
equation (1.2) represents a diagonalization of the mixture Hessian, and the diagonalizer equals
the mixing matrix A. Such a diagonalization is unique if the eigenspaces of the Hessian are onedimensional
at some point, and this is precisely the case if x(t) contains at most one Gaussian
component (Theis, 2004a, lemma 5). Hence, the mixing matrix and the sources can be extracted
algorithmically by simply diagonalizing the mixture Hessian evaluated at some point. The
Hessian ICA algorithm consists of local Hessian diagonalization of the logarithmic density (or
equivalently the easier to estimate characteristic function). In order to improve robustness,
multiple matrices are jointly diagonalized. Applying this algorithm to the mixtures from our
toy example from figure 1.1 yields very well recovered sources 1.1(c).
A similar algorithm has been proposed already by Lin (1998), but without considering the
necessary assumptions for successful algorithm application. In Theis (2004a, theorem 3), we
gave precise conditions for when to apply this algorithm and showed that points satisfying these
conditions can indeed be found if the sources contain at most one Gaussian component. Lin used
a discrete approximation of the derivative operator to approximate the Hessian; we suggested
using kernel-based density estimation, which can be directly differentiated. A similar algorithm
based on Hessian diagonalization had been proposed by Yeredor (2000) using the character of a
random vector. However, the character is complex-valued, and additional care has to be taken
when applying a complex logarithm. Basically, this is only well-defined locally at non-zeros.
In algorithmic terms, the character can be easily approximated by samples. Yeredor suggested
joint diagonalization of the Hessian of the logarithmic character evaluated at several points in
order to avoid the locality of the algorithm. Instead of joint diagonalization, we proposed to
use a combined energy function based on the previously defined separator. This also takes into
account global information, but does not have the drawback of being singular at zeros of the
density respectively character.