Mathematics in Independent Component Analysis

6 Chapter 1. Statistical machine learning of biomedical data
1.2 Uniqueness issues in independent component analysis
Application of ICA to BSS tacitly assumes that the data follow the model (1.1) i.e. x(t) admits
a decomposition into independent sources, and we want to find this decomposition. But neither
the mixing function f nor the source signals s(t) are known, so we should expect to find many
solutions for this problem. Indeed, the order of the sources cannot be recovered—the speakers
at the cocktail party do not have numbers—so there is always an inherent permutation indeterminacy.
Moreover, also the strength of each source cannot be extracted from this model alone,
because f and s(t) can interchange so-called scaling factors. In other words, by not knowing
the power of each speaker at the cocktail party, we can only extract his speech, but not the
volume—he could also be standing further away from the microphones, but shouting instead of
speaking.
One of the key questions in ICA-based source separation is the question: are there any other
indeterminacies? Without fully answering this question, ICA algorithms cannot be applied to
BSS, simply because we would not have any clue how to relate the resulting sources to the
original ones. But apparently, the set of indeterminacies cannot be very large—after all, at a
cocktail party, we ourselves are able to distinguish between the various speakers.
1.2.1 Linear case
In 1994, Comon was able to answer this question (Comon, 1994) in the linear case where f = A
by reducing it to the Darmois-Skitovitch theorem (Darmois, 1953, Skitovitch, 1953, 1954). Essentially,
he showed that if the sources contain at most one Gaussian component, the indeterminacies
of the above model are only scaling and permutation. This positive answer more or
less made the field popular; from then on the number of papers published in this field each year
increased considerably. However, it may be argued that Comon’s proof lacked two points: by
using the rather difficult to prove old theorem by the two statisticians, the central idea why
there are no more indeterminacies is not at all obvious. Hence not many attempts have been
made to extend it to more general situations. Furthermore, no algorithm can be extracted from
the proof, because it is non-constructive.
In (Theis, 2004a), we took a somewhat different approach. Instead of using Comon’s idea of
minimal mutual information, we reformulated the condition of source independence in a different
way: in simple terms, a two-dimensional source vector s is independent if its density ps factorizes
into two one-component densities ps1 and ps2 . But this is the case only if ln ps is the sum of
one-dimensional functions, each depending on a different variable. Hence taking the differential
with respect to s1 and then to s2 always yields zero. In other words, the Hessian Hln ps of
the logarithmic densities of the sources is diagonal—this is what we denoted by ps being a
‘separated function’ in Theis (2004a), see chapter 2. Using only this property, we were able to
prove Comon’s uniqueness theorem (Theis, 2004a, theorem 2) without having to resort to the
Darmois-Skitovitch theorem. Here Gl(n) defines the group of invertible (n × n)-matrices.
Theorem 1.2.1 (Separability of linear BSS). Let A ∈ Gl(n; R) and s be an independent random
vector. Assume that s has at most one Gaussian component and that the covariance of s exists.
Then As is independent if and only if A is the product of a scaling and permutation matrix.