Mathematics in Independent Component Analysis

Chapter 5. IEICE TF E87-A(9):2355-2363, 2004 105
2
2.1 Model
The overdetermined ICA model can be formulated as
follows: Let x be a given m-dimensional random vector.
An n × m-matrix W with m > n ≥ 2 is called
overdetermined ICA of x if
y = Wx (1)
is independent. In order to distinguish between overdetermined
and ordinary ICA, in the case m = n we call
W a square ICA of x.
Here W is not assumed to have full rank as usual.
Theorem 2.1 shows that under reasonable assumptions
this automatically holds.
Often overdetermined ICA is used to solve the
overdetermined BSS problem given by
x = As (2)
where s is an n-dimensional independent random vector
and A a m × n matrix with m > n ≥ 2. Note that
A can be assumed to have full rank (rank A = n),
otherwise the system could be reduced to the case n−1:
If A = (a1, . . . , an) with columns ai, in the case of
rank A < n we can without loss of generality assume
that an = � n−1
i=1 λiai. Then
x = As =
n�
j=1
n−1
ajsj =
�
(1 + λj)ajsj
j=1
so source sn does not appear in the mixture in this
case, thus the model can be reduced to the case n −
1. Overdetermined ICAs of x are usually considered
solutions to this BSS problem.
Often, overdetermined BSS is stated in the noisy
case,
x = As + ν (3)
where ν is a decorrelated Gaussian ’noise’ random vector,
independent of s. Without additional noise, the
sources can be found by solving for example the square
ICA problem, which is constructed from equation 1
after leaving away the last m − n observations given
non degenerate projected mixture matrix. In the presence
of noise usually projection by principal component
analysis (PCA) is chosen in order to reduce this problem
to the square case [14]. In the next section however,
the indeterminacies of the noise-free models represented
by equations 1 and 2 are studied because overdetermined
ICA will only be needed later after reduction of
the the bilinear model — and in this model we do not
allow any noise. However, the overdetermined noisy
model from equation 3 can easily be reduced to the
noise-free model by including ν as additional sources:
x = (A I)
� s
ν
�
.
IEICE TRANS. FUNDAMENTALS, VOL.E87–A, NO.9 SEPTEMBER 2004
In this case n increases and we could possibly deal with
underdetermined ICA (where extra care has to be taken
with the now increased number of Gaussians in the
sources). Uniqueness and separability results for this
case are given in [7], which shows that the following
theorems also hold in this noisy ICA model.
2.2 Indeterminacies
The following theorem presents the indeterminacy of
the unmixing matrix in the case of overdetermined mixtures,
with the slight generalization that this unmixing
matrix does not necessarily have to be of full rank.
Later in this section we show that it is necessary to assume
that the observed data set x is indeed a mixture.
Theorem 2.1 (Indeterminacies of overdetermined
ICA). Let m ≥ n ≥ 2. Let x = As as in
the model of equation 2, and let the n × m matrix W
be an overdetermined or square ICA of x such that at
most one component of Wx is deterministic. Furthermore
assume one of the following:
i. s has at most one Gaussian component and the
variances of s exist.
ii. s has no Gaussian component.
Then there exist a permutation matrix P and an invertible
scaling matrix L with
W = LP(A ⊤ A) −1 A ⊤ + C (4)
where C is an n × m matrix with rows lying the kernel
of A ⊤ that is with CA = 0. The converse also holds,
i.e. if W fulfills equation 4 then Wx is independent.
A less general form of this indeterminacy has been
given by Joho et al. [14]. In the square case, the above
theorem shows that it is not necessary to assume that
mixing and especially demixing matrix have full rank if
it is assumed that the transformation also has at most
one deterministic component. Obviously, this assumption
is not necessary if W is required to have full rank.
Since rank A = n, the pseudo inverse (Moore-
Penrose inverse) A + of A has the explicit form A + =
(A ⊤ A) −1 A ⊤ . Note that A + A = I. So from equation 4
we get WA = LP. We remark as corollary to the above
theorem that overdetermined ICA is separable, which
means that the sources are uniquely (except for scaling
and permutation) reconstructible, because for approximated
sources y we get y = Wx = WAs = LPs. This
of course is well-known because overdetermined BSS
can be reduced to square BSS (m = n) by projection;
yet the indeterminacies of the demixing matrix, which
are simple permutation and scaling in the square case,
are not so obvious for overdetermined BSS.
Proof of theorem 2.1. Consider B := WA and y :=
Bs. Let b1, . . . , bn ∈ R n denote the (transposed) rows
of B. Then