Mathematics in Independent Component Analysis

106 Chapter 5. IEICE TF E87-A(9):2355-2363, 2004
THEIS and NAKAMURA: QUADRATIC INDEPENDENT COMPONENT ANALYSIS
yi = b ⊤ i s. (5)
We will show that we can assume B to be invertible.
If B is not invertible, without loss of generality let
bn = �n−1 i=1 λibi with coefficients λi ∈ R. Then at least
one λi �= 0, otherwise yn = 0 is deterministic. Without
loss of generality let λ1 = 1. From equation 5 we then
get yn = �n−1 i=1 λiyi = y1+u with u := �n−1 i=2 λiyi independent
of y1. Application of the Darmois-Skitovitch
theorem [6, 16] to the two equations
y1 = y1
yn = y1 + u
shows that y1 is Gaussian or deterministic. Hence all
yi with λi �= 0 are Gaussian or deterministic. So we
may assume that y1, yn and u are square integrable.
Without loss of generality let those random variables
be centered. Then the cross-covariance of (y1, yn) can
be calculated as follows:
0 = E(y1y ⊤ n ) = E(y1y ⊤ 1 ) + E(y1u ⊤ ) = var y1,
so y1 is deterministic. Hence all yi with λi �= 0 are
deterministic, and then also yn. So at least two components
of y = Wx are deterministic in contradiction
to the assumption. Therefore B is invertible.
Using assumptions i) or ii) and the well-known
uniqueness result of square linear BSS — a corollary
[5, 7] of the Darmois-Skitovitch theorem, see [17] for a
proof without using this theorem — there exist a permutation
matrix P and an invertible scaling matrix L
with WA = LP. The properties of the pseudo inverse
then show that the equation
(LP) −1 WA = I
in W has solutions W = LP(A + + C ′ ) with C ′ A = 0,
or W = LPA + + C again with CA = 0.
The pseudo inverse is the unique solution of WA =
I with minimum Frobenius norm. In this case C = 0;
so if W is an ICA of x with minimal Frobenius norm,
then already W equals A + except for scaling and permutation.
This can be used for normalization. Using
singular value decomposition of A, this has been shown
and extended to the noisy case in [14].
Theorem 2.1 states that in the case where x is the
mixture of n sources,
s
A
.. x
W
W ′
. .
. .
y
.................... .
y ′
W ′ A
ignoring the always present scaling and permutation
(then y = y ′ = s), we have an n(m − n)-dimensional
affine vector space as ICAs of x i.e. (W − W ′ )A = 0.
However in the case of arbitrary x, the ICAs of x can
be quite unrelated. Indeed, in the diagram
x
W
W ′
..
. .
y
...
..
..
...
. .
y ′
∃B?
there does not always exist B that could make this
diagram commute (W ′ x = BWx), as for example in
the case m ≥ 2n and W projection along the first n
coordinates and W ′ projection along the last n ones
shows. In this case the square ICA argument in the
proof of theorem 2.1 cannot be applied, and W and W ′
do not necessarily have any relation. This of course is
not true in the case m = n, where uniqueness (but not
existence) can also be shown without explicitly knowing
that x is a mixture by inverting W. This could be
extended to the case n ≤ m < 2n to construct 2n − m
equations for y and y ′ .
2.3 Algorithm
The usual algorithm for finding an overdetermined ICA
is to first project x along its main principal components
to an n-dimensional random vector and to then
apply square linear ICA [14, 19]. In [19], the question
of where to place this projection stage (before or after
application of ICA) is posed and answered somewhat
heuristically. Here, a simple proof is given that in the
overdetermined BSS case, any possible ICA matrix factorizes
over (almost) any projection, so projecting first
and then applying ICA to recover the sources is always
possible.
Theorem 2.2. Let x = As as in the model of equation
2 such that s satisfies one of the conditions in theorem
2.1. Furthermore let W be an overdetermined ICA of
x. Then for almost all (in the measure sense) n × m
matrices V there exists a square ICA B of Vx such
that Wx = BVx.
Proof. Let V be an n×m matrix such that VA is invertible
— this is an open condition, so almost all matrices
are of this type. Then there exists a square ICA, say
B ′ of y := Vx. So B ′ V is an overdetermined ICA of x.
Applying separability of overdetermined ICA (corollary
of theorem 2.1) then proves that Wx = LPB ′ Vx for
a permutation P and a scaling L. Setting B := LPB ′
shows the claim.
In diagram-form, this means
A . .
s x y ′ W
..
V
. ..
y
. . ..
∃B
3