Mathematics in Independent Component Analysis

158 Chapter 10. IEEE TNN 16(4):992-996, 2005
994 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 4, JULY 2005
2) Normalizethecolumns ��Y � a IY FFFYxI of ˆI X �� a
��a���� and set 4b0.
Multiply each column �� by 0I if the first element of �� is negative.
3) Cluster ��Y� aIY FFFYxI in � 0 I groups qIY FFFYq�CI such
that for any � a IY FFFY�Y�� 0 �� ` 4YV�Y � P q�, and
�� 0 �� !4 for any �Y � belonging to different groups
4) Choseany �� P q� and put —� a ��. Thematrix e with
columns �—�� � �aI is an estimation of the mixing matrix, up to
permutation and scaling.
We should mention that the very sparse case in different settings is
already considered in the literature, but in more restrictive sense. In [6],
theauthors supposethat thesupports of theFourier transform of any
two source signals are disjoint sets—a much more restrictive condition
than our condition. In [1], theauthors supposethat for any sourcethere
exists a time-frequency window where only this source is nonzero and
that the time-frequency transform of each source is not constant on any
time-frequency window. We would like to mention that their condition
should include also the case when the the time-frequency transforms
of any two sources are not proportional in any time-frequency window.
Such a quantitative condition (without frequency representation) is presented
in our Theorem 2, condition ii).
�aI
Fig. 1. Original images.
Fig. 2. Mixed (observed) images.
Fig. 3. Estimated normalized images using the estimated matrix. The
signal-to-noiseratios with thesources from Fig. 1 are232, 239, and 228 dB,
respectively.
B. Identification of Sources
Theorem 3: (Uniqueness of Sparse Representation): Let r bethe
set of all � P � such that the linear system e� a � has a solution
with at least � 0 � CIzero components. If e fulfills A1), then there
exists a subset rH & rwith measure zero with respect to r, such
that for every � Pr�rHthis system has no other solution with this
property.
�
Proof: Obviously r is theunion of all
� 0 I a@�3Aa@@�0
These coefficients are uniquely determined if ��� does not
belong to the set rH with measure zero with respect to r
2.3)
(see Theorem 3);
Constructthesolution �� a ƒ@XY�A:itcontains!�Y� intheplace
�� for � aIYFFFY�0 I, the other its components are zero.
III. SCA
IA3@�0�CIA3 hyperplanes, produced by taking the linear hull of every
subsets of the columns of e with � 0 I elements. Let rH betheunion
of all intersections of any two such subspaces. Then rH has a measure
zero in r and satisfies the conclusion of the theorem. Indeed, assume
that � P r�rHand e� a e"� a �, where � and "� haveat least
� 0 � CIzeros. Since � TP rHY� belongs to only one hyperplane
produced as a linear hull of some � 0 I columns —� Y FFFY —� of
e. It means that the vectors � and "� have � 0 � CIzeros in places
with indexes in �IY FFFY�����IY FFFY��0I�. Now from theequation
e@�0"�A aHit follows that the �0I vector columns —� Y FFFY —�
of e are linearly dependent, which is a contradiction with A1).
From Theorem 3 it follows that the sources are identifiable generically,
i.e., up to a set with a measure zero, if they have level of sparse-
In this section, we develop a method for the complete solution of the
SCA problem. Now the conditions are formulated only in terms of the
data matrix ˆ.
Theorem 4: (SCA Conditions): Assumethat � � x and the
matrix ˆ P
ness grater than or equal to � 0�CI, and themixing matrix is known.
In the following, we present an algorithm, based on the observation in
Theorem 3.
Source Recovery Algorithm:
1) Identify thetheset of �-codimensional subspaces r produced
by taking the linear hull of every subsets of the columns of e
with � 0 I elements.
2) Repeat for � a 1toxX 2.1) Identify the space r Prcontaining �� Xa ˆ@XY�A, or,in
practical situation with presence of noise, identify the one to
which thedistancefrom �� is minimal and project �� onto
r to ���.
2.2) if r is produced by the linear hull of column vectors
—� Y FFFY —� , then find coefficients !�Y� such that
�0I
��� a !�Y�—� X
�2x satisfies the following conditions:
i) �
thecolumns of ˆ liein theunion r of different hy-
� 0 I
perplanes, each column lies in only one such hyperplane, each
hyperplane contains at least � columns of ˆ such that each
� 0 I of them are linearly independent;
ii) � 0 I
for each � P �IYFFFY�� there exist � a different
� 0 P
hyperplanes �r�Y�� �
�aI in r such that their intersection v� a
’ �
�aIr�Y� is 1-D subspace;
iii) any � different v� span thewhole � .
Then the matrix ˆ is representable uniquely (up to permutation and
scaling of thecolumns of e and rows of ƒ) in theform ˆ a eƒ,
where the matrices e P �2� and ƒ P �2x satisfy theconditions
A1) and A2), A3), respectively.
Proof: Let v� bespanned by —� and set e a �—�� � �aI. Condition
iii) implies that any hyperplane from r contains at most � 0 I vectors
from e. By i) and ii), it follows that these vectors are exactly � 0 I:
only in this case the calculation of the number of all hyperplanes by
� 0 I
�
ii) will givethenumber in i): � a@� 0 IA a
� 0 P � 0 I .Let
e be a matrix whose column vectors are all vectors from e (taken in
an arbitrary order). Since every column vector � of ˆ lies only in one
hyperplane from r, the linear system e� a � has uniquesolution,
which has at least � 0 � CIzeros (see the Proof of Theorem 3). Let
���� � �aI be � column vectors from ˆ, which span onehyperplane
from r, and � 0 I of them are linearly independent (such vectors
exist by i)). Then we have: e�� a ��, for some uniquely determined
vectors ��Y� aIYFFFY�0 I, which are linearly independent and have