Mathematics in Independent Component Analysis

Chapter 10. IEEE TNN 16(4):992-996, 2005 157
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 4, JULY 2005 993
are satisfied (see Theorem 1). When the sources are locally very sparse
(see condition i) of Theorem 2) the matrix identification algorithm
is much simpler. We used this simpler form for separation of mixtures
of images. After sparsification transformation (which is in fact
appropriate wavelet transformation) the algorithm works perfectly in
the complete case. We demonstrate the effectiveness of our general
matrix identification algorithm and the source recovery algorithm
in the underdetermined case for 7 artificially created sparse source
signals, such that thesourcematrix ƒ has at most 2 nonzero elements
in each column, mixed with a randomly generated (3 2 7) matrix.
For a comparison, we present a recovery using �I-norm minimization
[3], [4], which gives signals that are far from the original ones. This
implies that the conditions which ensure equivalence of �I-norm and
�H-norm minimization [4], Theorem 7, are generally not satisfied for
randomly generated matrices. Note that �I-norm minimization gives
solutions which haveat most � nonzeros [3], [4]. Another connection
with [4] is the fact that our algorithm for source recovery works “with
probability one,” i.e., for almost all data vectors � (in measure sense)
such that thesystem � a e� has a sparsesolution with less than
� nonzero elements, this solution is unique, while in [4] the authors
proved that for all data vectors � such that thesystem � a e� has
a sparsesolution with less than Spark@eAaP nonzero elements, this
solution is unique. Note that Spark@eA � CI, where Spark@eA is
the smallest number of linearly dependent columns of e.
exist � indexes ���� � �aI &�IY FFFYx� such that any � 0 I vector
columns of �ƒ@XY��A� � �aI form a basis of the @�0 IA-dimensional coordinatesubspaceof
� with zero coordinates given by �IY FFFY���t.
Because of the mixing model, vectors of the form
�� a ƒ@�Y ��A—�Y � aIYFFFY� �Pt
II. BLIND SOURCE SEPARATION
In this section, we develop a method for completely solving the BSS
problem if the following assumptions are satisfied:
A1) themixing matrix e P s‚ �2� belong to the data matrix ˆ. Now, by condition A1) it follows that any
�0I of thevectors ����
has theproperty that any
square � 2 � submatrix of it is nonsingular;
A2) each column of the source matrix ƒ has at most �0I nonzero
elements;
A3) the sources are sufficiently rich represented in the following
sense: for any index set of � 0 � C Ielements
s a ��IY FFFY��0�CI� &�IYFFFY�� there exist at least �
column vectors of the matrix ƒ such that each of them has
zero elements in places with indexes in s and each � 0 I of
them are linearly independent.
A. Matrix Identification
We describe conditions in the sparse BSS problem under which we
can identify the mixing matrix uniquely up to permutation and scaling
of thecolumns. Wegivetwo typeof such conditions. Thefirst one
corresponds to the least sparsest case in which such identification is
possible. Further, we consider the most sparsest case (for small number
of samples) as in this case the algorithm is much simpler.
1) General Case—Full Identifiability:
Theorem 1: (Identifiability Conditions—General Case): Assume
that in the representation ˆ a eƒ thematrix e satisfies condition
A1), thematrix ƒ satisfies conditions A2) and A3) and only the matrix
ˆ is known. Then the mixing matrix e is identifiable uniquely up to
permutation and scaling of the columns.
Proof: It is clear that any column —� of themixing matrix lies
� 0 I
in the intersection of all hyperplanes generated by those
� 0 P
columns of e in which —� participates.
We will show that these hyperplanes can be obtained by the columns
of thedata ˆ under the condition of the theorem. Let t betheset of
all subsets of �IY FFFY�� containing � 0 I elements and let t Pt.
�
Notethat t consists of elements. We will show that the hy-
� 0 I
perplane (denoted by rt ) generated by the columns of e with indexes
from t can beobtained by somecolumns of ˆ. By A2) and A3), there
�0I
�aI are linearly independent, which implies
that they will span the same hyperplane rt . By A1) and theabove,
�
it follows that wecan cluster thecolumns of ˆ in groups
� 0 I
�
r�Y� a IYFFFY uniquely such that each group r� con-
� 0 I
tains at least � elements and they span one hyperplane rt for some
t� Pt. Now we cluster the hyperplanes obtained in such a way in the
smallest number of groups such that the intersection of all hyperplanes
in each group gives a single one-dimensional (1-D) subspace. It is clear
that such 1-D subspacewill contain onecolumn of themixing matrix,
the number of these groups is � and each group consists of
hyperplanes.
� 0 I
� 0 P
The proof of this theorem gives the idea for the matrix identification
algorithm.
Algorithm for Identification of the Mixing Matrix:
1) Cluster the columns of ˆ in
�
� 0 I
groups r�Y� a
�
IY FFFY such that the span of the elements of each
� 0 I
group r� produces one hyperplane and these hyperplanes are
different.
2) Cluster the normal vectors to these hyperplanes in the smallest
number of groups q�Y� a IYFFFY� (which gives the number
of sources �) such that the normal vectors to the hyperplanes in
each group q� liein a new hyperplane ” r�.
3) Calculatethenormal vectors ”—� to each hyperplane ” r�Y� a
IY FFFY�. Notethat the1-D subspacespanned by ”—� is theintersection
of all hyperplanes in q�. Thematrix ” e with columns
”—� is an estimation of the mixing matrix (up to permutation and
scaling of thecolumns).
2) Degenerate Case—Sparse Instances:
Theorem 2: (Identifiability Conditions—Locally Very Sparse Representation):
Assume that the number of sources is unknown and the
following:
i) for each index � a IYFFFY� there are at least two columns of
ƒ X ƒ@XY�IAYand ƒ@XY�PA which have nonzero elements only in
position � (so each source is uniquely present at least twice);
ii) ˆ@XY�A Ta ˆ@XY�A for any P ,any�a IYFFFYx and
any � aIYFFFYxY� Ta � for which ƒ@XY�A has morethat one
nonzero element.
Then the number of sources and the matrix e areidentifiable
uniquely up to permutation and scaling.
Proof: We cluster in groups all nonzero normalized column vectors
of ˆ such that each group consists of vectors which differ only by
sign. From conditions i) and ii), it follows that thenumber of thegroups
containing more that one element is precisely the number of sources �,
and that each such group will represent a normalized column of e (up
to sign).
In thefollowing, weincludean algorithm for identification of the
mixing matrix based on Theorem 2.
Algorithm for Identification of the Mixing Matrix in the Very Sparse
Case:
1) Remove all zero columns of ˆ (if any) and obtain a matrix ˆI P
�2x
.