Mathematics in Independent Component Analysis

Chapter 20. Signal Processing 86(3):603-623, 2006 271
already eliminates the scaling indeterminacy. In order to use an ordinary ICA
algorithm, we simply have to add a ’postprocessing’ stage: to guarantee a
nonnegative matrix, column signs are flipped to have only (or as many as
possible) nonnegative column entries. Also note that statistical independence
meaning that the multivariate probability densities factorize is not related to
the synchrony in the firing of MUs [33] — otherwise overlapping MUs could
not be separated.
We finally want to remark that ICA can also be interpreted as sparse signal
decomposition method in the case of super-Gaussian sources. This follows from
the fact that a good and often-used contrast for ICA is given by maximization
of non-Gaussianity [34] — this can be approximately derived from the fact
that due to the central limit theorem, a mixture of independent sources tends
to be more Gaussian than the sources, so the process can be inverted by
maximizing non-Gaussianity. In our setting the sources are very sparse, hence
strongly non-Gaussian. An ICA decomposition is therefore closely related to
a decomposition into parts of maximal sparseness — at least if sparseness is
measured using kurtosis.
2.2.2 (Sparse) nonnegative matrix factorization
In contrast to other matrix factorization models such as PCA, ICA or SCA,
nonnegative matrix factorization (NMF) strictly requires both matrices A and
S to have nonnegative entries, which means that the data can be described
using only additive components. Such a constraint has many physical realizations
and applications, for instance in object decomposition [5]. If additionally
some sparseness constraints are put on A and S, we speak of sparse NMF, see
[14] for more details.
Typically, NMF is performed using a least-squares (Euclidean) contrast
E(A,S) = �X − AS� 2 , (2)
which is to be minimized. This optimization problem, albeit convex in each
variable separately, is not convex in both at the same time and hence direct
estimation is not possible. Paatero [35] minimizes (2) using a gradient
algorithm, whereas Lee and Seung [36] develop a multiplicative update rule
increasing algorithm performance considerably.
Although NMF has recently gained popularity due to its simplicity and power
in various applications, its solutions frequently fail to exhibit the desired sparse
object decomposition. Therefore, Hoyer [14] proposes a modification of the
NMF model to include sparseness. However, a simple modification of the cost
function (2) could yield undesirable local minima, so instead he chooses to
minimize (2) under the constraint of fixed sparseness of both A and S. Here,
10