Mathematics in Independent Component Analysis

1.3. Dependent component analysis 13
1.3 Dependent component analysis
In this section, we will discuss the relaxation of the BSS model by taking into account additional
structures in the data and dependencies between components. Many researchers have taken
interest in this generalization, which is crucial for the application in real-world settings where
such situations are to be expected.
Here, we will consider model indeterminacies as well as actual separation algorithms. For
the latter, we will employ a technique that has been the basis of one of the first ICA algorithms
(Cardoso and Souloumiac, 1993), namely joint diagonalization (JD). It has since become an
important tool in ICA-based BSS and in BSS relying on second-order time-decorrelation (Belouchrani
et al., 1997). Its task is, given a set of commuting symmetric n × n matrices Ci, to
find an orthogonal matrix A such that A ⊤ CiA is diagonal for all i. This generalizes eigenvalue
decomposition (i = 1) and the generalized eigenvalue problem (i = 2), in which perfect
factorization is always possible.
Other extensions of the standard BSS model such as including singular matrices (Georgiev
and Theis, 2004) will be omitted from the discussion in the following.
1.3.1 Algebraic BSS and multidimensional generalizations
Considering the BSS model from equation (1.1)—or a more general, noisy version x(t) = As(t)+
n(t)—the data can only be separated if we put additional conditions on the sources such as:
• they are stochastically independent: ps(s1, . . . , sn) = ps1 (s1) · · · psn(sn),
• each source is sparse i.e. it contains a certain number of zeros or has a low p-norm for
small p and fixed 2-norm,
• s(t) is stationary and for all τ, it has diagonal autocovariances E(s(t + τ) s(t) ⊤ ); here
zero-mean s(t) are assumed.
In the following, we will review BSS algorithms based on eigenvalue decomposition, JD and
generalizations. Thereby, one of the above conditions is denoted by the term source condition,
because we do not want to specialize on a single model. The additive noise n(t) is modeled by
a stationary, temporally and spatially white zero-mean process with variance σ 2 . Moreover, we
will not deal with the more complicated underdetermined case, so we assume that at most as
many sources as sensors are to be extracted, i.e. n ≤ m.
The signals x(t) are observed, and the goal is to recover A and s(t). Having found A, s(t)
can be estimated by A † x(t), which is optimal in the maximum-likelihood sense. Here † denotes
the pseudo-inverse of A, which equals the inverse in the case of m = n. So the BSS task reduces
to the estimation of the mixing matrix A, hence the additive noise n is often neglected (after
whitening). Note that in the following we will assume that all signals are real-valued. Extensions
to the complex case are straightforward.
Approximate joint diagonalization
Many BSS algorithms employ joint diagonalization (JD) techniques on some source condition
matrices to identify the mixing matrix. Given a set of symmetric matrices C := {C1, . . . , CK},