Mathematics in Independent Component Analysis

1.5. Machine learning for data preprocessing 33
1.5 Machine learning for data preprocessing
Machine learning denotes the task of computationally finding structures in data. Here we describe
some preprocessing techniques that rely on machine learning for denoising, dimension
reduction and data grouping (clustering).
1.5.1 Denoising
In many fields of signal processing the examined signals bear considerable noise, which is usually
assumed to be additive and decorrelated. For example in exploratory data analysis of medical
data using statistical methods like ICA, the prevalent noise greatly degrades the reliability of
the algorithms, and the underlying processes cannot be identified.
In Gruber et al. (2006), see chapter 15, we considered the situation, where a one-dimensional
signal s(t) ∈ R given at discrete timesteps t = 1, . . . , T is distorted as follows
sN(t) = s(t) + N(t), (1.15)
where N(t) are i.i.d. samples of a Gaussian random variable i.e. sN equals s up to additive
stationary white noise. Many denoising algorithms have been proposed for recovering s(t) from
its noisy observation sN(t), see e.g. Effern et al. (2000), Hyvärinen et al. (2001b), Ma et al.
(2000) to name but a few. Vetter et al. (2002) suggested an algorithm based on local linear
projective noise reduction. The idea was to observe the data in a high-dimensional space of
delayed coordinates
˜sN(t) := (sN(t), . . . , sN(t + m − 1)) ∈ R m
and to denoise the data locally through a projection on the lower dimensional subspace of the
deterministic signals.
We followed this approach and localized the problem by selecting k clusters of the delayed
time series { sN(t) ˜ | t = 1, . . . , n}. This can for example be done by a k-means cluster algorithm,
see section 1.5.3, which is appropriate for noise selection schemes based on the strength or the
kurtosis of the signal since the statistical properties do not depend on the signal structure.
After the local linearization, we may assume that the time-embedded signal can be linearly
decomposed into noise and a lower-dimensional signal subspace. We therefore analyzed these k
m-dimensional signals using PCA or ICA in order to determine the ‘meaningful’ components.
The unknown number of signal components in the high-dimensional noisy signal were determined
either by using Vetter’s MDL estimator or a 2-means clustering of the eigenvectors of the
covariance matrix (Liavas and Regalia, 2001). The latter gave a good estimate of the number of
signal components if the noise variances are not clustered well enough together but nevertheless
are separated for the signal by a large gap.
To reconstruct the noise reduced signal, we unclustered the data to get a signal ˜se : {1, . . . , n} →
Rm and then averaged over the candidates in the delayed data.
This idea has been illustrated in figure 1.16.
se(t) := 1
m−1 �
[ ˜se(t − i)] i . (1.16)
m
i=0