TESI DOCTORAL - La Salle

A.3. Data representations
assumption than their statistical independence. The application of ICA in feature extraction
is usually preceded by PCA with dimensionality reduction, as this procedure
is equivalent to the usual whitening step that simplifies ICA algorithms (Hyvärinen,
Karhunen, and Oja, 2001). Applying ICA on the PCA data yields an estimation of
the r independent latent variables which generated the observed data:
X r ICA = WX r PCA
(A.2)
where matrix W is known as the separating matrix. Equation (A.2) can be interpreted
as a linear transformation of the data through its projection on the basis vectors
contained in the rows of the separating matrix W. In this work, a version of the
FastICA algorithm (Hyvärinen, 1999) that maximizes the skewness of the data is
employed for obtaining the ICA representation of the data (Kaban and Girolami,
2000).
– Non-Negative Matrix Factorization (NMF) (Lee and Seung, 1999) is a feature extraction
technique based on linear representations of non-negative data—i.e. NMF can
only be applied when the original representation of the data is non-negative. Intuitively,
NMF can be interpreted as a linear generative model somewhat similar to that
of ICA but subject to non-negativity constraints, as it factorizes the non-negative data
matrix X into the approximate product of two non-negative matrices W and H, as
defined in equation (A.3). Thus, it can be argued that the data set is generated by
the sum of a set of the latent non-negative variables contained in matrix H, while the
elements of W are the weights of their linear combination.
X ≈ WH ,whereX r NMF
= H (A.3)
From a practical viewpoint, i) NMF is usually implemented by means of iterative
algorithms which try to minimize a cost function proportional to the reconstruction
error ||X−W·H|| (Lee and Seung, 2001), and ii) dimensionality reduction is achieved
by setting the respective sizes of the factorization matrices W and H to d × r and
r × n at the time of computing the approximate factorization of equation (A.3).
In this work, the NMF-based object representation XNMF is obtained by applying a
mean square reconstruction error minimization algorithm from NMFPACK, a software
package for NMF in Matlab (Hoyer, 2004). Besides its use as a feature extraction
technique, the vision of NMF as a means for obtaining a parts-based description of
the data has motivated alternative NMF-based clustering strategies (Xu, Liu, and
Gong, 2003; Shahnaz et al., 2004), alongside studies on the theoretical connections
between NMF and classic clustering approaches (Ding, He, and Simon, 2005).
Compared to PCA and ICA, NMF is advantageous as the non-negativity of its basis
vectors favours their interpretability. On the flip side, the derivation of the NMF
representation is usually more computationally demanding.
– Random Projection (RP) is a computationally efficient dimensionality reduction technique,
proposed as an alternative to those feature extraction techniques that become
too costly when the dimensionality of the original representation (d) isveryhigh
(Kaski, 1998). The rationale behind RP is pretty straightforward: a dimensionality
reduction method is effective as long as the distance between the objects in the
226