TESI DOCTORAL - La Salle

Appendix A. Experimental setup
generated by applying the following well-known feature extraction techniques 2 : Principal
Component Analysis (PCA), Independent Component Analysis (ICA), Non-Negative Matrix
Factorization (NMF) and Random Projection (RP). Besides providing diversity as
regards data representation, these techniques are also employed with dimensionality reduction
purposes. In this work, we refer to the reduced dimensionality of the resulting feature
space by r, which takes a whole range of values in the interval [3,d]. As a result of each
feature extraction procedure, a batch of r × n matrices (X r PCA , Xr ICA , Xr NMF or Xr RP )are
obtained.
The following paragraphs are devoted to a brief description of the main concepts regarding
the aforementioned feature extraction techniques.
– Principal Component Analysis, which is one of the most typical feature extraction
techniques, is based on projecting the data onto a dimensionally reduced feature space
such that i) the newly obtained features are decorrelated, and ii) thevarianceofthe
original data is maximally retained. For these reasons, PCA is said to be capable of
removing data redundancies while keeping the most relevant information contained
in the data. There exist several ways for conducting PCA, from the eigenanalysis of
the covariance matrix of X (Jolliffe, 1986) to neural network approaches (Oja, 1992).
In this work, PCA is implemented by means of Singular Value Decomposition (SVD),
following a similar approach to that of Latent Semantic Analysis (Deerwester et al.,
1990). More specifically, given the d×n data matrix X, its SVD is expressed according
to equation (A.1).
X = U · Σ · V T
(A.1)
Matrix Σ contains the singular values of X ordered in decreasing order and the
columns of matrices U and V are the left and right singular vectors of X, respectively.
Dimensionality reduction is achieved by retaining the r largest singular values in Σ
and the corresponding columns of matrix V, so that the r ×n matrix Xr T
PCA = ΣrVr
–where Σr and Vr are the reduced version of the singular values and right singular
vectors matrices, respectively– will contain the location of the n objects in the
r-dimensional PCA space, where clustering is conducted.
– Independent Component Analysis (ICA) can be regarded as an extension of PCA
for non-Gaussian data (Xu and Wunsch II, 2005), in which the projected data components
are forced to be statistically independent—a stronger condition than PCA’s
decorrelation. Being tightly bound to the blind source separation problem (Hyvärinen,
Karhunen, and Oja, 2001), the application of ICA for feature extraction usually assumes
the existence of a generative model that, in its simplest version, defines the
observed data as the result of an unknown linear noiseless combination of r statistically
independent latent variables (the so-called independent components). The
goal of ICA algorithms is to recover the independent components making no further
2 We choose applying feature extraction over feature selection given its greater ease of application (Jain,
Murty, and Flynn, 1999), as our main goal is creating representational diversity, not elaborating on object
representations. In informal experiments not reported here, other object representations based on feature
selection plus change of basis (Srinivasan, 2002) were also tested but finally discarded as, in general terms,
gave rise to lower quality clustering results.
225