TESI DOCTORAL - La Salle

Chapter 1. Framework of the thesis
they should be integrated in the knowledge extraction process.
In the multimodal clustering context, the field that has motivated the largest amount
of research efforts is the clustering of web image search results based not only on visual
features, but also using the surrounding text and also link information—as organizing the
results into different semantic clusters might facilitates users browsing (Cai et al., 2004).
In that work, each image returned by the search engine is represented using three kinds
of information: visual information, textual information and link information (text and link
data are recovered from the surroundings of the image). The rationale of this approach is
based on the fact that the textual and link based representations can reflect the semantic
relationship of images better than visual features. The proposed system implements a
two level clustering algorithm: in the first level, clustering is conducted using the textual
and link representation of images (separately or jointly). In the second level, clustering
is conducted on the images assigned to each cluster resulting from the previous stage. In
this case, low level visual features are employed to re-organize the images in the first level
clusters, so as to group visually similar images to facilitate users browsing.
A second paper dealing with web image search results clustering was (Gao et al., 2005).
In that work, a tripartite graph was used to model the relations among low-level features,
images and their surrounding texts. Thus, the method was formulated as a constrained
multiobjective optimization problem, which can be efficiently solved by semi-definite programming.
In a similar context, clustering was applied for image sense discrimination for web images
retrieved from ambiguous keywords (Loeff, Ovesdotter-Alm, and Forsyth, 2006). Its goal
was presenting the image search results in semantically sensible clusters for improved image
browsing. To do so, spectral clustering was applied on multimodal features: simple local and
global image features, and a bag of words representation of the text in the embedding web
page. Multimodal fusion was achieved by combining pairwise object similarities measured
on both image and textual features in the graph affinity matrix of the spectral clustering
algorithm.
Finally, the notion that each of the multiple modalities in a multimedia collection contributes
its own perspective to the collections organization was the driving force behind the
proposal in (Bekkerman and Jeon, 2007). That work presents the Comraf* model, a lightweight
version of combinatorial Markov random fields. In Comraf*, multimodal clustering
is faced as the problem of simultaneously constructing a partition of each data modality.
By clustering modalities simultaneously, the statistical sparseness of the data representation
can be overcome, obtaining a dense and smooth joint distribution of the modalities.
However, not every modality has to be clustered, as long as the so-called target modality
is.
The reader interested in multimedia indexing and retrieval is referred to the recent
and complete survey of (Chen, 2006), although it is mainly focused on text plus image
modalities.
1.4 Clustering indeterminacies
As mentioned at the end of section 1.1, the accomplishment of a knowledge discovery process
requires making several critical decisions at each of its stages, which may have to be re-
19