TESI DOCTORAL - La Salle

1.3. Multimodal clustering
in cluster l according to λ, nh,l denotes the number of objects in cluster h according to γ as
well as in group l according to λ, n is the number of objects contained in the data set, and
k is the number of clusters into which objects are clustered according to λ and γ (Strehl,
2002).
Thus, the more similar the clustering solutions represented by the label vector λ and
the ground truth γ, the closer to 1 φ (NMI) (γ, λ) will be. As the ground truth γ is assumed
to represent the true partition of the data, high quality clusterings will attain φ (NMI) (γ, λ)
values close to unity. As a consequence, given two label vectors λ1 and λ2, the former will
be considered to be better than the latter if φ (NMI) (γ, λ1) >φ (NMI) (γ, λ2), and vice versa.
1.3 Multimodal clustering
The ubiquity of multimedia data has motivated an increasing interest in clustering techniques
capable of dealing with multimodal data. In the following paragraphs we review
some of the most relevant works on clustering multimodal data.
Possibly one of the first works that mention the multimedia clustering problem was
(Hinneburg and Keim, 1998). The authors place special emphasis in highlighting the two
main challenges faced by clustering algorithms in this context: the high dimensionality
of the feature vectors and the existence of noise. To tackle these problems, the authors
proposed DENCLUE, a density-based clustering algorithm capable of dealing satisfactorily
with both issues. However, in that work, multimodality seemed to be more of a pretext to
justify the challenges of clustering high dimensional noisy data than an interest in itself.
This was not the case of the browsing and retrieval system for collections of text annotated
web images presented in (Chen et al., 1999), which was a multimodal extension
of the Scatter-Gather document browser of (Cutting et al., 1992). In this case, multiple
clusterings were created upon text and image features independently. In particular, clustering
on image features was employed as part of a query refinement process. Therefore, this
proposal is multimodal in the sense that the features of the distinct modalities are employed
for clustering the image collection, but, still, they were not fully integrated in the clustering
process.
In contrast, the true multimodality of the clustering approach proposed in (Barnard
and Forsyth, 2001) is guaranteed by modeling the probabilities of word and image feature
occurrences and co-occurrences. It consisted of a statistical hierarchical generative model
fitted with the EM algorithm, which organizes image databases using both image features
and their associated text. In subsequent works, the learnt joint distribution of image regions
and words was exploited in several applications, such as the prediction of words associated
with whole images or with image regions (Barnard et al., 2003).
Multimodal clustering has also been applied to the discovery of perceptual clusters for
disambiguating the semantic meaning of text annotated images (Benitez and Chang, 2002).
To do so, the images are clustered based on the visual or the text feature descriptors. Moreover,
the system could also conduct multimodal clustering upon any combination of text
and visual feature descriptors by conducting an early fusion of these. Principal Component
Analysis was used to integrate and to reduce the dimensionality of feature descriptors
before clustering. As regards the results obtained by multimodal clustering, the authors
highlighted the uncorrelatedness of visual and text feature descriptors, which suggests that
18