TESI DOCTORAL - La Salle

5.3. Multimodal consensus clustering results
φ (NMI)
1
0.8
0.6
0.4
0.2
0
image
λ agglo−cos−upgma
c
E
CSPA
EAC
HGPA
MCLA
ALSAD
KMSAD
SLSAD
(a) Modality 1
φ (NMI)
speech
λ agglo−cos−upgma
c
1
0.8
0.6
0.4
0.2
0
E
CSPA
EAC
HGPA
MCLA
ALSAD
KMSAD
SLSAD
(b) Modality 2
φ (NMI)
image+speech
λ agglo−cos−upgma
c
1
0.8
0.6
0.4
0.2
0
E
CSPA
EAC
HGPA
MCLA
ALSAD
KMSAD
SLSAD
(c) Multimodal
φ (NMI)
1
0.8
0.6
0.4
0.2
0
λ c agglo−cos−upgma
E
CSPA
EAC
HGPA
MCLA
ALSAD
KMSAD
SLSAD
(d) Intermodal
Figure 5.3: φ (NMI) boxplots of the unimodal, multimodal and intermodal consensus clustering
solutions using the agglo-cos-upgma algorithm on the IsoLetters data set.
is a determining factor in that the qualities of the consensus clustering solutions λ image
c
are notably higher than those of λ speech
c
, as average relative percentage φ (NMI) differences
between both modalities of 57.4% are obtained. That is, despite consensus clustering manages,
in both modalities, to yield reasonable quality results, selecting a single modality for
clustering a multimedia data collection can be a highly suboptimal option —as it can totally
limit the quality of the obtained partition.
Thirdly, when consensus clustering is conducted on the multimodal modality resulting
from feature-level fusion (figure 5.3(c)), even better consensus clustering results are obtained
–in average relative φ (NMI) terms, a 13.6% better than those obtained on the image
modality–, which indicates the existence of positive synergies between both modalities on
this data collection.
And finally, if the λ image
c
, λ speech
c
and λ image+speech
c
consensus clustering solutions are
combined –figure 5.3(d)– pretty good clusterings are obtained, specially when the CSPA,
ALSAD and KMSAD consensus functions are employed (in these cases, relative φ (NMI) differences
with respect the image and image+speech modalities below 5% are obtained). For
the remaining consensus functions, the intermodal consensus clustering λc attains lower
φ (NMI) scores, thus constituting a trade-off between the consensus clusterings of the unimodal
and fused modalities.
Figure 5.4 presents the results of the same process, but executed on the multimodal
cluster ensemble E created by means of the direct-cos-i2 clustering algorithm. In this case,
we can observe a very similar behaviour to the one just reported. In this case, however,
the intermodal consensus clustering solution λc (see figure 5.4(d)) is, in some cases (e.g.
when consensus is based on CSPA), better (from a 3.1% to a 13.9% in relative percentage
φ (NMI) differences) than any of its unimodal and multimodal counterparts —figures 5.4(a)
to 5.4(c).
The quality of the unimodal, multimodal and intermodal consensus clustering solutions
obtained by the application of the multimodal DHCA on the cluster ensemble generated
upon the graph-cos-i2 clustering algorithm of the CLUTO toolbox are presented in figure
5.5. In this case, a larger performance uniformity among consensus functions is observed,
at least as far as the image and image+speech modalities are concerned (figures 5.5(a) and
5.5(c)). Otherwise, the consensus clusterings obtained upon the multimodal representation
142