TESI DOCTORAL - La Salle

A.3. Data representations
Data set name
CAL500
(audio/text)
IsoLetters
(speech/image)
InternetAds
(object/collateral)
Corel
(image/text)
Number of Number of Number of
objects attributes per classes
(n) mode (d 1/d 2) (k)
Class imbalance
297 78/127 16 14.5%–1.3%
1559 617/16 26 3.8%–3.8%
2359 133/1425 2 83.8%–16.2%
3960 500/371 44 2.2%–2.3%
Table A.3: Summary of the multimodal data sets employed in the experimental sections of
this thesis. The “Class imbalance” column presents the percentage of objects in the data
set belonging to the most and least populated categories, respectively.
in the sense that some attributes are directly related to the object (as the image geometry
features, the caption and the alt text, which totalize 133 features) while the
remaining 1425 features are referred to collateral elements such as the anchor text or
the URL. We have removed those objects in the data set with missing features (28%
of the total), obtaining a reduced version of the InternetAds data set containing 2359
objects.
4. Corel: this is a rather classic multimodal data set (Duygulu et al., 2002), consisting of
5000 text-annotated images from 50 Corel Stock Photo CDs. Each CD contains 100
images on the same topic, such as “Sunrises and Sunsets”, “Mountains of America”
or “Wild Animals” (Bekkerman and Jeon, 2007). Experiments have been conducted
on the subset of 3960 images of the training subset of the Corel collection which are
assigned at least one topic. The visual modality is codified as follows (Duygulu et
al., 2002): images, represented using 33 color features, are segmented into regions,
and these regions are clustered into 500 smaller connected regions (aka blobs), which
are deemed visual terms, so that each image can be expressed in terms of these. As
regards the textual modality, every image has a caption (i.e. a brief description of the
scene) and an annotation (a list of objects appearing in the image). The vocabulary
contains 371 words, and the term vectors are parameterized using the tfidf weighting
scheme (van Rijsbergen, 1979).
A.3 Data representations
A.3.1 Unimodal data representations
As mentioned in section 1.1, in this work objects are expressed in terms of d numerical
attributes, so each object is represented as a column vector x ∈ Rd . Therefore, a whole
data set containing n objects is mathematically expressed by means of a d × n matrix X.
This original object representation is referred to as baseline throughout the thesis.
Starting from the baseline representation, four other object representations have been
224