Proceedings - Österreichische Gesellschaft für Artificial Intelligence

most commonly used (Turney and Pantel, 2010).
The simple way of obtaining a raw word
co-occurrence count representation for N target
words is to consider C context words that occur
inside a window of length l positioned around
each occurrence of the target words. An accumulation
of the co-occurrences creates a wordoccurrence
matrix X C×N . Different context
sizes yield representations with different information.
Sahlgren (2006) notes that small contexts
(of a few words around the target word),
give rise to paradigmatic relationships between
words, whereas longer contexts find words with
syntagmatic relationship between them. For a
review on the current state of the art for vector
space models using word-document, wordcontext
or pair-pattern matrices using singular
value decomposition-based approaches in dimensionality
reduction, see Turney and Pantel (2010).
2.2 Word spaces with SVD, ICA and SENNA
The standard co-occurrence vectors for words can
be very high-dimensional even if the intrinsic dimensionality
of word context information is actually
low (Karlgren et al., 2008; Kivimäki et al.,
2010), which calls for an informed way to reduce
the data dimensionality, while retaining enough
information. In our experiments, we apply two
computational methods, singular value decomposition
(SVD) and ICA, to reduce the dimensionality
of the data vectors and to restructure the word
space.
Both the SVD and ICA methods extract components
that are linear mixtures of the original
dimensions. SVD is a general dimension reduction
method, applied for example in latent semantic
analysis (LSA) (Landauer and Dumais, 1997)
in the linguistic domain. The LSA method represents
word vectors in an orthogonal basis. ICA
finds statistically independent components which
is a stronger requirement and the emerging features
are easier to interpret than the SVD features
(Honkela et al., 2010).
Truncated SVD approximates the matrix
X C×N as a product UDV T in which D d×d is a
diagonal matrix with square roots of the d largest
eigenvalues of X T X (or XX T ), U C×d has the d
corresponding eigenvectors of XX T , and V N×d
has the d corresponding eigenvectors of X T X.
The rows ofV N×d give ad-dimensional representation
for the target words.
ICA (Comon, 1994; Hyvärinen et al., 2001)
represents the matrix X C×N as a product AS,
where A C×d is a mixing matrix, and S d×N contains
the independent components. The columns
for the matrix S d×N give a d-dimensional representation
for the target words. The FastICA
algorithm for ICA estimates the model in two
stages: 1) dimensionality reduction and whitening
(decorrelation and variance normalization),
and 2) rotation to maximize the statistical independence
of the components (Hyvärinen and Oja,
1997). The dimensionality reduction and decorrelation
step can be computed, for instance, with
principal component analysis or SVD.
We compare the results obtained with dimension
reduction to a set of 50 feature vectors from
a system called SENNA (Collobert et al., 2011) 1 .
SENNA is a labeling system suitable for several
tagging tasks: part of speech tagging, named
entity recognition, chunking and semantic role
labeling. The feature vectors for a vocabulary
of 130 000 words are obtained by using large
amounts of unlabeled data from Wikipedia. In
training, unlabeled data is used in a supervised
setting. The system is presented a target word in
its context of 5+5 (preceding+following) words
with a ’correct’ class label. An ’incorrect’ class
sample is constructed by substituting the target
word with a random one and keeping the context
otherwise intact. The results in the tagging
tasks are at the level of the state of the art, which
is why we want to compare these representations
with the direct evaluation tests.
2.3 Direct evaluation
In addition to indirect evaluation of vector space
models in applications, several tests for direct
evaluation of word vector spaces have been proposed
see e.g., Sahlgren (2006) and Bullinaria
and Levy (2007). First we describe the semantic
and syntactic category tests. Here, a category
means a group of words with a given class label.
The precisionP in the category task is calculated
according to (Levy et al., 1998). A centroid for
each category is calculated as an arithmetic mean
of the word vectors belonging to that category.
1 http://ronan.collobert.com/senna/
99
Proceedings of KONVENS 2012 (Main track: oral presentations), Vienna, September 20, 2012