Statistical Language Models based on Neural Networks - Faculty of ...

”Every time I fire a linguist out of my group, the accuracy goes up 3 .”
We may understand Jelinek’s statement as an observation that with decreased com-
plexity of the system and increased generality of the approaches, the performance goes up.
It is then not so surprising to see the general purpose algorithms to beat the very specific
ones, although clearly the task specific algorithms may have better initial results.
Neural network language models will be described in more detail in Chapter 2. These
models are today among state of the art techniques, and we will demonstrate their per-
formance on several data sets, where on each of them their performance is unmatched by
other techniques.
The main advantage of NNLMs over n-grams is that history is no longer seen as exact
sequence of n − 1 words H, but rather as a projection of H into some lower dimensional
space. This reduces number of parameters in the model that have to be trained, resulting
in automatic clustering of similar histories. While this might sound the same as the
motivation for class <strong>based</strong> models, the main difference is that NNLMs project all words
into the same low dimensional space, and there can be many degrees of similarity between
words.
The main weak point of these models is very large computational complexity, which
usually prohibits to train these models on full training set, using the full vocabulary. I will
deal with these issues in this work by proposing simple and effective speed-up techniques.
Experiments and results obtained with neural network models trained on over 400M words
while using large vocabulary will be reported, which is to my knowledge the largest set
that a proper NNLM has been trained on 4 .
2.4 Introduction to Data Sets and Experimental Setups
In this work, I would like to avoid mistakes that are often mentioned when it comes to
criticism of the current research in the statistical language modeling. It is usually claimed
that the new techniques are studied in very specific systems, using weak or ambiguous
baselines. Comparability of the achieved results is very low, if any. This leads to much
3 Although later, Jelinek himself claimed that the original statement was ”Every time a linguist leaves
my group, the accuracy goes up”, the former one gained more popularity.
4 I am aware of experiments with even more training data (more than 600M words) [8], but the resulting
model in that work uses a small hidden layer, which as it will be shown later prohibits to train a model
with competitive performance on such amount of training data.
24