Statistical Language Models based on Neural Networks - Faculty of ...

Table 7.4: Accuracy of different language modeling techniques on the Microsoft Sentence
Completion Challenge task. Human performance is 91% accuracy [83].
Model Perplexity Accuracy [%]
random - 20.0
GT3 92 36.0
GT5 87 38.4
KN5 84 40.0
RNNME-50 68 43.5
RNNME-100 65 45.1
RNNME-200 63 47.7
RNNME-300 60 49.3
models, the perplexity - with the difference that the sentence completion challenge focuses
on the informative words that occur infrequently. Results obtained with various n-gram
models and RNNME models are summarized in Table 7.4. The models were trained on
about 50M tokens using 200K vocabulary, as a link to the training data was provided
in [83].
RNNME language models perform much better that the usual n-gram models on this
task: obviously, their ability to represent longer context patterns is very useful. While
n-gram models perform about 20% better than is the random performance, the largest RN-
NME model is almost 30% better. Still, the performance is far from human performance,
which is 91% accuracy.
We can think of models that would focus more on the task itself - basic objective
function for usual language models is to minimize entropy of the training data, while in
the case of sentence completion challenge, we are more interested in capturing patterns
between infrequent words. A simple task-specific modification can involve models that
are trained on data where frequent words are discarded. This reduces amount of possible
parameters of n-gram models for capturing regularities between infrequent words. In the
following experiments, the 200 most frequent words were discarded both from the training
and test data.
It can be observed that n-gram models that are trained on such modified training
data give much better accuracy. However, as a lot of possibly important information
is discarded, the RNNME models do not have possibility to significantly overcome the
99