Statistical Language Models based on Neural Networks - Faculty of ...

Table 7.5: Additional results on the Microsoft Sentence Completion Challenge task.
Model Accuracy [%]
filtered KN5 47.7
filtered RNNME-100 48.8
RNNME combination 55.4
n-gram models. These results are summarized in Table 7.5, where models trained on
modified training data are denoted as filtered. Combination of RNNME models trained
on the original and the filtered training data then provides the best result on this task so
far, about 55% accuracy.
As the task itself is very interesting and shows what language modeling research can
focus on in the future, the next chapter will include some of my ideas how good test sets
for measuring quality of language models should be created.
7.4 Speech Recognition of Morphologically Rich <strong>Language</strong>s
N-gram language models usually work quite well for English, but not so much for other
languages. The reason is that for morphologically rich languages, the number of word
units is much larger, as new words are formed easily using simple rules, by adding new
word ending etc. Having two or more separate sources of information (such as stem and
ending) in a single token increases amount of parameters in n-gram models that have to
be estimated from the training data. Thus, higher order n-gram models usually do not
give much improvement. Other problem is that for these languages, much less training
data is usually available.
To illustrate the problem, we have used the Penn Treebank Corpus as described in
Chapter 4, and added two bits of random information to every token. This should increase
perplexity of the model that is trained on these modified data by more than two bits, as
it is not possible to revert the process (the information that certain words are similar has
to be obtained just from the statistical similarity of occurrence).
As the n-gram models cannot perform any clustering, it must be expected that their
performance will degrade significantly. On the other hand, RNN models can perform clus-
tering well, thus the increase of entropy should be lower. Results with simple RNN models
with the same architecture and KN5 models with no discounts are shown in Table 7.6.
100