Statistical Language Models based on Neural Networks - Faculty of ...

Entropy per word on the WSJ test data
9
8.8
8.6
8.4
8.2
8
7.8
7.6
7.4
7.2
7
10 5
10 6
Training tokens
10 7
KN5
KN5+RNN
Figure 5.1: Improvements with increasing amount of training data - WSJ (JHU setup).
Note that size of the hidden layer is tuned for the optimal performance, and increases with
the amount of the training data.
5.1.3 Conclusion of WSJ Experiments (JHU setup)
The possible improvements increase with more training data on this particular setup. This
is a very positive result; the drawback is that with increased amount of the training data,
such as billions of words, the computational complexity of RNN models is prohibitively
large. However, we dealt with the computational complexity in the previous chapter, and
it should be doable to train good RNN models even on data sets with more than a billion
words by using the class-<strong>based</strong> RNNME architecture.
Similarly to the experiments with the Penn Treebank Corpus, I tried to achieve the
lowest possible perplexity. However, this time just two RNN LMs were used, and the
combination of models did include just static RNN LMs, dynamic RNN LMs (with a
single learning rate α = 0.1) and a Kneser-Ney smoothed 5-gram model with a cache.
Good-Turing smoothed trigram has perplexity 246 on the test data; the best combination
of models had perplexity 108 - this by more than 56% lower (entropy reduction 15.0%).
The 5-gram with modified Kneser-Ney smoothing has perplexity 212 on this task, thus
the combined result is by 49% lower (entropy reduction 12.6%). Thus, although the
combination experiments were much more restricted than in the case of PTB, the entropy
65
10 8