Statistical Language Models based on Neural Networks - Faculty of ...

Table 5.1: Comparison of advanced language modeling techniques on the WSJ task (37M
training tokens).
Model Dev WER[%] Eval WER[%]
Baseline - KN5 12.2 17.2
Discriminative LM [79] 11.5 16.9
Joint LM [23] - 16.7
Static RNN 10.3 14.5
Static RNN + KN 10.2 14.5
Adapted RNN 9.7 14.2
Adapted RNN + KN 9.7 14.2
3 interpolated RNN LMs 9.5 13.9
Table 5.2: Comparison of results on the WSJ dev set (JHU setup) obtained with models
trained on different amount of the data.
# words PPL WER Improvement[%]
KN5 +RNN KN5 +RNN Entropy WER
223K 415 333 - - 3.7 -
675K 390 298 15.6 13.9 4.5 10.9
2233K 331 251 14.9 12.9 4.8 13.4
6.4M 283 200 13.6 11.7 6.1 14.0
37M 212 133 12.2 10.2 8.7 16.4
to the way humans process natural language. I believe that advanced techniques exist
that are able to model richer set of patterns in the language, and these should be actu-
ally getting increasingly better than n-grams with more training data. Thus, I performed
experiments to check if RNN LMs behave in this way.
Results with increasingly large subset of the training data for the WSJ-JHU task
are shown in Table 5.2. Both relative entropy reductions and relative word error rate
reductions are increasing with more training data. This is a very optimistic result, and
it confirms that the original motivation for using neural net language models was correct:
by using distributed representation of the history instead of the sparse coding, the neural
net models can represent certain patterns in the language more efficiently than the n-gram
models. The same results are also shown at Figure 5.1, where it is easier to see the trend.
64