Statistical Language Models based on Neural Networks - Faculty of ...

Table 6.6: Perplexity with the new RNNME models, using more features and more memory.
Model PPL on dev PPL on eval PPL on eval + KN4
KN 4-gram baseline 144 140 140
RNNME-0, 1G features 157 - -
RNNME-0, 2G features 150 144 129
RNNME-0, 4G features 146 - -
RNNME-0, 8G features 142 137 126
RNNME-0, 16G features 140 135 125
RNNME-10, 8G features 133 127 120
RNNME-20, 8G features 124 120 115
RNNME-40, 8G features 120 117 112
old RNNME-40, 1G features 134 131 119
RNNME-0, 2G + skip-1 145 140 125
RNNME-0, 8G + skip-1 136 132 121
RNNME-10, 8G + 5-gram 133 128 120
training data.
• With more training data, performance of RNNME models also seems to degrade,
although more slowly than for RNN models.
• The RNNME-20 model trained on all data is better than RNN-80 model.
• Although this is not shown in the figure, even combination of RNN-80, ME and
baseline KN4 models is still much worse than RNNME-80 combined with the KN4.
On small data sets, the RNN model with small hidden layer can encode most of the
information easily - but on large data sets, the model must use a lot of parameters to
encode basic patterns that can be also described by normal n-grams. On the other hand,
RNNME architecture that uses n-gram features as part of the model focuses on discovering
complementary information to the n-grams. Thus, training neural network together with
some kind of n-gram model seems to be a crucial technique for successful application of
neural net language models to very large data sets, as training models with thousands of
hidden neurons seems to be intractable.
89