Statistical Language Models based on Neural Networks - Faculty of ...

Table 4.5: Results on Penn Treebank corpus (evaluation set) after combining all models.
The weight of each model is tuned to minimize perplexity of the final combination.
Model Weight Model PPL
3-gram, Good-Turing smoothing (GT3) 0 165.2
5-gram, Kneser-Ney smoothing (KN5) 0 141.2
5-gram, Kneser-Ney smoothing + cache 0.079 125.7
Maximum entropy 5-gram model 0 142.1
Random clusterings LM 0 170.1
Random forest LM 0.106 131.9
Structured LM 0.020 146.1
Across sentence LM 0.084 116.6
Log-bilinear LM 0 144.5
Feedforward neural network LM [50] 0 140.2
Feedforward neural network LM [40] 0 141.8
Syntactical neural network LM 0.083 131.3
Combination of static RNNLMs 0.323 102.1
Combination of dynamic RNNLMs 0.306 101.0
ALL 1 83.5
4.5 Combination of all models
The most interesting experiment is to combine all language models together: <strong>based</strong> on
that, we can see which models can truly provide useful information in the state of the art
combination, and which models are redundant. It should be stated from the beginning
that we do not compare computational complexity or memory requirements of different
models, as we are only interested in achieving the best accuracy. Also, the conclusions
about accuracies of individual models and their weights should not be interpreted as that
the models that provide no complementary information are useless - further research can
prove otherwise.
Table 4.5 shows weights of all studied models in the final combination, when tuned
for the best performance on the development set. We do not need to use all techniques
to achieve optimal performance: weights of many models are very close to zero. The
combination is dominated by the RNN models, which together have a weight of 0.629. It
is interesting to realize that some individual models can be discarded completely without
hurting the performance at all. On the other hand, the combination technique itself is
58