Statistical Language Models based on Neural Networks - Faculty of ...

Table 4.7: Results on Penn Treebank corpus (evaluation set) with different linear interpolation
techniques.
Model PPL
Static LI of all models 83.5
Static LI of all models + dynamic RNNs with α = 0.5 80.5
Adaptive LI of all models + dynamic RNNs with α = 0.5 79.4
4.5.1 Adaptive Linear Combination
All the experiments above use fixed weights of models in the combination, where the
weights are estimated on the PTB validation set. We have extended the usual linear
combination of models to a case when weights of all individual models are variable, and are
estimated during processing of the test data. The initial distribution of weights is uniform
(every model has the same weight), and as the test data are being processed, we compute
optimal weights <strong>based</strong> on the performance of models on the history of the last several words
(the objective is to minimize perplexity). In theory, the weights can be estimated using
the whole history. However, we found that it is possible to use multiple lengths of history
- a combination where interpolation weights are estimated using just a few preceding
words can capture short context characteristics that can vary rapidly between individual
sentences or paragraphs, while a combination where interpolation weights depend on the
whole history is the most robust.
It should be noted that an important motivation for this approach is that a combination
of adaptive and static RNN models with fixed weights is suboptimal. When the first word
in the test data is processed, both static and adaptive models are equal. As more data is
processed, the adaptive model is supposed to learn new information, and thus its optimal
weight can change. If there is a sudden change of topic in the test data, the static model
might perform better for several sentences, while if there are repeating sentences or names
of people, the dynamic model can work better.
Further improvement was motivated by the observation that adaptation of RNN models
with the learning rate α = 0.1 leads usually to the best individual results, but models in
combination are more complementary if some are processed with larger learning rate. The
results are summarized in Table 4.7. Overall, the adaptive learning rate provides small
improvement, and has an interesting advantage: it does not require any validation data
for tuning the weights of individual models.
60