Statistical Language Models based on Neural Networks - Faculty of ...

were: 400 classes, hidden layer size up to 800 neurons. Other hyper-parameters such as
interpolation weights were tuned on the WSJ’92 set (333 sentences), and the WSJ’93 set
used for evaluation consists of 465 sentences.
Note that this setup is very simple as the acoustic models that were used to generate
n-best lists for this task were not the state of the art. Also, the corresponding language
models used in the previous research were trained just on limited amount of the training
data (37M-70M words), although by using more training data that are easily affordable
for this task, better performance can be expected. The same holds for the vocabulary - a
20K word list was used, although it would be simple to use more. Thus, the experiments
on this setup are not supposed to beat the state of the art, but to allow comparison to
other LM techniques and to provide more insight into the performance of the RNN LMs.
5.1.1 Results on the JHU Setup
Results with RNN models and competitive techniques are summarized in Table 5.1. The
best RNN models have very high optimal weight when combined with KN5 baseline model,
and actually by discarding the n-gram model completely, the results are not significantly
affected. Interpolation of three RNN models gives the best results - the word error rate
is reduced relatively by about 20%. Other techniques, such as discriminatively trained
language model and joint LM (structured model) provide smaller improvements, only
about 2-3% reduction of WER on the evaluation set.
The adapted RNN model is not evaluated as a dynamic RNN LM described in the
previous chapters, but simply a static model that is re-trained on the 1-best lists. This was
done due to performance issues; it becomes relatively slow to work with RNN models that
are continuously updated, especially in the n-best list rescoring framework. Adaptation
itself provides relatively small improvement, especially with the large models.
5.1.2 Performance with Increasing Size of the Training Data
It was observed by Joshua Goodman that with increasing amount of the training data,
improvements provided by many advanced language modeling techniques vanish, with
a possible conclusion that it might be sufficient to train basic n-gram models on huge
amounts of data to obtain good performance [24]. This is sometimes interpreted as an
argument against language modeling research; however, as was mentioned in the introduc-
tion of this thesis, simple counting of words in different contexts is far from being close
63