Statistical Language Models based on Neural Networks - Faculty of ...

improvements actually increased - this can also be explained by the fact that the WSJ-JHU
setup is about 40x larger.
5.2 Kaldi WSJ Setup
Additional experiments on the Wall Street Journal task were performed using n-best lists
generated with an open source speech recognition toolkit Kaldi [60] trained on SI-84 data
further described in [62]. The acoustic models used in the following experiments were
<strong>based</strong> on triphones and GMMs. Several advantages of using Kaldi such as better re-
peatability of the performed experiments were already mentioned in the beginning of this
chapter (although Kaldi is still being developed, it should be easy to repeat the following
experiments with slightly better results, as RNN rescoring code is integrated in the Kaldi
toolkit). Note that this setup is also not the state of the art, as with more training data
and advanced acoustic modeling techniques, it is possible to get better baseline results.
Rescoring experiments with RNN LMs on a state of the art setup is subject of the following
chapter.
I used 1000-best lists generated by Stefan Kombrink in the following experiments. The
test sets are the same as for the JHU setup. This time I trained RNNME models to save
time - it is possible to achieve very good results even with tiny size of the hidden layer. For
the ME part of the model, I used unigram, bigram, trigram and fourgram features, with
hash size 2G parameters. The vocabulary was limited to 20K words used by the decoder.
Training data consisted of 37M tokens, from which 1% was used as heldout data. The
training data were shuffled to increase speed of convergence during training, however, due
to homogeneity of the corpus, the automatic sorting technique as described in Chapter 6
was not used. The results are summarized in Table 5.3.
It can be seen that RNNME models improve PPL and WER significantly even with
tiny size of the hidden layer, such as 10 neurons. However, for reaching top performance, it
is useful to train models as large as possible. While training of small RNNME models (such
as with less than 100 neurons in the hidden layer) takes around several hours, training the
largest models takes a few days. After combining all RNNME models, the performance
still improves; however, adding unsupervised adaptation resulted in rather insignificant
improvement - note that the Eval 92 contains 333 utterances and Eval 92 only 213, thus
there is noise in the WER results due to small amount of test data.
66