Statistical Language Models based on Neural Networks - Faculty of ...

WER [%]
12.5
12
11.5
11
10.5
10
9.5
9
10 0
10 1
Figure 5.2: WER on Eval 92 after rescoring with increasing size of N-best list, the baseline
is obtained with 5-gram model.
probably more useful to either produce wider lattices to allow more diverse paths to be
encoded in the lattice, or to use neural net language models directly during decoding.
5.2.1 Approximation of RNNME using n-gram models
As was described in Section 3.4.3, it is possible to approximate complex generative lan-
guage models by sampling huge amount of data and building usual n-gram models <strong>based</strong>
on the generated data. This can be seen as an attempt to precompute large lookup ta-
ble for likely n-gram entries. To demonstrate potential of this technique which allows
trivial integration of RNN and RNNME models directly into decoders, 15 billion words
were generated from RNNME-480 model. A Good-Turing smoothed 5-gram model was
built on top of these data, and various results are reported in Table 5.5. It should be
noted that while using modified Kneser-Ney smoothing provides slightly better results for
standalone models <strong>based</strong> on the generated data, the results after interpolation with the
baseline 5-gram model are worse than if Good-Turing smoothing is used.
N
Based on the results reported in Table 5.5, it is possible to obtain around 0.6% WER
reduction by rescoring lattices using models that were trained on additional data that
were generated from the RNNME-480 model. However, the n-gram model <strong>based</strong> on the
68
10 2
10 3