Statistical Language Models based on Neural Networks - Faculty of ...

Table 6.5: Word error rate on the RT04 evaluation set after lattice rescoring with various
models, with and without interpolation with the baseline 4-gram model.
Model WER[%]
Single Interpolated
KN4 (baseline) 13.11 13.11
model M - 12.49
RNN-40 13.36 12.90
RNN-80 12.98 12.70
RNN-160 12.69 12.58
RNN-320 12.38 12.31
RNN-480 12.21 12.04
RNN-640 12.05 12.00
RNNME-0 13.21 12.99
RNNME-40 12.42 12.37
RNNME-80 12.35 12.22
RNNME-160 12.17 12.16
RNNME-320 11.91 11.90
3xRNN - 11.70
performs well with more data, as shown in Figure 6.11. It should be noted that models
in these experiments were trained on subsets from the Reduced-Sorted data set, and thus
some of the observed improvement also comes from the adaptation effect.
Additional experiments were performed using training data with randomized order of
sentences - this is important to remove the adaptation effect when models are trained
on sorted data, as this time we are interested in comparison of performance of RNN and
RNNME models trained on large homogeneous data sets. Also, the baseline KN4 model
does not use any count cutoffs or pruning for the following experiments. Figure 6.12 shows
several interesting results:
• Even the hash-<strong>based</strong> ME model with simple classes can provide significant improve-
ment over the best n-gram model, and the improvement seems to be slowly increasing
with more data.
• The improvements from RNN models with fixed size are still vanishing with more
88