Statistical Language Models based on Neural Networks - Faculty of ...

Perplexity (Penn corpus)
130
125
120
115
110
105
100
95
0 5 10 15 20 25
Number of RNN models
RNN mixture
RNN mixture + KN5
Figure 4.2: Linear interpolation of RNN models trained with different random initialization
of the weights.
4.4 Comparison of Different Neural Network Architectures
As neural network language models seem to work very well, we will describe them and
compare their performances in more detail. As can be seen in Table 4.1, the performance
of neural network <strong>based</strong> models with feedforward architectures is almost identical. The
syntactical NNLM has an advantage of having more input features: it uses a syntactical
parser to obtain part of speech tags for words in a sentence. We can observe that a signif-
icant improvement was obtained by using these additional features. On the other hand,
application of this technique to new languages and domains might not be straightforward,
as it relies on a syntactical parser that has to be trained on (usually) hand-annotated
data. There has been a following work done by Emami [22], where it is shown how differ-
ent linguistic features affect results, both perplexity and word error rate. The conclusion
of that experiments seems to be that the linguistic features improve only perplexity, but
do not reduce the word error rate.
Additional input features make interpretation of the results quite difficult: it can be
seen that the PTB corpus is quite small, and thus having additional information from a
parser that is trained on additional data (POS tags of words) can be a great boost to the
results. However, with increased amount of the training data, such additional information
would be probably less useful. Thus, it would be more convincing if the results obtained
54