Statistical Language Models based on Neural Networks - Faculty of ...

Entropy
8.4
8.2
8
7.8
7.6
7.4
7.2
7
6.8
1 2 3 4 5 6 7 8
Epoch
KN4
RNN-40
RNNME-40 (1G)
RNNME-40 (8G)
Figure 6.10: Comparison of training progress of RNN model with 40 neurons and RNNME
with 40 neurons (1G hash and 8G hash). Entropy is calculated on the development set.
6.6.4 <strong>Language</strong> Learning by RNN
<strong>Statistical</strong> language modeling has been criticized by linguists, for example by Chomsky as
mentioned in the introduction, for inability to distinguish grammatical and ungrammatical
sentences that are completely novel. Chomsky’s famous examples were ’colorless green
ideas sleep furiously’ and ’furiously sleep ideas green colorless’. Unless we would use
enormous amount of the training data, the n-gram models will not be able to assign
different probability to these two sentences, although the first one is grammatical and
thus should be more likely than the second one.
For the following simple experiment, the language models introduced in the previous
sections were used - namely, the RNN-640 model and the large KN4 n-gram model trained
on the Broadcast News data. Interestingly, the n-gram model does not contain any bigrams
that would correspond to those found in the test sentences, thus it has to back off to
unigram statistics for estimation of probability of every word (except the first word that
is in the context of start of sentence symbol, and the end of sentence symbol - for these
cases, bigram statistics were used).
The difference in probability of the test sentences given by the n-gram model is just
minor, as can be seen in Table 6.7. On the other hand, the RNN-640 model assigns about
90