Statistical Language Models based on Neural Networks - Faculty of ...

1
a
a
a
1
2
3
P(w(t)|*)
ONE
TWO
THREE
Figure 6.6: Maximum entropy model with unigram features.
w(t-1) P(w(t)|w(t-1))
ONE
TWO
THREE
B
ONE
TWO
THREE
Figure 6.7: Maximum entropy model with bigram features.
For example, assume a vocabulary V with three words, V=(ONE, TWO, THREE).
Figure 6.6 shows a graphical representation of a unigram maximum entropy model with
the given vocabulary. We can see that the model has three parameters, a1, a2, a3. The
probability distribution P (w(t)|history) is not conditioned on any previous information,
thus the model can be seen as a bias in a neural network model - the single input neuron
that has always activation 1.
Figures 6.7 and 6.8 illustrate how bigram and trigram features can be represented in a
maximum entropy model: in a case of a bigram model, we have a matrix B that connects
all possible previous words w(t−1) and current words w(t). In case of a trigram model, we
have to use all possible combinations of two previous words as inputs. Thus, the number
of parameters for a maximum entropy model with full feature set with order N is V N .
It is important to see that the input word w(t − 1) in a case of the bigram model will
cause activation of exactly one neuron at any given time, among neurons that represent
bigram connections (if we considered also out of vocabulary words, then there might be
even no active neuron). For the model with trigram features, the situation is the same
- again, we will have a single neuron active. Thus, for N-gram maximum entropy model
with a full feature set consisting of unigrams, bigrams, ..., N-grams, there will be N active
input neurons at any given time, if we do not consider out of vocabulary words.
The problem with such model representation is that for higher orders and for large
vocabularies, the V N term will become impractically large. Such full weight matrix would
actually represent all possible combinations of N words, and with finite amount of the
training data, most of these combinations will never be seen. Many other features would
83