Statistical Language Models based on Neural Networks - Faculty of ...

group that we are interested in.
w(t)
s(t-1)
U
s(t)
V
W X
y(t)
c(t)
Figure 3.4: Factorization of the output layer.
The original idea can be tracked back to Goodman [25], who used classes for speeding
up training of the maximum entropy models. Figure 3.4 illustrates this approach: first, the
probability distribution over classes is computed. Then, a probability distribution for the
words that belong to the specific class are computed. So instead of computing V outputs
and doing softmax over V elements, only C + V ′ outputs have to be computed, and the
softmax function is applied separately to both C and V ′ , where C are all the classes, and
V ′ are all words that belong to the particular class. Thus, C is constant and V ′ can be
variable.
I have proposed an algorithm that assigns words to classes <strong>based</strong> just on the unigram
frequency of words [50]. Every word wi from the vocabulary V is assigned to a single
ci. Assignment to classes is done before the training starts, and is <strong>based</strong> just on relative
frequency of words - the approach is commonly referred to as frequency binning. This
results in having low amount of frequent words in a single class, thus frequently V ′ is
small. For rare words, V ′ can still be huge, but rare words are processed infrequently.
This approach is much simpler than the previously proposed ones such as using Wordnet
for obtaining the classes [57], or learning hierarchical representations of the vocabulary [54].
As we will see in the next chapter, the degradation of accuracy of models that comes from
using classes and the frequency binning approach is small.
Following the notation from section 3.2, the computation between the hidden and the
38