Statistical Language Models based on Neural Networks - Faculty of ...

By considering popular machine learning techniques from the computational point of
view, we can see that the principal component analysis, logistic regression and neural net-
works use small, fixed algorithms with huge amount of learnable parameters. Thus, these
techniques are still somewhere between general learning systems and simple databases.
Based on the experimental results described in the previous chapters, we can observe
that by allowing the function f to learn more complex patterns, the generalization ability
increases. Thus, neural networks with one hidden layer (with non-linear activation func-
tion) have potential to work better than logistic regression. From the computational point
of view, we can see the non-linearity in the neural networks as the IF command: we can-
not express regularity in the data such as IF OBJECT IS SMALL AND RED, THE OBJECT
IS APPLE without using the IF command, as weighted combination such as SMALL IS
APPLE, RED IS APPLE would classify any small object as an apple.
We do not have to stop with just a single non-linearity; it can be seen that some other
patterns can require two subsequent IF commands to be expressed efficiently. It is often
mentioned that single non-linearity in computational models is sufficient to approximate
any function - while this is true, such approximation can collapse to the basic database
approach; the logic in function f is then expressed in the form of large number of rules such
as IF (X=1.05 AND Z=2.11) THEN Y=1.14. Contrary, deeper architecture can represent
some patterns using much fewer rules, which leads to better generalization when new data
are presented as inputs.
A possible future work in machine learning can be to use models with more non-
linearities, such as deep neural networks that are composed of several layers of neurons
with non-linear activation function [6]. This way, it is possible to express efficiently more
patterns. Recurrent neural networks are another example of deep architecture. On the
other hand, while the increased depth of the computational models increases the number of
patterns that can be expressed efficiently, it becomes more difficult to find good solutions
using techniques such as simple gradient descent.
Even the deep neural network architectures are severely limited, as the number of
hidden layers (computational steps) is defined by the programmer in advance, as a hyper-
parameter of the model. However, certain patterns cannot be described efficiently by using
constant number of computational steps (for example any algorithm that involves loop
with variable amount of steps, or recursive functions). Additionally, feedforward neural
networks do not have ability to represent patterns such as memory; however, practice shows
104