Statistical Language Models based on Neural Networks - Faculty of ...

There is a great variability of approaches how one can choose the size of the popula-
tion, the probabilities that certain action will be taken, the way how both crossover and
mutation operators affect the individuals, how population in new epoch is exactly formed
etc. It is possible to even apply genetic algorithm to this optimization process itself, to
avoid the need to tune these constants; this is called meta-genetic programming.
Genetic programming can be successfully applied to small problems and it was reported
that many novel algorithms were found using these or similar approaches. However, for
large problems, the GP seems to be very inefficient; the hope is that through the recom-
bination of solutions using the crossover operator, the GP will find basic atomic functions
first, and these will be used later to compose more complex functions. However, it is
sometimes mentioned that GP itself is not really guaranteed to work better than other
simple search techniques, and clearly for difficult large problems, random search is very
inefficient as space of possible solutions increases exponentially with the length of their
description.
My own attempts to train recurrent neural network language models using genetic al-
gorithms were not very promising; even on small problems, the stochastic gradient descent
is orders of magnitude faster. However, certain problems cannot be efficiently learned by
SGD, such as storing some information for long time periods. For toy problems, RNNs
trained by genetic algorithms were able to easily learn patterns that otherwise cannot be
learned by SGD (such as to store single bit of information for 100 time steps). Thus, an
interesting direction for future research might be to combine GP and SGD.
8.3 Incremental Learning
The way humans naturally approach a complex problem is through its decomposition into
smaller problems that are solved separately. Such incremental approach can be also used
for solving many machine learning problems - even more, it might be crucial to learn com-
plex algorithms that are represented either by deep architectures, or by complex computer
programs. It can be shown that certain complex problems can be solved exponentially
faster, if additional supervision in the form of simpler examples is provided.
Assume a task to find a six digit number, with a supervision giving information if
the candidate number is correct or not. On average, we would need 106
2
attempts to find
the number. However, if we can search for the number incrementally with additional
106