Unsupervised Recursive Sequence Processing - Institute of ...

from each other. For training and testing we have taken the concatenation of randomly
generated words, such preparing sequences of 3 · 10 6 and 10 6 input vectors,
respectively. The map has got a map radius of 5 and contains 617 neurons on an
hyperbolic grid. For the initialization and the training, the same parameters as in the
previous experiment were used, except for an initially larger neighborhood range of
14, corresponding to the larger map. Context influence was taken into account by
decreasing η from 1 to 0.8 during training. A number of 338 neurons developed a
specialization for Reber strings with an average length of 7.23 characters. Figure 5
shows that the neuron specializations produce strict clusters on the circular grid,
ordered in a topological way by the last character. In agreement with the grammar,
the letter T takes the largest sector on the map. The underlying hyperbolic lattice
gives rise to sectors, because they clearly minimize the boundary between the 7
classes. The symbol separation is further emphasized by the existence of idle neurons
between the boundaries, which can be seen analogously to large values in a
U-Matrix. Since neuron specialization takes place from the most common states
–which are the 7 root symbols– to the increasingly special cases, the central nodes
have fallen idle after they have served as signposts during training; finally the most
specialized nodes with their associated strings are found at the lattice edge on the
outer ring. Much in contrast to the such ordered hyperbolic target lattice, the result
for the Euclidean grid in figure 7 shows a neuron arrangement in the form of
polymorphic coherent patches.
Similar to the binary automata learning tasks, we have analyzed the map representation
by the reconstruction of the trained data by backtracking all possible context
sequences of each neuron up to length 3. Only 118 of all 343 combinatorially possible
trigrams are realized. In a ranked table the most likely 33 strings cover all
attainable Reber trigrams. In the log-probability plot 6 there is a leap between entry
number 33 (TSS, valid) and 34 (XSX, invalid), emphasizing the presence of the Reber
characteristic. The correlation of the probabilities of Reber trigrams and their
relative frequencies found in the map is 0.75. An explicit comparison of the probabilities
of valid Reber strings can be found in figure 8. The values deviate from the
true probabilities, in particular for cycles of the Reber graph, such as consecutive
letters T and S, or the VPX-circle. This effect is due to the magnification factor
different from 1 for SOM, which further magnifies when sequences are processed
in the proposed recursive manner.
5.4 Finite memory models
In a final series of experiments, we examine a SOM-S trained on Markov models
with noisy input sequence entries. We investigate the possibility to extract temporal
dependencies on real-valued sequences from a trained map. The Markov model
possesses a memory length of 2 as depicted in figure 9. The basic symbols are denoted
by a, b, and c. These are embedded in two dimensions, disrupted by noise, as
22