Unsupervised Recursive Sequence Processing - Institute of ...

5.2 Binary automata
The second experiment is also inspired by Voegtlin. A discrete 0/1-sequence generated
by a binary automaton with P (0|1) = 0.4 and P (1|0) = 0.3 shall be learned.
For discrete data, the specialization of a neuron can be defined as the longest sequence
that still leads to unambiguous winner selection. A high percentage of specialized
neurons indicates that temporal context has been learned by the map. In
addition, one can compare the distribution of specializations with the original distribution
of strings as generated by the underlying probability. Figure 3 shows the
specialization of a trained H-SOM-S. Training has been carried out with 3·10 6 presentations,
increasing the context influence (1 − η) exponentially from 0 to 0.06.
The remaining parameters have been chosen as in the first experiment. Finally, the
receptive field has been computed by providing an additional number of 10 6 test
iterations. Putting more emphasis on the context results in a smaller number of active
neurons representing rather long strings that cover only a small part of the total
input space. If a Euclidean lattice is used instead of a hyperbolic neighborhood,
the resulting quantizers differ only slightly, which indicates that the representation
of binary symbols and their contexts in the 2-dimensional output space representations
does barely benefit from exponential branching. In the depicted run, 64 of
the neurons express a clear profile, whereas the other neurons are located at sparse
locations of the input data topology, between cluster boundaries, and thus do not
win for the presented stimuli. The distribution corresponds nicely to the 100 most
characteristic sequences of the probabilistic automaton as indicated by the graph.
Unlike RecSOM (presented in [41]), also neurons at interior nodes of the tree are
expressed for H-SOM-S. These nodes refer to transient states, which are represented
by corresponding winners in the network. RecSOM, in contrast to SOM-S,
does not rely on the winner index only, but it uses a more complex representation:
since the transient states are spared, longer sequences can be expressed by
RecSOM. In addition to the examination of neuron specialization, the whole map
11
10
9
8
7
6
5
4
3
2
1
0
100 most likely sequences
H-SOM-S, 100 neurons
64 specialized neurons
Fig. 3. Receptive fields of a H-SOM-S compared to the most probable sub-sequences of the
binary automaton. Left hand branches denote 0, right is 1.
20