Unsupervised Recursive Sequence Processing - Institute of ...

In a trained map, neurons spread in regions of the data space where a high sample
density can be observed, resulting in large U-values at borders between clusters.
Consequently, the U-Matrix forms a 3D landscape on the lattice of neurons with
valleys corresponding to meaningful clusters and hills at the cluster borders. The
U-Matrix of weight vectors can be constructed also for SOM-S. Based on this matrix,
the sequence entries can be clustered into meaningful categories, based on
which the extraction of Markov models as described above is possible. Note that
the U-Matrix is built by using the weights assigned to the neurons only, while the
context information of SOM-S is yet ignored. 6 However, since context information
is used for training, clusters emerge which are meaningful with respect to the
temporal structure, and this way they contribute implicitly to the topological ordering
of the map and to the U-Matrix. Partially overlapping, noisy, and ambiguous
input elements are separated during the training, because the different temporal
contexts contain enough information to activate and produce characteristic clusters
on the map. Thus, the temporal structure captured by the training allows a reliable
reconstruction of the input sequences, which could not have been achieved by the
standard SOM architecture.
5 Experiments
5.1 Mackey-Glass time series
The first task is to learn the dynamic of the real-valued chaotic Mackey-Glass time
series dx = bx(τ) + ax(τ−d) using a = 0.2, b = −0.1, d = 17. This is the same
dτ 1+x(τ−d) 10
setup as given in [41] making a comparison of the results possible. 7 Three types
of maps with 100 neurons have been trained: a 6-neighbor map without context
giving standard SOM, a map with 6 neighbors and with context (SOM-S), and
a 7-neighbor map providing a hyperbolic grid with context utilization (H-SOM-
S). Each run has been computed with 1.5 · 10 5 presentations starting at random
positions within the Mackey-Glass series using a sample period of ∆t = 3; the
neuron weights have been initialized white within [0.6, 1.4]. The context has been
considered by decreasing the parameter from η = 1 to η = 0.97. The learning rate
is exponentially decreased from 0.1 to 0.005 for weight and context update. Initial
neighborhood cooperativity is 10 which is annealed to 1 during training.
Figure 2 shows the temporal quantization error for the above setups: the temporal
quantization error is expressed by the average standard deviation of the given sequence
and the mean unit receptive field for 29 time steps into the past. Similar
6 Preliminary experiments indicate that the context also orders topologically and yields
meaningful clusters. The number of neurons in context clusters is thereby small compared
to the number of neurons and statistically significant results could not be obtained.
7 We would like to thank T.Voegtlin for providing data for comparison.
18