Unsupervised Recursive Sequence Processing - Institute of ...

Training is carried out by presenting a pattern s = (s 1 , . . . , s t ), determining the
winner n j0 , and updating the weight and the context. Adaptation affects all neurons
on the breadth first search graph around the winning neuron according to their
grid distances in a Hebbian style. Hence, for the sequence entry s i , weight w j is
updated by △w j = ɛ · h σ (nhd(n j0 , n j )) · (s i − w j ). The learning rate ɛ is typically
exponentially decreased during training; as above, h σ (nhd(n j0 , n j )) describes the
influence of the winner n j0 to the current neuron n j as a decreasing function of
grid distance. The context update is analogous: the current context, expressed in
terms of neuron triangle corners and coordinates, is moved towards the previous
winner along a shortest path. This adaptation yields positions on the grid only.
Intermediate positions can be achieved by interpolation: if two neurons N i and N j
exist in the triangle with the same distance, the midway is taken for the flat grids
obtained by our grid generator. This explains why the update path, depicted as the
dotted line in figure 1, for the current context towards D 2 is via D 1 . Since the grid
distances are stored in a static matrix, a fast calculation of shortest path lengths is
possible. The parameter η in the recursive distance calculations controls the balance
between pattern and context influence; since initially nothing is known about the
temporal structure, this parameter starts at 1, thus indicating the absence of context,
and resulting in standard SOM. During training it is decreased to an application
dependent value that mediates the balance between the externally presented pattern
and the internally gained model about historic contexts.
Thus, we can combine the flexibility of general triangular and possibly hyperbolic
lattice structures with the efficient context representation as proposed in [11].
4 Evaluation measures of SOM
Popular methods to evaluate the standard SOM are the visual inspection, the identification
of meaningful clusters, the quantization error, and measures for topological
ordering of the map. For recursive self organizing maps, an additional dimension
arises: the temporal dynamic stored in the context representations of the map.
4.1 Temporal quantization error
Using ideas of Voegtlin [41] we introduce a method to assess the implicit representation
of temporal dependencies in the map, and to evaluate to which amount
faithful representation of the temporal data takes place. The general quantization
error refers to the distortion of each map unit with respect to its receptive field,
which measures the extent of data space coverage by the units. If temporal data are
considered, the distortion needs to be assessed back in time. For a formal definition,
assume that a time series (s 1 , s 2 , . . . , s t , . . .) is presented to the network, again
14