Unsupervised Recursive Sequence Processing - Institute of ...

and
△c j = ɛ ′ · h σ (nhd(n j0 , n j )) · (C SOMSD (s i+1 , . . . , s t ) − c j )
with learning rates ɛ, ɛ ′ ∈ (0, 1). n j0 denotes the winner for sequence entry i.
As demonstrated in [11], a generalization of this approach to tree structures can
reliably model structured objects and their respective topological ordering.
We would like to point out that, although these approaches seem different, they
constitute instances of the same recursive computation scheme. As proved in [14],
the underlying recursive update dynamics comply with
d((s 1 , . . . , s t ), n j ) = η 1 ‖s 1 − w j ‖ 2 + η 2 ‖C(s 2 , . . . , s n ) − c j ‖ 2
in all the cases. Their specific similarity measures for weights and contexts are denoted
by the generic ‖ · ‖ expression. The approaches differ with respect to the
concrete choice of the context C: TKM and RSOM refer to only the neuron itself
and are therefore restricted to local fractal codes within the weight space; RecSOM
uses the whole map activation, which is powerful but also expensive and subject
to random neuron activations; SOMSD relies on compressed information, the location
of the winner. Note that also standard supervised recurrent networks can be
put into the generic dynamic framework by choosing the context as the output of
the sigmoidal transfer function [14]. In addition, alternative compression schemes,
such as a representation of the context by the winner content, are possible [37].
To summarize this section, essentially four different models have been proposed
for processing temporal information. The models are characterized by the way in
which context is taken into account within the map. The models are:
Standard SOM: no context representation; standard distance computation; standard
competitive learning.
TKM and RSOM: no explicit context representation; the distance computation
recursively refers to the distance of the previous time step; competitive learning
for the weight whereby (for RSOM) the averaged signal is used.
RecSOM: explicit context representation as N-dimensional activity profile of the
previous time step; the distance computation is given as mixture of the current
match and the match of the context stored at the neuron and the (recursively computed)
current context given by the processed time series; competitive learning
adapts the weight and context vectors.
SOMSD: explicit context representation as low-dimensional vector, the location
of the previously winning neuron in the map; the distance is computed recursively
the same way as for RecSOM, whereby a distance measure for locations
in the map has to be provided; so far, the model is only available for standard
rectangular Euclidean lattices; competitive learning adapts the weight and context
vectors, whereby the context vectors are embedded in the Euclidean space.
10