Unsupervised Recursive Sequence Processing - Institute of ...

R N is provided, N denoting the number of neurons, which explicitly represents
the contextual map activation of all neurons in the previous time step. Thus, the
temporal context is represented in this model in an N-dimensional vector space, N
denoting the number of neurons. One can think of the context as an explicit storage
of the activity profile of the whole map in the previous time step. More precisely,
distance is recursively computed by
d RecSOM ((s 1 , . . . , s t ), n j ) = η 1 ‖s 1 − w j ‖ 2 + η 2 ‖C RecSOM (s 2 , . . . , s t ) − c j ‖ 2
where η 1 , η 2 > 0.
C RecSOM (s) = (exp(−d RecSOM (s, n 1 )), . . . , exp(−d RecSOM (s, n N )))
constitutes the context. Note that this vector is almost the vector of distances of all
neurons computed in the previous time step. These are exponentially transformed
to avoid an explosion of the values. As before, the above distance can be decomposed
into two parts: the winner computation similar to standard SOM, and, as in
the case of RSOM and TKM, a term which assesses the context match. For Rec-
SOM the context match is a comparison of the current context when processing
the sequence, i.e. the vector of distances of the previous time step, and the expected
context c j which is stored at neuron j. That is to say, RecSOM explicitly stores context
vectors for each neuron and compares these context vectors to their expected
contexts during the recursive computation. Since the entire map activation is taken
into account, sequences of any given fixed length can be stored, if enough neurons
are provided. Thus, the representation space for context is no longer restricted by
the weight space and its capacity now scales with the number of neurons.
For RecSOM, training is done in Hebbian style for both weights and contexts. Denote
by n j0 the winner for sequence entry i, then the weight changes are
△w j = ɛ · h σ (nhd(n j0 , n j )) · (s i − w j )
and the context adaptation is
△c j = ɛ ′ · h σ (nhd(n j0 , n j )) · (C RecSOM (s i+1 , . . . , s t ) − c j )
The latter update rule makes sure that the context vectors of the winner neuron
and its neighborhood become more similar to the current context vector C RecSOM ,
which is computed when the sequence is processed. The learning rates are ɛ, ɛ ′ ∈
(0, 1). As demonstrated in [41], this richer representation of context allows a better
quantization of time series data. In [41], various quantitative measures to evaluate
trained recursive maps are proposed, such as the temporal quantization error and
the specialization of neurons. RecSOM turns out to be clearly superior to TKM and
RSOM with respect to these measures in the experiments provided in [41].
8