Unsupervised Recursive Sequence Processing - Institute of ...

2 Unsupervised processing of sequences
Let input sequences be denoted by s = (s 1 , . . . , s t ) with entries s i in an alphabet
Σ which is embedded in a real-vector space R n . The element s 1 denotes the most
recent entry of the sequence and t is the sequence length.The set of sequences of
arbitrary length over symbols Σ is Σ ∗ , and Σ l is the space of sequences of length l.
Popular recursive sequence processing models are the temporal Kohonen map,
recurrent SOM, recursive SOM, and SOM for structured data [8,11,39,41]. The
SOMSD has originally been proposed for the more general case of tree structure
processing. Here, only sequences, i.e. trees with a node fan-out of 1 are considered.
As for standard SOM, a recursive neural map is given by a set of neurons n 1 , . . . ,
n N . The neurons are arranged on a grid, often a two-dimensional regular lattice.
All neurons are equipped with weights w i ∈ R n .
Two important ingredients have to be defined to specify self-organizing network
models: the data metric and the network update. The metric addressed the question,
how an appropriate distance can be defined to measure the similarity of possibly
sequential input signals to map units. For this purpose, the sequence entries
are compared with the weight parameters stored at the neuron. The set of input signals
for which a given neuron i is closest, is called the receptive field of neuron i,
and neuron i is the winner and representative for all these signals within its receptive
field. In the following, we will recall the distance computation for the standard
SOM and also review several ways found in the literature to compute the distance
of a neuron from a sequential input. Apart from the metric, the update procedure or
learning rule for neurons to adapt to the input is essential. Commonly, Hebbian or
competitive 1 learning takes place, referring to the following scheme: the parameters
of the winner and its neighborhood within a given lattice structure are adapted
such that their response to the current signal is increased. Thereby, neighborhood
cooperation ensures a topologically faithful mapping.
Standard SOM relies on a simple winner-takes-all scheme and does not account
for the temporal structure of inputs. For a stimulus s i ∈ R n the neuron n j responds,
for which the squared distance
d SOM (s i , w j ) = ‖s i − w j ‖ 2 ,
s i ∈ R n
is minimum, where ‖ · ‖ is the standard Euclidean metric. Training starts with
randomly initialized weights w i and adapts the parameters iteratively as follows:
denote by n 0 the index of the winning neuron for the input signal s i . Assume a
function nhd(n j , n k ) which indicates the degree of neighborhood of neuron j and
k within the chosen lattice structure is fixed. Adaptation of all weights w j takes
1 We will use these two terms interchangeably in the following.
4