Unsupervised Recursive Sequence Processing - Institute of ...

of neurons, preferably in a topology preserving manner. Prominent applications of
the SOM are WEBSOM for the retrieval of text documents and PicSOM for the
recovery and ordering of pictures [18,25]. Various alternatives and extensions to
the standard SOM exist, such as statistical models, growing networks, alternative
lattice structures, or adaptive metrics [3,4,19,27,28,30,33].
If temporal or spatial data are dealt with – like time series, language data, or DNA
strings – sequences of potentially unrestricted length constitute a natural domain for
data analysis and classification. Unfortunately, the temporal scope is unknown in
most cases, and therefore fixed vector dimensions, as used for standard SOM, cannot
be applied. Several extensions of SOM to sequences have been proposed; for
instance, time-window techniques or the data representation by statistical features
make a processing with standard methods possible [21,28]. Due to data selection or
preprocessing, information might get lost; for this reason, a data-driven adaptation
of the metric or the grid is strongly advisable [29,33,36]. The first widely used application
of SOM in sequence processing employed the temporal trajectory of the
best matching units of a standard SOM in order to visualize speech signals and the
variations of which [20]. This approach, however, does not operate on sequences
as they are; rather, SOM is used for reducing the dimensionality of single sequence
entries and acts as a preprocessing mechanism this way. Proposed alternatives substitute
the standard Euclidean metric by similarity operators on sequences by incorporating
autoregressive processes or time warping strategies [16,26,34]. These
methods are very powerful, but a major problem is their computational costs.
A fundamental way for sequence processing is a recursive approach. Supervised
recurrent networks constitute a well-established generalization of standard feedforward
networks to time series; many successful applications for different sequence
classification and regression tasks are known [12,24]. Recurrent unsupervised models
have also been proposed: the temporal Kohonen map (TKM) and the recurrent
SOM (RSOM) use the biologically plausible dynamics of leaky integrators [8,39],
as they occur in organisms, and explain phenomena such as direction selectivity
in the visual cortex [9]. Furthermore, the models have been applied with moderate
success to learning tasks [22]. Better results have been achieved by integrating these
models into more complex systems [7,17]. Recent more powerful approaches are
the recursive SOM (RecSOM) and the SOM for structured data (SOMSD) [10,41].
These are based on a richer and explicit representation of the temporal context: they
use the activation profile of the entire map or the index of the most recent winner.
As a result, their representation ability is superior to RSOM and TKM.
A proposal to put existing unsupervised recursive models into a taxonomy can be
found in [1,2]. The latter article identifies the entity ’time context’ used by the models
as one of the main branches of the given taxonomy [2]. Although more general,
the models are still quite diverse, and the recent developments of [10,11,35] are
not included in the taxonomy. An earlier, simple, and elegant general description of
recurrent models with an explicit notion of context has been introduced in [13,14].
2