Unsupervised Recursive Sequence Processing - Institute of ...

too much contortion. A global isometric embedding, however, is not possible in
general. Interestingly, for all such tessellations a data similarity measure is defined
and possibly non-isometric visualization in the 2D plane can be achieved. While 6
neighbors per neuron lead to standard Euclidean triangular meshes, for a grid with
7 neighbors or more, the graph becomes part of the 2-dimensional hyperbolic plane.
As already mentioned, exponential neighborhood growth is possible and hence an
adequate data representation can be expected for the visualization of domains with
a high connectivity of the involved objects. SOM with hyperbolic neighborhood
(HSOM) has already proved well-suited for text representation as demonstrated for
a non-recursive model in [29].
3 SOM for sequences (SOM-S)
In the following, we introduce the adaptation of SOMSD for sequences and the
general triangular grid structure, SOM for sequences (SOM-S). Standard SOMs
operate on a rectangular neuron grid embedded in a real-valued vector space. More
flexibility for the topological setup can be obtained by describing the grid in terms
of a graph: neural connections are realized by assigning each neuron a set of direct
neighbors. The distance of two neurons is given by the length of a shortest path
within the lattice of neurons. Each edge is assigned the unit length 1. The number of
neighbors might vary (also within a single map). Less than 6 neighbors per neuron
lead to a subsiding neighborhood, resulting in graphs with small numbers of nodes.
Choosing more than 6 neighbors per neuron yields, as argued above, an exponential
increase of the neighborhood size, which is convenient for representing sequences
with potentially exponential context diversification.
Unlike standard SOM or HSOM, we do not assume that a distance preserving embedding
of the lattice into the two dimensional plane or another globally parameterized
two-dimensional manifold with global metric structure, such as the hyperbolic
plane, exists. Rather, we assume that the distance of neurons within the grid
is computed directly on the neighborhood graph, which might be obtained by any
non-overlapping triangulation of the topological two-dimensional plane. 4 For our
experiments, we have implemented a grid generator for a circular triangle meshing
around a center neuron, which requires the desired number of neurons and the
neighborhood degree n as parameters. Neurons at the lattice edge possess less than
n neighbors, and if the chosen total number of neurons does not lead to filling up
the outer neuron circle, neurons there are connected to others in a maximum symmetric
way. Figure 1 shows a small map with 7 neighbors for the inner neurons,
and a total of 29 neurons perfectly filling up the outer edge. For ≥ 7 neighbors, the
exponential neighborhood increase can be observed, for which an embedding into
4 Since the lattice is fixed during training, these values have to be computed only once.
12