MIT Encyclopedia of the Cognitive Sciences - Cryptome

Figure 2. A generic recurrent network. The presence of cycles in the
graph distinguishes these networks from the class of feedforward
networks. Note that there is no requirement of reciprocity as in the
Hopfield or Boltzmann networks.
constraints between nodes. The resulting equations of constraint,
known as “mean field equations,” generally turn out
to once again have the form of the perceptron update rule,
although the sharp decision of Eq. (2) is replaced with a
smoother nonlinear function (Peterson and Anderson 1987).
Mean field methods can also be utilized with decreasing
T. If T is decreased to zero, this idea, referred to as “deterministic
annealing,” can be applied to the Hopfield network.
In fact deterministic annealing has become the method of
choice for the update of Hopfield networks, replacing the
simple dynamics of Eq. (2).
General recurrent networks are usually specified by
drawing a directed graph (see figure 2). In such graphs,
arbitrary connectivity patterns are allowed; that is, there is
no requirement that nodes are connected reciprocally. We
associate a real-valued weight J ij with the link from node j
to node i, letting J ij equal zero if there is no link.
At time t, the ith node in the network has an activation
value S i [t], which can either be a discrete value or a continuous
value. Generally the focus is on discrete-time systems
(see also AUTOMATA), in which t is a discrete index,
although continuous-time systems are also studied (see also
CONTROL THEORY). The update rule defining the dynamics
of the network is typically of the following form:
⎛ ⎞
S
i
[ t + 1]
= f⎜∑J ij
S
j
[] t ⎟,
⎝ ⎠
j
where the function f is generally taken to be a smooth nonlinear
function.
General recurrent networks can show complex patterns
of dynamic behavior (including limit cycles and chaotic patterns),
and it is difficult to place conditions on the weights
J ij that guarantee particular kinds of desired behavior. Thus,
researchers interested in time-varying behavior of recurrent
networks have generally utilized learning algorithms as a
method of “programming” the network by providing examples
of desired behavior (Giles, Kuhn, and Williams1994).
A general purpose learning algorithm for recurrent networks,
known as backpropagation-in-time, can be obtained
(3)
Recurrent Networks 711
Figure 3. An “unrolled” recurrent network. The nodes S i in the
recurrent network in Figure 2 are copied in each of T + 1 time
slices. These copies represent the time-varying activations S i [t].
The weights in each slice are time-invariant; they are copies of the
corresponding weights in the original network. Thus, for example,
the weights between the topmost nodes in each slice are all equal to
J ii , the value of the weight from unit S i to itself in the original
network.
by a construction that “unrolls” the recurrent network
(Rumelhart et al. 1986). The unrolled network has T + 1
layers of N nodes each, obtained by copying the N of the
recurrent network at every time step from t = 0 to t = T (see
figure 3). The connections in the unrolled network are feedforward
connections that are copies of the recurrent connections
in the original network. The result is an unrolled
network that is a standard feedforward network. Applying
the standard algorithm for feedforward networks, in particular
backpropagation, yields the backpropagation-in-time
algorithm.
Backpropagation-in-time and similar algorithms have a
general difficulty in training networks to hold information
over lengthy time intervals (Bengio, Simard, and Frasconi
1994). Essentially, gradient-based methods utilize the derivative
of the state transition function of the dynamic system,
and for systems that are able to hold information over
lengthy intervals this derivative tends rapidly to zero. Many
new ideas in recurrent network research, including the use
of embedded memories and particular forms of prior knowledge,
have arisen as researchers have tried to combat this
problem (Frasconi et al. 1995; Omlin and Giles 1996).
Finally, there has been substantial work on the use of
recurrent networks to represent finite automata and the
problem of learning regular languages (Giles et al. 1992).
—Michael I. Jordan
References
Bengio, Y., P. Simard, and P. Frasconi. (1994). Learning long-term
dependencies with gradient descent is difficult. IEEE Transactions
on Neural Networks 5: 157–166.
Frasconi, P., M. Gori, M. Maggini, and G. Soda. (1995). Unified
integration of explicit rules and learning by example in recurrent
networks. IEEE Transactions on Knowledge and Data
Engineering 7: 340–346.
Geman, S., and D. Geman. (1984). Stochastic relaxation, Gibbs
distributions, and the Bayesian restoration of images. IEEE
Transactions on Pattern Analysis and Machine Intelligence 6:
721–741.