Connectionist Modeling of Experience-based Effects in Sentence ...

OUTPUT
PUT PLAN
Chapter 3 Connectionist Modelling of Language Comprehension
vides much contextual information helping to interpret current input. The context of
an utterance has a great influence on ambiguity resolution and predictions of content.
There have been someThere accounts are many of providing ways in connectionist which this can networks be with temporal
representation which were accomplished, of explicit and nature. a number This posed of interesting limits to the number and richness
of representations. proposals Elman (1990) have appeared describesin athe simple literature way to (e.g. provide a connectionist
network with memory, called Jordan, a simple 1986; recurrent Tank & network Hopfield, (SRN, 1987; figure 3.1). The hidden
representations in the network Stornetta, are Hogg, copied & into Huberman, a so-called1987; context layer, which influences
the hidden representations Watrous in the next & Shastri, step through 1987; weighted Waibel, activation feeding. This
memory loop goes without Hanazawa, any explicit Hinton, representation Shikano, & of Lang, time1987; or relations between input
chunks. It is the iterative Pineda, procedure 1988; Williams of copying & Zipser, and back-feeding 1988). One itself that produces
temporal relations on anof implicit the most level. promising Becausewas every suggested copy of the by activation pattern has
been influenced by all earlier Jordan copies, (1986). the Jordan contextual described memory a network reaches into the “past” in a
continuously graded way (shown over several in Figure input1) steps. containing The information recurrent of earlier input representations
is still in the connections representation which aswere a trace, used but to associate newer input a has more influential
power. Elman (1990) writes: static pattern (a “Plan”) with a serially
ordered output pattern (a sequence of
“In this account, “Actions”). memory isThe neither recurrent passive connections nor a separate allowsubsystem.
One
cannot properly speak the network’s of a memory hidden forunits sequences; to see that its own memory is inextricably
bound up withprevious the rest output, of the processing so that the mechanism.” subsequent
behavior can be shaped by previous
This very simple wayresponses. of memory These supply recurrent yields connections architecturally are determined plausible
properties that can abstractly what give be described the network asmemory. storage limitations, memory span or decay
of memorized representations over time. These are properties explicitly accounted for in
symbolic models like ACT-R or CC-READER.
rchitecture used by Jordan (1986).
from output to state units are one-forxed
weight of 1.0. Not all connections
ach can be modified in
ing way. Suppose a
own in Figure 2) is
at the input level by
nits; call these Context
units are also “hidden”
se that they interact
with other nodes
he network, and not the
ld.
that there is a
nput to be processed,
clock which regulates
of the input to the
cessing would then
e following sequence of
time t, the input units
first input in the sequence. Each input might be a single scalar value or a vector,
n the nature of the problem. The context units are initially set to 0.5. 2 OUTPUT UNITS
HIDDEN UNITS
INPUT UNITS CONTEXT UNITS
Figure 2. A simple recurrent network in which activations are
Figure 3.1: copied Architecture from hidden of layer a simple to context recurrent layer network on a one-for-one (SRN, Elman, 1990). The
solid linebasis, represents with fixed fixed weight one-to-one of 1.0. Dotted connections lines represent to thetrainable context layer. Dashed lines
connections.
represent trainable connections.
Both the input
ntext units activate the hidden units; and then the hidden units feed forward to
48
tion function used here bounds values between 0.0 and 1.0.
Page 4