Connectionist Modeling of Experience-based Effects in Sentence ...

Chapter 3 Connectionist Modelling of Language Comprehension
their experiment in detail. MC02 used a standard simple recurrent network (SRN) with
a hidden and context layer of 60 units each. In- and output layers of 31 units each represented
30 words plus an end-of-sentence (EOS) symbol. The corpora each consisted
of 10,000 English sentences constructed randomly from a simple artificial probabilistic
context-free grammar (PCFG). Subject- or object-modifying relative clauses were contained
in 5% of the sentences. Half were subject extracted and half were object extracted
RCs. The rest of each corpus consisted of simple mono-clausal sentences. Verbs differed
by transitivity and shared a number agreement with their subject nouns. Each corpus
consisted of about 55,000 words. The sentence length was 3 to 27 words with a mean of
4.5. Notably relative clauses could be embedded recursively in each noun phrase. The
RC attachment probability in the PCFG (0.05) limited the embedding depth. MC02
trained 10 networks with randomly distributed initial weights 1 , each on a different corpus.
The learning rate was set to 0.1. The training phase covered only three epochs,
each consisting of one corpus length. The networks learned to predict the next word
in a sentence without being provided with any probabilistic information. The output
unit activations were calculated by a cross-entropy algorithm which ensured that all
activation values summed to one. In that way the networks’ output was comparable to
continuation likelihoods assigned to each possible word. After training the networks were
assessed on 10 sentences of all three types (SRC, ORC, and simple clause), respectively.
For interpreting the network output in terms of processing difficulty MC02 calculated
the so-called grammatical prediction error 2 (GPE). The GPE value is a measure
for the network’s difficulty in making the correct predictions on each word. The measure
was then used to map the relative word-by-word differences between the conditions on
reading times from the study by King and Just (1991). Besides RC type MC02 used
training epochs as a second factor. The network performances after one, two, and three
epochs of training were compared to low-, mid-, and high-span readers’ reading speed.
The results of MC02’s network simulation are shown in figure 3.3. Pooled over all
three epochs the results show a clear subject preference on the main verb (praised) and
the preceding region (embedded object in the SRC and embedded verb in the ORC).
Furthermore the ORC performance shows significant improvement on the embedded and
main verb through the three epochs of training. Notably, the SRC data does not show
such an improvement. Rather the performance was relatively good from the start with
no change during training. This indicates a clause type × exposure interaction. The
same interaction (in this case clause type × reading span) is seen in King and Just’s
empirical data (figure 2.1). Notably, the simple SRN model seems to make better predictions
than the CC-READER model by Just and Carpenter (1992) since CC-READER
captures the span effect but not the interaction with clause type (see figure 2.4). Importantly,
MC02 call the mentioned interaction a F requency × Regularity interaction.
Specifically, the regular nature of English SRCs with respect to word order (SVO) serves
1 Between -0.15 and 0.15.
2 See chapter 4 for a detailed description
52