Connectionist Modeling of Experience-based Effects in Sentence ...

Chapter 1 Preliminaries
predictor. This shows – as Mitchell et al. (1995) also admit – that the exposure-based
approach has to find a balance between coarse and more fine grained measures and that
different structures might require different grain sizes for the tabulation of frequencies.
Structural Frequency in a Connectionist Network
A type of models that specifically base their predictions on the records of structural
frequencies are connectionist network models. Mitchell et al. (1995) say that “in a
connectionist system the records would consist of a set of activation weights distributed
throughout the network.” (p. 472). This is partly true. The network does not explicitly
count frequencies, nor are frequencies stored anywhere in the network. It is rather that
every exposure of the network to a specific structure immediately changes the weight
distributions and thus the whole behavior of the network. So, one could say that the
weight distributions contain implicit structural knowledge. This can be observed, for
example, in the activations in hidden layers. Inputs of similar structures result in similar
activation patterns in the hidden layer of simple recurrent networks (SRN, Elman, 1990).
The comparison of these patterns reveals the structural generalization levels that drive
the networks predictions. Different from symbolic exposure-based accounts there is no
explicitly fixed structural grain size the network is sensitive to. There is of course a
limit to the lowest grain size, which is defined by the encoding level of the input. If
the input string is encoded on word level, the network has no information below that
level to work with. The upper limit depends on the networks architecture and can be
affected by the size of the hidden layer, the learning mechanism, and specifically for
recurrent networks by their “memory span”. What levels the network actually choses
is hard to say in advance. Learning is a walk through a state space trying to find the
optimal solution to the desired input-output pairing. The choice of grain size is part
of that optimization process and can change during the learning phase. A commitment
to a specific grain size implicitly involves a commitment to the number of structures to
distinguish. A fine-grained model will consequently have to keep apart lots of structural
representations, while a very coarse-grained model has only few structures to deal with.
This relation means that a network with very few hidden nodes where the information
has to be passed through will only be able to do very high level generalizations. The
final choice of grain size will ideally be the most useful structural level of the internal
input representation to meet the output requirements given the networks architecture.
In chapter 3 the properties of connectionist networks will be discussed in more detail.
Frequency and Regularity
Structural regularity is the occurrence of similarities between different structures on a
certain grain level. For example, the English SRC is more regular than the ORC because
on the level of functional categories (SVO) the SRC is similar to many other structures.
In contrast, the corpus frequency of OSV as in the ORC is very low. In that sense,
16