MIT Encyclopedia of the Cognitive Sciences - Cryptome

finding relevant pages on the world wide web). The second,
the one most relevant to cognitive science, seeks to better
understand how language comprehension and generation
occurs in humans. Rather than performing experiments on
humans as done in psycholinguistics, or developing theories
that account for the data with a focus on handling possible
counterexamples as in linguistics and philosophy, researchers
in natural language processing test theories by building
explicit computational models to see how well they behave.
Most research in the field is still more in the exploratory
stage of this endeavor and trying to construct “existence
proofs” (i.e., find any mechanism that can understand language
within limited scenarios), rather than building computational
models and comparing them to human
performance. But once such existence-proof systems are
completed, the stage will be set for more detailed comparative
study between human and computational models.
Whatever the motivation behind the work in this area, however,
computational models have provided the inspiration
and starting point for much work in psycholinguistics and
linguistics in the last twenty years.
Although there is a diverse set of methods used in natural
language processing, the techniques can generally be
broadly classified in three general approaches: statistical
methods, structural/pattern-based methods and reasoningbased
methods. It is important to note that these approaches
are not mutually exclusive. In fact, the most comprehensive
models combine all three techniques. The approaches differ
in the kind of processing tasks they can perform and in the
degree to which systems require handcrafted rules as
opposed to automatic training/learning from language data.
A good source that gives an overview of the field involving
all three approaches is Allen 1995.
Statistical methods involve using large corpora of language
data to compute statistical properties such as word
co-occurrence and sequence information (see also STATISTI-
CAL TECHNIQUES IN NATURAL LANGUAGE PROCESSING). For
instance, a bigram statistic captures the probability of a
word with certain properties following a word with other
properties. This information can be estimated from a corpus
that is labeled with the properties needed, and used to predict
what properties a word might have based on its preceding
context. Although limited, bigram models can be
surprisingly effective in many tasks. For instance, bigram
models involving part of speech labels (e.g., noun, verb) can
typically accurately predict the right part of speech for over
95 percent of words in general text. Statistical models are
not restricted to part of speech tagging, however, and they
have been used for semantic disambiguation, structural disambiguation
(e.g., prepositional phrase attachment), and
many other properties. Much of the initial work in statistical
language modeling was performed for automatic speechrecognition
systems, where good word prediction can double
the word-recognition accuracy rate. The techniques have
also proved effective in tasks such as information retrieval
and producing rough “first-cut” drafts in machine translation.
A big advantage to statistical techniques is that they
can be automatically trained from language corpora. The
challenge for statistical models concerns how to capture
higher level structure, such as semantic information, and
Natural Language Processing 593
structural properties, such as sentence structure. In general,
the most successful approaches to these problems involve
combining statistical approaches with other approaches. A
good introduction to statistical approaches is Charniak
1993.
Structural and pattern-based approaches have the closest
connection to traditional linguistic models. These
approaches involve defining structural properties of language,
such as defining FORMAL GRAMMARS for natural languages.
Active research issues include the design of
grammatical formalisms to capture natural language structure
yet retain good computational properties, and the
design of efficient parsing algorithms to interpret sentences
with respect to a grammar. Structural approaches are not
limited solely to syntax, however. Many more practical systems
use semantically based grammars, where the primitive
units in the grammar are semantic classes rather than syntactic.
And other approaches dispense with fully analyzing
sentence structure altogether, using simpler patterns of lexical,
syntactic and semantic information that match sentence
fragments. Such techniques are especially useful in limiteddomain
speech-driven applications where errors in the input
can be expected. Because the domain is limited, certain
phrases (e.g., a prepositional phrase) may have only one
interpretation possible in the application. Structural models
also appear at the DISCOURSE level, where models are developed
that capture the interrelationships between sentences
and build models of topic flow. Structural models provide a
capability for detailed analysis of linguistic phenomena, but
the more detailed the analysis, the more one must rely on
hand-constructed rules rather than automatic training from
data. An excellent collection of papers on structural
approaches, though missing recent work, is Grosz, Sparck
Jones, and Webber 1986.
Reasoning-based approaches involve encoding knowledge
and reasoning processes and use these to interpret language.
This work has much in common with work in
KNOWLEDGE REPRESENTATION as well as work in the philosophy
of language. The idea here is that the interpretation
of language is highly dependent on the context in which the
language appears. By trying to capture the knowledge a
human may have in a situation, and model common-sense
reasoning, problems such as word sense and sentencestructure
disambiguation, analysis of referring expressions,
and the recognition of the intentions behind language can be
addressed. These techniques become crucial in discourse,
whether it be extended text that needs to be understood or a
dialogue that needs to be engaged in. Most dialogue-based
systems use a speech-act–based approach to language and
computational models of PLANNING and plan recognition to
define a conversational agent. Specifically, such systems
first attempt to recognize the intentions underlying the utterances
they hear, and then plan their own utterances based on
their goals and knowledge (including what was just recognized
about the other agent). The advantage of this approach
is that is provides a mechanism for contextual interpretation
of language. The disadvantage is the complexity of the models
required to define the conversational agent. Two good
sources for work in this area are Cohen, Morgan, and Pollack
1990 and Carberry 1991.