Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
8.1.12 Markov models in human syntactic analysis?
(92) Shannon (1948, pp42-43) says:
We can also approximate to a natural language by means of a series of simple artificial language…To
give a visual idea of how this series approaches a language, typical sequences in the
approximations to English have been constructed and are given below…
5. First order word approximation…Here words are chosen independently but with their appropriate
frequencies.
REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL
HERE HE THE IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE
MESSAGE HAD BE THESE
6. Second order word approximation.
further structure is included.
The word transition probabilities are correct but no
THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARAC-
TER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE
TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED
The resemblance to ordinary English text increases quite noticeably at each of the above
steps…It appears then that a sufficiently complex stochastic source process will give a satisfactory
representation of a discrete source.
(93) Damerau (1971) confirms this trend in an experiment that involved generating 5th order approximations.
All these results are hard to interpret though, since (i) sparse data in generation will tend to yield near
copies of portions of the source texts (on the sparse data problem, remember the results from Jelinek
mentioned in 95, above), and (ii) human linguistic capabilities are not well reflected in typical texts.
(94) Miller and Chomsky objection 1: The number of parameters to set is enormous.
Notice that for a vocabulary of 100, 000 words, where each different word is emitted by a different event,
we would need at least 100,000 states. The full transition matrix then has 100, 0002 = 10 10 entries.
Notice that the last column of the transition matrix is redundant, and so a 109 matrix will do.
Miller and Chomsky (1963, p430) say:
We cannot seriously propose that a child learns the value of 10 9 parameters in a childhood
lasting only 10 8 seconds.
Why not? This is very far from obvious, unless the parameters are independent, and there is no reason
to assume they are.
(95) Miller and Chomsky (1963, p430) objection 2: The amount of input required to set the parameters of
a reasonable model is enormous.
Jelinek (1985) reports that after collecting the trigrams from a 1,500,000 word corpus, he found that,
in the next 300,000 words, 25% of the trigrams were new.
No surprise! Some generalization across lexical combinations is required. In this context, the “generalization”
is sometimes achieved with various “smoothing” functions, which will be discussed later. With
generalization, setting large numbers of parameters becomes quite conceivable.
Without a better understanding of the issues, I find objection 2 completely unpersuasive.
(96) Miller and Chomsky (1963, p425) objection 3:
Since human messages have dependencies extending over long strings of symbols, we know that
any pure Markov source must be too simple…
147