Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
809 as
707 for
664 but
659 his
654 with
628 you
589 have
533 at
530 all
497 anne
496 been
485 s
467 him
451 could
434 very
433 they
426 were
418 by
416 which
398 is
396 on
359 so
356 no
....
1 tranquility
1 trained
1 trafalgar
1 tradespeople
1 toys
1 toward
1 tossed
1 torn
1 topic
1 tolerated
1 tolerate
The command sort -nr sorts the file in numeric reverse order, looking for a number at the beginning
of each line.
Notice that almost all of these most frequent words are one syllable.
(16) Zipf (1949) observed that longer words tend to be infrequent, and that frequencies are distributed in a
regular, non-normal way. This is discussed below.
(17) 33%more persu11.wds
persuasion
by
jane
austen
chapter
sir
walter
elliot
of
kellynch
hall
in
somersetshire
was
a
man
who
for
his
own
amusement
never
(18) 34%tail +2 persu11.wds > persu11.w2
35%more persu11.w2
by
jane
austen
chapter
sir
walter
elliot
of
kellynch
hall
in
somersetshire
was
a
man
who
for
his
own
amusement
never
took
(19) 36%paste persu11.wds persu11.w2 > bigrams
37%more bigrams
persuasion by
by jane
121