Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
jane austen
austen chapter
chapter sir
sir walter
walter elliot
elliot of
of kellynch
kellynch hall
hall in
in somersetshire
somersetshire was
was a
a man
man who
who for
for his
his own
own amusement
amusement never
never took
(20) 38%sort -d bigrams > bigrams.srt
39%more bigrams.srt
a bad
a bad
a bad
a bad
a bad
a ball
a baronet
a baronet
a baronet
a baronet
a baronet
a baronetcy
a baronetcy
a baronight
a barouche
a beautiful
a bed
(21) 40%uniq -c bigrams.srt > bigrams.cnt
41%more bigrams.cnt
5 a bad
1 a ball
5 a baronet
2 a baronetcy
1 a baronight
1 a barouche
1 a beautiful
1 a bed
1 a beloved
1 a bend
Here, wc -l bigrams.cnt gives us 42,728 – the number of bigram types – sequences of two word
types that occur in the text.
Notice that 42,728 < 57422 =32,970,564, – the number of bigrams is very much less than the number
of possible word pairs, which is the square of the number of word-types.
(22) 42%sort -nr bigrams.cnt > bigrams.sct
43%more bigrams.sct
429 of the
378 to be
323 in the
255 had been
227 she had
220 it was
196 captain wentworth
191 he had
176 to the
174 mr elliot
164 she was
148 could not
147 lady russell
134 he was
131 sir walter
129 of her
127 all the
125 i have
123 i am
117 have been
114 she could
112 of his
112 and the
111 for the
109 they were
96 to her
96 that he
96 did not
95 on the
94 to have
91 a very
90 of a
89 would be
88 it is
84 that she
122