Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
Word length in naturally occurring text has a similar relation to frequency – with dictionaries as unsurprisingly
exceptional:
Proportion of the number of tokens
0.3
0.25
0.2
0.15
0.1
0.05
0
Bible
Shakespeare
WSJ
Chambers
Brown
LOB
WSJ
Jefferson
Austen
Shakespeare
Bible
Chambers
2 4 6 8 10 12 14 16 18
Word length
Most words types are rare; bigrams, trigrams more so
Cumulative percentage of the number of
types, bigrams or trigrams
100
90
80
70
60
50
40
1
2
trigrams
3
bigrams
4
5
types
10
20
30
Frequency
40 50
130
100
200