E E 3J2 D ata M ining L ecture 3 Z ipf's L aw Stem m ing & Stop L ists

Relating probability and rank
By setting these equal we can get an expression for pk in
terms of rk , which is what you need to compare with the
Zipf curve:
( ) α
C
+ B
=
r
k
p
k
This is of the same form as Zipf. The values of B and C
depend on M and α
The value of α depends of M (See Belew, pages 150-152
for details)
For M=26, α = 1.012, C = 0.02, B = 0.54
EE3J2 Data Mining 2008 – lecture 3
Slide 18