Notes for the Lifebox, the Seashell, and the Soul - Rudy Rucker

Notes for The Lifebox, the Seashell, and the Soul, by Rudy Rucker
Zipf’s Law Table
The table below illustrates Zipf’s law, based on a word sample consisting of some
five hundred old articles from Time magazine totaling about a quarter of a million words. 8
The idea is that, row by row, the last two columns should be roughly equal to each other. 9
8 I found this data on a web page by computer scientist Jamie Callan of the University of
Massachusetts, http://web.archive.org/web/20001005120011/hobart.cs.umass.edu/~allan/cs646-
f97/char_of_text.html. For the mother of all Zipf’s law web pages, see Wentian Li’s site,
http://linkage.rockefeller.edu/wli/zipf/.
9 He scanned an XML edition of Webster's 1913 Revised Unabridged Dictionary, as revised and
extended by the GNU Collaborative International Dictionary of English project, found at
http://www.ibiblio.org/webster/
We defined two words to be linked if either appears in the definition of the other. Walker created a
table of pairs (L, N) which states the number of words N having a given number of links L, with L from 1 to
149. There are three traditional ways of describing this kind of data.
An inverse power law of the form N ~ 1/L D . This means that the number of words N that have a given
linkiness L is proportional to 1/ L.
A Zipf style law (to be discussed a bit later in this section) in which we rank words from the most
linked to the least, and if R is a word’s rank order, then the linkiness L ~ 1 / R E .
A Pareto style law that would say that for any linkiness L, the number of words M having linkiness
greater than L is M ~ 1 / L F .
A paper by Lada A. Adamic, Zipf, Power-laws, and Pareto - a Ranking Tutorial at
http://www.hpl.hp.com/research/idl/papers/ranking/ranking.html, points out that we can get from a power law to
a Pareto style law to a Zipf style law and vice versa. Let’s see how to go from our power law data to a Zipf
form.
Power. I have N = c/L D . L is a linkiness level and N is the number of words at that level.
Pareto. To get the number of words with linkiness greater than L 0 , integrate c*L -D with respect to L
from L 0 to infinity. Assume D > 1. I get (c/(1-D))*L (1-D) evaluated from L 0 to ∞, which cooks down to (c/(D-
1)) / L 0 (D-1) . So I can say that if M is the number of words with linkiness greater than L, then M = (c/(D-1)) /
L (D-1) .
Zipf. If I rank words in order of linkiness, and R is the Rth ranking word and it has linkiness L, then all
the R words of higher rank have linkiness better than the word in question, so in fact R is the same as the M of
the Pareto form, so I can say R = (c/(D-1)) / L (D-1) . And now if I turn this around to solve for L in terms of R, I
get L = e / R (1/(D-1)) , where e = (c/(D-1)) (1/(D-1)) .
Some Mathematica curve-fitting to Walker’s data gave me these numbers:
Power Law. N = 1,000,000 / L 2.2 .
Pareto Style Law. M is the number of words with linkiness above L, and M = 833,333 / L 1.1
Zipf Style Law. L is linkiness of the Rth ranking word and L = 244,312 / R^0.91
p. 90