Notes for the Lifebox, the Seashell, and the Soul - Rudy Rucker

More documents

Recommendations

Info

Notes for The Lifebox, the Seashell, and the Soul, by Rudy Rucker Zipf’s Law Table The table below illustrates Zipf’s law, based on a word sample consisting of some five hundred old articles from Time magazine totaling about a quarter of a million words. 8 The idea is that, row by row, the last two columns should be roughly equal to each other. 9 8 I found this data on a web page by computer scientist Jamie Callan of the University of Massachusetts, http://web.archive.org/web/20001005120011/hobart.cs.umass.edu/~allan/cs646- f97/char_of_text.html. For the mother of all Zipf’s law web pages, see Wentian Li’s site, http://linkage.rockefeller.edu/wli/zipf/. 9 He scanned an XML edition of Webster's 1913 Revised Unabridged Dictionary, as revised and extended by the GNU Collaborative International Dictionary of English project, found at http://www.ibiblio.org/webster/ We defined two words to be linked if either appears in the definition of the other. Walker created a table of pairs (L, N) which states the number of words N having a given number of links L, with L from 1 to 149. There are three traditional ways of describing this kind of data. An inverse power law of the form N ~ 1/L D . This means that the number of words N that have a given linkiness L is proportional to 1/ L. A Zipf style law (to be discussed a bit later in this section) in which we rank words from the most linked to the least, and if R is a word’s rank order, then the linkiness L ~ 1 / R E . A Pareto style law that would say that for any linkiness L, the number of words M having linkiness greater than L is M ~ 1 / L F . A paper by Lada A. Adamic, Zipf, Power-laws, and Pareto - a Ranking Tutorial at http://www.hpl.hp.com/research/idl/papers/ranking/ranking.html, points out that we can get from a power law to a Pareto style law to a Zipf style law and vice versa. Let’s see how to go from our power law data to a Zipf form. Power. I have N = c/L D . L is a linkiness level and N is the number of words at that level. Pareto. To get the number of words with linkiness greater than L 0 , integrate c*L -D with respect to L from L 0 to infinity. Assume D > 1. I get (c/(1-D))*L (1-D) evaluated from L 0 to ∞, which cooks down to (c/(D- 1)) / L 0 (D-1) . So I can say that if M is the number of words with linkiness greater than L, then M = (c/(D-1)) / L (D-1) . Zipf. If I rank words in order of linkiness, and R is the Rth ranking word and it has linkiness L, then all the R words of higher rank have linkiness better than the word in question, so in fact R is the same as the M of the Pareto form, so I can say R = (c/(D-1)) / L (D-1) . And now if I turn this around to solve for L in terms of R, I get L = e / R (1/(D-1)) , where e = (c/(D-1)) (1/(D-1)) . Some Mathematica curve-fitting to Walker’s data gave me these numbers: Power Law. N = 1,000,000 / L 2.2 . Pareto Style Law. M is the number of words with linkiness above L, and M = 833,333 / L 1.1 Zipf Style Law. L is linkiness of the Rth ranking word and L = 244,312 / R^0.91 p. 90
Notes for The Lifebox, the Seashell, and the Soul, by Rudy Rucker Word Number of occurrences in a Quarter-Million-Word Sample Rank 0.1/Rank Probability (= Occurances/250,000) the 15861 1 0.1000 0.0646 of 7239 2 0.0500 0.0295 to 6331 3 0.0333 0.0258 a 5878 4 0.0250 0.0240 and 5614 5 0.0200 0.0229 in 5294 6 0.0167 0.0216 that 2507 7 0.0143 0.0102 for 2228 8 0.0125 0.0091 was 2149 9 0.0111 0.0088 with 1839 10 0.0100 0.0075 his 1815 11 0.0091 0.0074 is 1810 12 0.0083 0.0074 he 1700 13 0.0077 0.0069 as 1581 14 0.0071 0.0064 on 1551 15 0.0067 0.0063 by 1467 16 0.0063 0.0060 at 1333 17 0.0059 0.0054 it 1290 18 0.0056 0.0053 from 1228 19 0.0053 0.0050 but 1138 20 0.0050 0.0046 u 955 21 0.0048 0.0039 had 940 22 0.0045 0.0038 last 930 23 0.0043 0.0038 be 915 24 0.0042 0.0037 have 914 25 0.0040 0.0037 who 894 26 0.0038 0.0036 not 882 27 0.0037 0.0036 has 880 28 0.0036 0.0036 an 873 29 0.0034 0.0036 s 865 30 0.0033 0.0035 were 848 31 0.0032 0.0035 their 815 32 0.0031 0.0033 are 812 33 0.0030 0.0033 one 811 34 0.0029 0.0033 week 793 35 0.0029 0.0032 they 697 36 0.0028 0.0028 govern 687 37 0.0027 0.0028 all 672 38 0.0026 0.0027 year 672 39 0.0026 0.0027 its 620 40 0.0025 0.0025 britain 89 41 0.0024 0.0004 when 579 42 0.0024 0.0024 out 577 43 0.0023 0.0024 would 577 44 0.0023 0.0024 new 572 45 0.0022 0.0023 up 559 46 0.0022 0.0023 been 554 47 0.0021 0.0023 more 540 48 0.0021 0.0022 which 539 49 0.0020 0.0022 into 518 50 0.00200.0021 Table 5.1: The Probability of a Word is About 0.1 Divided by the Word’s Rank. p. 91
Page 1 and 2:
Notes for The Lifebox, the Seashell
Page 3 and 4:
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40: Notes for The Lifebox, the Seashell
Page 89: Notes for The Lifebox, the Seashell
Page 141:
show all

Notes for the Lifebox, the Seashell, and the Soul - Rudy Rucker

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?