Part 4: Index Construction

More documents

Recommendations

Info

Google Web Farm The best guess is that Google now has more than 450,000 servers (2 Petabytes of RAM 2*10 6 Gigabytes) Spread over at least 25 locations around the world Connecting these centers is a high-capacity fiber optic network that the company has assembled over the last few years. J. Markoff, NYT, June 2006 Google is building two computing centers, top and left, each the size of a football field, in The Dalles, Ore.
Hardware assumptions Sec. 4.1 symbol statistic value s average seek time 5 ms = 5 x 10 −3 s b transfer time per byte 0.02 μs = 2 x 10 −8 s processor’s clock rate 10 9 s −1 p low-level operation 0.01 μs = 10 −8 s (e.g., compare & swap a word) size of main memory several GB size of disk space 1 TB or more Example: Reading 1GB from disk • If stored in contiguous blocks: 2 x 10 −8 s x 10 9 = 20s • If stored in 1M chunks of 1KB: 20s + 10 6 x 5 x 10 −3 s = 5020 s = 1.4 h
Page 1 and 2: Part 4: Index Construction Francesc
Page 3 and 4: Sec. 4.1 Hardware basics Many desi
Page 5: Sec. 4.1 Hardware basics Servers u
Page 9 and 10: Reuters RCV1 statistics Sec. 4.2 sy
Page 11 and 12: Key step After all documents have
Page 13 and 14: Sort-based index construction Sec.
Page 15 and 16: Bottleneck Sec. 4.2 Parse and buil
Page 17 and 18: Blocks consider different documents
Page 19 and 20: Block sorted-based indexing Sec. 4.
Page 21 and 22: Remaining problem with sort-based a
Page 23 and 24: SPIMI-Invert Sec. 4.3 When the memo
Page 25 and 26: Distributed indexing Sec. 4.4 For
Page 27 and 28: Google data centers If in a non-fa
Page 29 and 30: Parallel tasks Sec. 4.4 We will us
Page 31 and 32: Inverters Sec. 4.4 An inverter col
Page 33 and 34: MapReduce Sec. 4.4 The index const
Page 35 and 36: Schema for index construction in Ma
Page 37 and 38: Simplest approach Sec. 4.5 Maintai
Page 39 and 40: Sec. 4.5 Logarithmic merge Maintai
Page 41 and 42: Logarithmic merge Sec. 4.5 Auxilia
Page 43 and 44: Dynamic indexing at search engines
Page 45 and 46: Other sorts of indexes Sec. 4.5 P

Part 4: Index Construction

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?