Part 4: Index Construction

More documents

Recommendations

Info

Scaling index construction Sec. 4.2 In-memory index construction does not scale How can we construct an index for very large collections? Taking into account the hardware constraints we just learned about . . . Memory, disk, speed, etc.
Sort-based index construction Sec. 4.2 As we build the index, we parse docs one at a time • While building the index, we cannot easily exploit compression tricks (you can, but much more complex) The final postings for any term are incomplete until the end At 12 bytes per non-positional postings entry (term, doc, freq), demands a lot of space for large collections T = 100,000,000 in the case of RCV1 – so 1.2GB • So … we can do this in memory in 2009, but typical collections are much larger - e.g. the New York Times provides an index of >150 years of newswire Thus: We need to store intermediate results on disk.
Page 1 and 2: Part 4: Index Construction Francesc
Page 3 and 4: Sec. 4.1 Hardware basics Many desi
Page 5 and 6: Sec. 4.1 Hardware basics Servers u
Page 7 and 8: Hardware assumptions Sec. 4.1 symb
Page 9 and 10: Reuters RCV1 statistics Sec. 4.2 sy
Page 11: Key step After all documents have
Page 15 and 16: Bottleneck Sec. 4.2 Parse and buil
Page 17 and 18: Blocks consider different documents
Page 19 and 20: Block sorted-based indexing Sec. 4.
Page 21 and 22: Remaining problem with sort-based a
Page 23 and 24: SPIMI-Invert Sec. 4.3 When the memo
Page 25 and 26: Distributed indexing Sec. 4.4 For
Page 27 and 28: Google data centers If in a non-fa
Page 29 and 30: Parallel tasks Sec. 4.4 We will us
Page 31 and 32: Inverters Sec. 4.4 An inverter col
Page 33 and 34: MapReduce Sec. 4.4 The index const
Page 35 and 36: Schema for index construction in Ma
Page 37 and 38: Simplest approach Sec. 4.5 Maintai
Page 39 and 40: Sec. 4.5 Logarithmic merge Maintai
Page 41 and 42: Logarithmic merge Sec. 4.5 Auxilia
Page 43 and 44: Dynamic indexing at search engines
Page 45 and 46: Other sorts of indexes Sec. 4.5 P

Part 4: Index Construction

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?