Part 4: Index Construction

More documents

Recommendations

Info

Distributed indexing Sec. 4.4 Maintain a master machine directing the indexing job – considered “safe” Break up indexing into sets of (parallel) tasks Master machine assigns each task to an idle machine from a pool.
Parallel tasks Sec. 4.4 We will use two sets of parallel tasks • Parsers • Inverters Break the input document collection into splits Document collection split split split split split Each split is a subset of documents (corresponding to blocks in BSBI/SPIMI)
Page 1 and 2: Part 4: Index Construction Francesc
Page 3 and 4: Sec. 4.1 Hardware basics Many desi
Page 5 and 6: Sec. 4.1 Hardware basics Servers u
Page 7 and 8: Hardware assumptions Sec. 4.1 symb
Page 9 and 10: Reuters RCV1 statistics Sec. 4.2 sy
Page 11 and 12: Key step After all documents have
Page 13 and 14: Sort-based index construction Sec.
Page 15 and 16: Bottleneck Sec. 4.2 Parse and buil
Page 17 and 18: Blocks consider different documents
Page 19 and 20: Block sorted-based indexing Sec. 4.
Page 21 and 22: Remaining problem with sort-based a
Page 23 and 24: SPIMI-Invert Sec. 4.3 When the memo
Page 25 and 26: Distributed indexing Sec. 4.4 For
Page 27: Google data centers If in a non-fa
Page 31 and 32: Inverters Sec. 4.4 An inverter col
Page 33 and 34: MapReduce Sec. 4.4 The index const
Page 35 and 36: Schema for index construction in Ma
Page 37 and 38: Simplest approach Sec. 4.5 Maintai
Page 39 and 40: Sec. 4.5 Logarithmic merge Maintai
Page 41 and 42: Logarithmic merge Sec. 4.5 Auxilia
Page 43 and 44: Dynamic indexing at search engines
Page 45 and 46: Other sorts of indexes Sec. 4.5 P

Part 4: Index Construction

Create successful ePaper yourself

Delete template?

Save as template?