Introduction and MapReduce - SNAP - Stanford University

More documents

Recommendations

Info

Suppose we have a large web corpus Look at the metadata file Lines of the form (URL, size, date, …) For each host, find the total number of bytes i.e., the sum of the page sizes for all URLs from that host Other examples: Link analysis and graph processing Machine Learning algorithms 1/8/2012 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 53
Statistical machine translation: Need to count number of times every 5-word sequence occurs in a large corpus of documents Very easy with MapReduce: Map: Extract (5-word sequence, count) from document Reduce: Combine counts 1/8/2012 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 54
Page 1 and 2: CS246: Mining Massive Datasets Jure
Page 3 and 4: Course website: http://cs246.stanfo
Page 5 and 6: Short weekly quizzes: 20% Short e-
Page 7 and 8: Recitation sessions: Review of pro
Page 9 and 10: CS345a: Data mining got split into
Page 11 and 12: Much of the course will be devoted
Page 13 and 14: High-dimensional data: Locality Se
Page 16 and 17: Discovery of patterns and models th
Page 18 and 19: Scalability Dimensionality Comple
Page 20 and 21: A big data-mining risk is that you
Page 22: He told these people they had ESP a
Page 25 and 26: 20+ billion web pages x 20KB = 400+
Page 27 and 28: Large-scale computing for data mini
Page 29 and 30: Problem If nodes fail, how to stor
Page 31 and 32: Reliable distributed file system D
Page 33 and 34: Case 1: File too large for memory,
Page 35 and 36: Input key-value pairs k k k … v v
Page 37 and 38: Input: a set of key/value pairs Pr
Page 39 and 40: map(key, value): // key: document n
Page 41 and 42: MAP: reads input and produces a set
Page 43 and 44: Input and final output are stored o
Page 45 and 46: Map worker failure Map tasks compl
Page 47 and 48: Fine granularity tasks: map tasks >
Page 49 and 50: Often a map task will produce many
Page 51: 1/8/2012 Jure Leskovec, Stanford CS
Page 55 and 56: Use a hash function h from B-values
Page 57 and 58: For a map-reduce algorithm: Commun
Page 59 and 60: Total communication cost = O(|R|+|S
Page 61 and 62: Google Not available outside Googl
Page 63 and 64: Jeffrey Dean and Sanjay Ghemawat: M
Page 65 and 66: Releases from Apache download mirro

Introduction and MapReduce - SNAP - Stanford University

Create successful ePaper yourself

Delete template?

Save as template?