Introduction and MapReduce - SNAP - Stanford University

More documents

Recommendations

Info

Ability to rent computing by the hour Additional services e.g., persistent storage Amazon’s “Elastic Compute Cloud” (EC2) Aster Data and Hadoop can both be run on EC2 For CS341 (offered next quarter) Amazon will provide free access for the class 1/8/2012 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 63
Jeffrey Dean and Sanjay Ghemawat: MapReduce: Simplified Data Processing on Large Clusters http://labs.google.com/papers/mapreduce.html Sanjay Ghemawat, Howard Gobioff, and Shun- Tak Leung: The Google File System http://labs.google.com/papers/gfs.html 1/8/2012 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 64
Page 1 and 2:
CS246: Mining Massive Datasets Jure
Page 3 and 4:
Course website: http://cs246.stanfo
Page 5 and 6:
Short weekly quizzes: 20% Short e-
Page 7 and 8:
Recitation sessions: Review of pro
Page 9 and 10:
CS345a: Data mining got split into
Page 11 and 12: Much of the course will be devoted
Page 13 and 14: High-dimensional data: Locality Se
Page 16 and 17: Discovery of patterns and models th
Page 18 and 19: Scalability Dimensionality Comple
Page 20 and 21: A big data-mining risk is that you
Page 22: He told these people they had ESP a
Page 25 and 26: 20+ billion web pages x 20KB = 400+
Page 27 and 28: Large-scale computing for data mini
Page 29 and 30: Problem If nodes fail, how to stor
Page 31 and 32: Reliable distributed file system D
Page 33 and 34: Case 1: File too large for memory,
Page 35 and 36: Input key-value pairs k k k … v v
Page 37 and 38: Input: a set of key/value pairs Pr
Page 39 and 40: map(key, value): // key: document n
Page 41 and 42: MAP: reads input and produces a set
Page 43 and 44: Input and final output are stored o
Page 45 and 46: Map worker failure Map tasks compl
Page 47 and 48: Fine granularity tasks: map tasks >
Page 49 and 50: Often a map task will produce many
Page 51 and 52: 1/8/2012 Jure Leskovec, Stanford CS
Page 53 and 54: Statistical machine translation: N
Page 55 and 56: Use a hash function h from B-values
Page 57 and 58: For a map-reduce algorithm: Commun
Page 59 and 60: Total communication cost = O(|R|+|S
Page 61: Google Not available outside Googl
Page 65 and 66: Releases from Apache download mirro
show all

Introduction and MapReduce - SNAP - Stanford University

Create successful ePaper yourself

Delete template?

Save as template?