Graph Processing & Bulk Synchronous Parallel Model

More documents

Recommendations

Info

Fault Tolerance • Checkpoint: master instructs workers to save state to persistent storage (e.g. HDFS) – Vertex values – Edge values – Incoming messages • Master saves to disk aggregator values • Worker failure is detected using a heartbeat. • New worker is created using state from previous checkpoint (which could be several supersteps before current superstep) Lecture 14 : 590.02 Spring 13 24
Summary • Map-‐reduce has no naFve support for iteraFons – No Loop construct – Write to disk and read from disk in each step, even if a the data is an invariant in the loop. • Systems like HaLoop introduce inter-‐iteraFon locality and caching to help iteraFons on map-‐reduce. • Pregel is a vertex oriented programming model and system for graph processing with built in features for iteraFve processing on graphs. Lecture 14 : 590.02 Spring 13 25
Page 1 and 2: Graph Processing & Bulk Synchro
Page 3 and 4: This Class • IteraFon Aware M
Page 5 and 6: IteraFve ComputaFons PageRank: d
Page 7 and 8: HaLoop Map Reduce Map Reduce Loop
Page 9 and 10: PREGEL Lecture 14 : 590.02 Spr
Page 11 and 12: Pregel Overview • Processing o
Page 13 and 14: Example 3 6 2 1 Superstep 0 6 6 2
Page 15 and 16: Vertex API • MessageIterator c
Page 17 and 18: Combiners • If messages are a
Page 19 and 20: AggregaFon • Global communicaFo
Page 21 and 22: PREGEL ARCHITECTURE Lecture 14
Page 23: Processing • Master coordinates

Graph Processing & Bulk Synchronous Parallel Model

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?