An NSA Big Graph experiment Paul Burkhardt, Chris Waring May 20, 2013

Recommendations

Info

But for bulk-processing, use MapReduce Big Graph processing suitable for MapReduce. . . large, bulk writes to HDFS are faster (no need to sort) temporary scratch space (local writes) greater control over parallelization of algorithms Paul Burkhardt, Chris Waring An NSA Big Graph experiment NSA-RD-2013-056001v1
Quick overview of MapReduce NSA-RD-2013-056001v1 MapReduce processes data as 〈key, value〉 pairs in three steps: 1 MAP map tasks independently process blocks of data in parallel and output 〈key, value〉 pairs map tasks execute user-defined “map” function 2 SHUFFLE sorts 〈key, value〉 pairs and distributes to reduce tasks ensures value set for a key is processed by one reduce task framework executes a user-defined “partitioner” function 3 REDUCE reduce tasks process assigned (key, value set) in parallel reduce tasks execute user-defined “reduce” function Paul Burkhardt, Chris Waring An NSA Big Graph experiment
Page 1 and 2: An NSA Big Graph experiment Paul Bu
Page 3 and 4: Scale of the first graph NSA-RD-201
Page 5 and 6: Big Data begets Big Graphs Increasi
Page 7 and 8: Web scale. . . 50 billion vertices,
Page 9 and 10: Benchmarking scalability on Big Gra
Page 11 and 12: NSA-RD-2013-056001v1 Problem: How d
Page 13 and 14: Poor locality increases latency. .
Page 15 and 16: If it fits. . . NSA-RD-2013-056001v
Page 17 and 18: Larger systems, greater latency. .
Page 19 and 20: Top supercomputer installations NSA
Page 21 and 22: Distributed 〈key, value〉 reposi
Page 23: Using Apache Accumulo for Big Graph
Page 27 and 28: Good principles for MapReduce algor
Page 29 and 30: How well does this really work? Pro
Page 31 and 32: Graph500 Experiment Graph500 Huge c
Page 33 and 34: Graph500 Experiment Graph500 Huge c
Page 35: Future work What are the tradeoffs

An NSA Big Graph experiment Paul Burkhardt, Chris Waring May 20, 2013

Create successful ePaper yourself

Delete template?

Save as template?