Hadoop Development - CSC

More documents

Recommendations

Info

Velocity • The rate at which volume and variety are increasing are themselves increasing! • “real time” analysis needs – Increasingly becoming necessary to process new data as it streams in rather than over-night TBSC 2009 11/10/2011 12:53 PM 0725-23_TBSC 2009 8
What Is Hadoop? • Google famously encountered these problems when it wanted to index the World Wide Web. Their solution: – Google File System (GFS) for storage – Google Map-Reduce to be able to rapidly process, in a highly parallel way, data stored in GFS • Google‟s solution is proprietary and their success created demand for similar capabilities that other companies could use, including their competitors, most notably, Yahoo! • The result of that demand is an Apache Open Source project called Hadoop that provides: – HDFS (Hadoop Distributed File System), equivalent in capability to GFS – MapReduce to process the data stored in HDFS, equivalent in capability to Google Map-Reduce TBSC 2009 11/10/2011 12:53 PM 0725-23_TBSC 2009 9
Page 1 and 2: ©2011 CSC INNOVATE AND DELIVER AN
Page 3 and 4: The Data Challenge •3 dimensions:
Page 5 and 6: Volume (cont’d) • IDC now predi
Page 7: Variety • Continuing increase in
Page 11 and 12: What Is Hadoop? (cont’d) • Hado
Page 13 and 14: Hadoop Architectural Overview (cont
Page 15 and 16: Hadoop Architectural Overview (cont
Page 17 and 18: What is Map-Reduce? • Map-Reduce
Page 19 and 20: What is Map-Reduce? (cont’d) Map
Page 21 and 22: What is Map-Reduce? (cont’d) •
Page 23 and 24: Real world example - a Cyber proble
Page 31 and 32: How to get started... • Download
Page 33 and 34: Questions? TBSC 2009 ? 11/10/2011 1
Page 35: TBSC 2009 11/10/2011 12:53 PM 0725-

Hadoop Development - CSC

Create successful ePaper yourself

Delete template?

Save as template?