Hadoop Development - CSC

More documents

Recommendations

Info

Hadoop Architectural Overview (cont’d) • HDFS is designed for Write Once/Read Many operations • HDFS block sizes are big – 64MB, 128MB and 256MB are common – To maximise disk read throughput • Hadoop runs one Map task for each HDFS block in the data to be processed and takes approx one minute to start a map task so execution needs to take at least one minute • Increase block size to increase task execution time TBSC 2009 11/10/2011 12:53 PM 0725-23_TBSC 2009 14
Hadoop Architectural Overview (cont’d) • Hadoop scales linearly – Suppose 100TB of data to process. With one server and a read rate of 100MB/s, it would take: • (100 x 10^12) / (100 x 10^6) = 10^6 = 1,000,000 seconds to read the files. – With 100 servers each with a 100MB/s read rate and the equal distribution of files across all 100 servers, i.e. each server has 1TB to read, it will take: • (1 x 10^12) / (100 x 10^6) = 10^4 = 10,000 seconds • Since all 100 read the data in parallel, total read time is 10,000 seconds, i.e. 100 times faster. – If have 1000 servers and equal distribution of the data across the cluster, then can read all 100TB in just 1,000 seconds, i.e. less than 20 minutes. TBSC 2009 11/10/2011 12:53 PM 0725-23_TBSC 2009 15
Page 1 and 2: ©2011 CSC INNOVATE AND DELIVER AN
Page 3 and 4: The Data Challenge •3 dimensions:
Page 5 and 6: Volume (cont’d) • IDC now predi
Page 7 and 8: Variety • Continuing increase in
Page 9 and 10: What Is Hadoop? • Google famously
Page 11 and 12: What Is Hadoop? (cont’d) • Hado
Page 13: Hadoop Architectural Overview (cont
Page 17 and 18: What is Map-Reduce? • Map-Reduce
Page 19 and 20: What is Map-Reduce? (cont’d) Map
Page 21 and 22: What is Map-Reduce? (cont’d) •
Page 23 and 24: Real world example - a Cyber proble
Page 31 and 32: How to get started... • Download
Page 33 and 34: Questions? TBSC 2009 ? 11/10/2011 1
Page 35: TBSC 2009 11/10/2011 12:53 PM 0725-

Hadoop Development - CSC

Create successful ePaper yourself

Delete template?

Save as template?