Hadoop Development - CSC

More documents

Recommendations

Info

Real world example - a Cyber problem (cont’d) • As always, there is more than one way to approach this, but one simple way would seem to be to have a map-reduce job that takes as input: – The auditd logs, and – A list of parent process ids that we want to trace back • and which outputs a file that contains: – A list of parent process-ids as the key to a list of auditd events that all have this process-id as a ppid entry. – A list of parent process-ids found that also have "terminal" entries (i.e. for which we have now found the user). • We can then repeatedly run this map-reduce job using the output of one run as input, with the auditd logs, to the next, until there is no difference in the output file between two consecutive runs, or until the output file is empty (whichever occurs first). (Note that since we will be dealing with audit logs from many different servers, we will need to use the IP address of the server with the process-id to form a unique key) TBSC 2009 11/10/2011 12:53 PM 0725-23_TBSC 2009 26
Real world example - a Cyber problem (cont’d) • A first question is where to get the initial list of process IDs from, and there are two obvious options: – Wait for the SOC staff to spot SYSCALL events that they are interested in, or – Make a first pass through the audit logs and for a given day, extract all the SYSCALL events on that day and then find the owner UIDs for all of them (Note that since the dataset used for development was quite small, option 2 was practical) TBSC 2009 11/10/2011 12:53 PM 0725-23_TBSC 2009 27
Page 1 and 2: ©2011 CSC INNOVATE AND DELIVER AN
Page 3 and 4: The Data Challenge •3 dimensions:
Page 5 and 6: Volume (cont’d) • IDC now predi
Page 7 and 8: Variety • Continuing increase in
Page 9 and 10: What Is Hadoop? • Google famously
Page 11 and 12: What Is Hadoop? (cont’d) • Hado
Page 13 and 14: Hadoop Architectural Overview (cont
Page 15 and 16: Hadoop Architectural Overview (cont
Page 17 and 18: What is Map-Reduce? • Map-Reduce
Page 19 and 20: What is Map-Reduce? (cont’d) Map
Page 21 and 22: What is Map-Reduce? (cont’d) •
Page 23 and 24: Real world example - a Cyber proble
Page 25: Real world example - a Cyber proble
Page 29 and 30: Real world example - a Cyber proble
Page 31 and 32: How to get started... • Download
Page 33 and 34: Questions? TBSC 2009 ? 11/10/2011 1
Page 35: TBSC 2009 11/10/2011 12:53 PM 0725-

Hadoop Development - CSC

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?