Data Warehousing and Analytics Infrastructure at Facebook

More documents

Recommendations

Info

Hive & Hadoop Usage @ Facebook Statistics per day: – 800TB of I/O per day – 10K – 25K Hadoop jobs per day Hive simplifies Hadoop: – New engineers go though a Hive training session – Analysts (non-engineers) use Hadoop through Hive – Most of jobs are Hive Jobs
Data Collection and Real time Availability Challenge: Large Volume of data to be queryable as soon as it is produced: – Logs produce 100 TB per day uncompressed – Store in Scribe-HDFS clusters co-located with web-tier Query-able within 15 seconds (via tail –f ) Overlapping HDFS clusters to avoid Single Point of Failure Two separate NameNodes Two DataNode instances on same physical machine – Gzip compressed, uploaded into central warehouse Query-able within 15 minutes (via Hive external tables)
Page 1 and 2: Data Warehousing and Analytics Infr
Page 3 and 4: Key Challenge (I) Data, data and mo
Page 5 and 6: Key Challenge (III) Flexibility Sup
Page 7 and 8: Data Flow Architecture at Facebook
Page 9 and 10: Analysis and Data Organization > 9
Page 11 and 12: Data Discovery (10,000 feet) 1. Tag
Page 13: Hadoop & Hive Cluster @ Facebook H
Page 17 and 18: Scribe-HDFS: 101 Scribed HDFS Data
Page 19 and 20: Warehouse Utilization and Workload
Page 21 and 22: Storage Costs Challenge: disk is c
Page 23 and 24: Isolation for Business Critical Pip

Data Warehousing and Analytics Infrastructure at Facebook

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?