Data Warehousing and Analytics Infrastructure at Facebook

More documents

Recommendations

Info

Warehouse is Business Critical Challenge: Remove all single points of failure (SPOF) Hadoop NameNode was a SPOF Hadoop AvatarNode – Active-passive Hot Standby pair of NameNodes – Failover time for 20 PB file system having 65 million files is 10 seconds – Work-in progress to support active-active AvatarNode
Warehouse Utilization and Workload Compute Map-Reduce cluster is CPU bound – Peak usage of 95% CPU utilization – Peak network usage is 70% (network is not a bottleneck) – 70% tasks find data on local rack – We do not compress map outputs Storage HDFS cluster is capacity bound – 75% storage full (current size is 21 PB, 2000 nodes) – Disk bandwidth used is 20% of capacity (IO is not a bottleneck) – All Hive tables compressed using gzip
Page 1 and 2: Data Warehousing and Analytics Infr
Page 3 and 4: Key Challenge (I) Data, data and mo
Page 5 and 6: Key Challenge (III) Flexibility Sup
Page 7 and 8: Data Flow Architecture at Facebook
Page 9 and 10: Analysis and Data Organization > 9
Page 11 and 12: Data Discovery (10,000 feet) 1. Tag
Page 13 and 14: Hadoop & Hive Cluster @ Facebook H
Page 15 and 16: Data Collection and Real time Avail
Page 17: Scribe-HDFS: 101 Scribed HDFS Data
Page 21 and 22: Storage Costs Challenge: disk is c
Page 23 and 24: Isolation for Business Critical Pip

Data Warehousing and Analytics Infrastructure at Facebook

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?