HDFS - Custom Training Courses - Coreservlets.com

More documents

Recommendations

Info

Namenode's fault-tolerance• Namenode daemon process must be running atall times– If process crashes then cluster is down• Namenode is a single point of failure– Host on a machine with reliable hardware (ex. sustain a diskfailure)– Usually is not an issue• Hadoop 2+– High Availability Namenode• Active Standby is always running and takes over in case mainnamenode fails• Still in its infancy– Learn more @ http://hadoop.apache.org/docs/r2.0.2-alpha/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailability.html20HDFS Access• Access Patterns– Direct• Communicate with HDFS directly through native client• Java, C++– Proxy Server• Access HDFS through a Proxy Server – middle man• REST, Thrift, and Avro Servers21
Direct Access• Java and C++ APIs• Clients retrievemetadata such asblocks’ locationsfrom Namenode• Client directlyaccess datanode(s)• Java API– Most commonly used– Covered in this course• Used byMapReduceJava ClientJava Client...Java ClientNamenodeDatanodeDatanode...Datanode22Source: White, Tom. Hadoop The Definitive Guide. O'Reilly Media. 2012Proxy Based Access• Clients communicate through a proxy– Strives to be language independent• Several Proxy Servers are packaged withHadoop:– Thrift – interface definition language– WebHDFS REST – response formatted in JSON, XML orProtocol Buffers– Avro – Data Serialization mechanismClientClientProxyServerNamenodeDatanode...ClientProxyServer...Datanode23Source: White, Tom. Hadoop The Definitive Guide. O'Reilly Media. 2012
Page 1 and 2: © 2012 coreservlets.com and Dima M
Page 3 and 4: Use Commodity Hardware• “Cheap
Page 5 and 6: HDFS DaemonsSecondaryNamenodeNameno
Page 7: Block Replication• Namenode deter
Page 12 and 13: Resources: BooksHadoop: The Definit
Page 14: Summary• We learned about- HDFS U

HDFS - Custom Training Courses - Coreservlets.com

Create successful ePaper yourself

Delete template?

Save as template?