Hadoop NameNode Automatic Failover | Hadoop Tutorials: Brief Introduction - Software Development PDF Download

1. Objective

This article is all about Hadoop NameNode Automatic failover. Here we will cover Introduction to failover in Hadoop. It also covers types of Failover i.e. Graceful Failover and Automatic Failover. The components add by failover in Hadoop HDFS- ZooKeeper quorum, ZKFailoverController Process (ZKFC) are also covered on this blog in detail.

2. What is Failover?

Failover is the process in which system transfers its control to the secondary system when it detects a fault or failure.

There are two types of Failover:

• Graceful Failover- Administrator manually initiates Graceful Failover, for example, in the case of routine maintenance. The system will not trigger a failover automatically from the active to standby namenode, even if the active node has failed in Graceful Failover.
• Automatic Failover- In the case of NameNode failure, Failover will start automatically. This failover is known as Automatic failover.

3. Apache Hadoop NameNode Automatic Failover

Automatic failover adds below components to an HDFS deployment:

• ZooKeeper quorum.
• ZKFailoverController Process (ZKFC).

3.1. Zookeeper Quorum

It is a basic service for maintaining small amounts of coordination data, configuration information, and naming. And it provides distributed synchronization and group services. It informs the client of changes in the data, and track clients for failures.

Implementation of automatic HDFS failover relies on Zookeeper for:

• Failure detection- Zookeeper maintains a session with the namenodes. At the time of failure, the session will expire. And the Zookeeper will inform other namenodes to start the failover process.
• Active NameNode election- Zookeeper provides a mechanism to only elect a node as active. As a result, when active namenode fails, other namenode can take an exclusive lock in Zookeeper. It states that it wants to become the next active namenode.

3.2. ZKFailoverController process (ZKFC)

ZKFC is a client of Zookeeper that monitors and manages the namenode status. So, Each of the machines which run namenode service also runs a ZKFC.

ZKFC handles:

• Health monitoring- ZKFS pings its local namenode with a health check command. The ZKFC considers the namenode healthy, as long as the namenode responds at a time. The health check will mark a node as unhealthy, when the node has crashed or entered an unhealthy state.
• Zookeeper session management- ZKFC holds a session open in Zookeeper when the local namenode is healthy. It also holds a special “lock” znode, if the local namenode is active. If the session expires, then the lock will be automatically deleted.
• Zookeeper-based election- In HDFS, if the local namenode is healthy, and the ZKFC sees that no other node currently holds the lock znode, it will itself try to get the lock. So, If it succeeds, then it has won the election, and it handles running a failover to make its local namenode active.

4. Conclusion

In conclusion, we can say that it starts automatically in case of NameNode failure. Automatic failover adds ZooKeeper quorum and ZKFC components to an HDFS deployment. Hence, Zookeeper is a service that provides the automatic failover capability in HDFS.