Hadoop Architecture | HDFS Tutorial For Beginners | HDFS Architecture | Hadoop Training |Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
broadly HDFS architecture is known as
the master and slave architecture so how
does it work a master node that is the
main node is responsible for accepting
jobs from the clients it's task is to
ensure that the data required for the
operation is loaded and segregated into
chunks of data blocks HDFS exposes a
file system namespace and allows user
data to be stored in files a file is
split into one or more blocks stored and
replicated in the slave nodes known at
the data nodes the data blocks are then
distributed to the data node systems
within the cluster this ensures that
replicas of the data are maintained data
node serves read or write requests it
also creates deletes and replicates
blocks on the instructions from the name
node we discussed in the previous topic
that it is the metadata which stores the
block location and its replication there
is a secondary name node which performs
tasks for name node and is also
considered as a master node prior to
Hadoop 2.0 point 0 the name node was a
single point of failure or SPO F in an
HDFS cluster each cluster had a single
name node in case of an unplanned event
such as a system failure the cluster
would be unavailable until an operator
restarted the name node also planned
maintenance events such as software or
hardware upgrades on the name node
system would result in cluster downtime
the HDFS high availability or H a
feature addresses these problems by
providing the option of running two
redundant name nodes in the same cluster
in an active/passive configuration with
a hop standby this allows a fast
failover to a new name node in case the
system crashes or an administrator
initiates a failover for the purpose of
a planned maintenance in an
a cluster two separate systems are
configured as name nodes at any instance
one of the name nodes is in an active
state and the other is in a standby
state the active name node is
responsible for all client operations in
the cluster
while the standby simply acts as a slave
maintaining enough state to provide fast
failover if necessary an HDFS cluster
can be managed using the following
features quorum based storage and shared
storage using network file system quorum
based storage refers to the H a
implementation that uses quorum journal
manager or qjm during this
implementation the standby node keeps
its state synchronized with the active
node through a group of separate daemons
called journal nodes daemons are long
running processes that typically start
up with the system and listen for
requests from the client processes each
daemon runs in its own Java Virtual
Machine JVM when any namespace
modification is performed by the active
node it durably logs a record of the
modification to a majority of the
journal nodes the standby node reads the
edits from the journal nodes and
constantly watches for changes to the
edit log as the standby node sees the
edit it applies them to its own
namespace in the event of a failover the
standby ensures that it has read all the
edits from the journal nodes before it
promotes itself to the active state this
ensures the namespace state is fully
synchronized before a failover occurs in
shared storage using NFS implementation
the standby node keeps its state
synchronized with the active node
through an access to a directory on a
shared storage device now in the next
screens we will discuss in detail the
components of HDFS which include name
node data node secondary name node and
file system namespace the name node
server is the core component of an HDFS
cluster there can be only one name node
server in an entire cluster name node
maintains and execute the file system
namespace operation
such as opening closing and renaming of
files and directories which are present
in HDFS the namespace image and the edit
log stores information of the data and
the metadata name node also determines
the linking of blocks to data nodes
furthermore the name node is a single
point of failure the data node is a
multiple instant server there can be n
number of data node servers the number
depends on the type of network and the
storage system the data node serves
stores and maintains the data blocks the
name node server provisions the data
blocks on the basis of the type of job
submitted by the client data node also
stores and retrieves the blocks when
asked by clients or the name node
furthermore it reads rights requests and
performs block creation deletion and
replication on instruction from the name
node there can be only one secondary
name node server in a cluster note that
you cannot treat the secondary name node
server as a disaster recovery server
however it partially restores the name
node server in case of a failure
the secondary name node server maintains
the edit log and namespace image
information in sync with the name node
server at times the name space images
from the name node server are not
updated therefore you cannot totally
rely on the secondary name node server
for the recovery process HDFS exposes a
file system namespace and allows user
data to be stored in files HDFS has a
hierarchical file system with
directories and files the name node
manages the file system namespace
allowing clients to work with files and
directories a file system supports
operations like create remove move and
rename the name node apart from
maintaining the file system namespace
records any change to metadata
information now that we have learned
about HDFS components let's see how name
node works along with the other
components name node maintains to
persistent files a transaction log
called an edit log
and a namespace image called an FS image
the edit log records every change that
occurs in the filesystem metadata such
as creating a new file the name nodes
local file system stores the edit log
the entire file system namespace
including mapping of blocks files and
file system properties is stored in FS
image this is also stored in the name
nodes local file system when new data
nodes join a cluster metadata loads the
blocks that reside on specific data node
into its memory at startup metadata then
periodically loads the data at
user-defined
or default intervals when the name node
starts up it retrieves the edit log and
FS image from its local file system it
then updates the FS image with edit log
information and stores a copy of the FS
image on the file system as a checkpoint
the metadata size is limited to the RAM
available on the name node a large
number of small files would require more
metadata than a small number of large
files
hence the in-memory metadata management
issue
explains why HDFS favors a small number
of large files if a name node runs out
of RAM it will crash and the
applications will not be able to use
HDFS until the name node is operational
again data block split is an important
process of HDFS architecture as
discussed earlier each file is split
into one or more blocks stored and
replicated in data nodes data nodes
manage names and locations of file
blocks by default each file block is 128
megabytes
however this potentially reduces the
amount of parallelism that can be
achieved as the number of blocks per
file decreases each map task operates on
one block so if tasks are lesser than
nodes in the cluster the jobs will run
slowly
however this issue is lesser when the
average MapReduce job involves more
files or larger individual files let's
look at some of the benefits of the data
block approach
the data block approach provides
simplified replication fault tolerance
and reliability
it also helps by shielding users from
storage subsystem details hey want to
become an expert in Big Data then
subscribe to the simply learned Channel
and click here to watch more such videos
centered up and get certified in Big
Data click here