YARN Tutorial | YARN Architecture | Hadoop Tutorial For Beginners | YARN In Hadoop | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
yarn is the acronym for yet another
resource negotiator yarn is a resource
manager created by separating the
processing engine and the management
function of Knapp reduce it monitors and
manages workloads maintains a
multi-tenant environment manages the
high availability features of Hadoop and
implements security controls the for
2012 users could write MapReduce
programs using scripting languages such
as Java Python and Ruby they could also
use Pig a language used to transform
data no matter what language was used
its implementation dependent on the
MapReduce processing model in May 2012
during the release of Hadoop version 2.0
yarn was introduced you are no longer
limited to working with the MapReduce
framework anymore as yarn supports
multiple processing models in addition
to MapReduce such as spark other
features of yarn includes significant
performance improvement and a flexible
execution engine now let's discuss yarn
with the help of an example
Yahoo was the first company to embrace
Hadoop and this became a trendsetter
within the Hadoop ecosystem in late 2012
Yahoo struggled to handle iterative and
stream processing of data on the Hadoop
infrastructure due to MapReduce
limitations both iterative and stream
processing were important to Yahoo in
facilitating its move from batch
computing to continuous computing after
implementing yarn in the first quarter
of 2013
Yahoo installed more than 30,000
production nodes on SPARC for iterative
processing storm for stream processing
and Hadoop for batch processing allowing
it to handle more than 100 billion
events such as clicks impressions
email content metadata and so on per day
this was possible only after yarn was
introduced and multiple processing
frameworks were implemented the single
cluster approach provides a number of
advantages including higher cluster
utilization where resources unutilized
by a framework can be consumed by
another lower operational costs because
only one do-it-all cluster needs to be
managed reduced data motion as there's
no need to move data between hadoop yarn
and systems running on different
clusters of computers the yarn
infrastructure is responsible providing
computational resources such as CPU or
memory needed for application executions
yarn infrastructure and HDFS are
completely independent
the former provides resources for
running an application while the latter
provides storage the MapReduce framework
is only one of the many possible
frameworks that run on yarn the
fundamental idea of MapReduce version 2
is to split the two major
functionalities of resource management
and job scheduling and monitoring into
separate daemons yarn and its
architecture in this topic we will
discuss yarn and its architecture
three elements of yarn architecture the
three important elements of the yarn
architecture are Resource Manager
application master and node managers the
resource manager or RM which is usually
one per cluster is the master server
resource manager knows the location of
the data node and how many resources
they have this information is referred
to as RAC awareness the RM run several
services the most important of which is
the resource scheduler that decides how
to assign the resources the application
master is a framework specific process
that negotiates resources for a single
application that is a single job
or a directed acyclic graph of jobs
which runs in the first container
allocated for the purpose each
application master request resources
from the resource manager and then works
with the containers provided by node
managers the node managers can be many
in one cluster
they are the slaves of the
infrastructure when it starts it
announces itself to the RM and
periodically sends a heartbeat to the RM
each node manager offers resources to
the cluster the resource capacity is the
amount memory and the number of V cores
short for virtual core at runtime the
resource scheduler decides how to use
this capacity a container is a fraction
of the node manager capacity and it is
used by the client to run a program each
node manager takes instructions from the
resource manager and reports and handles
containers on a single node in the next
few screens you will see a detailed
explanation of the three elements the
first element of yarn architecture is
resource manager the RM mediates the
available resources in the cluster among
competing applications with the goal of
maximum cluster utilization it includes
a pluggable scheduler called the yarn
scheduler which allows different
policies for managing constraints such
as capacity fairness and service level
agreements the resource manager has two
main components scheduler and
Applications Manager the scheduler is
responsible for allocating resources to
various running applications depending
on the common constraints of capacities
queues and so on the scheduler does not
monitor or track the status of the
application also it does not restart the
tasks in case of any application or
hardware failures the scheduler performs
its function based on the resource
requirements of the applications it does
so based on the abstract notion of a
resource container that incorporates
elements such as men
CPU disk and network the scheduler has a
policy plug-in which is responsible for
partitioning the cluster resources among
various queues and applications the
current MapReduce schedulers such as
capacity scheduler and the fare
scheduler are some examples of the
plug-in the capacity scheduler supports
hierarchical queues to enable a more
predictable sharing of cluster resources
the application manager is an interface
which maintains a list of applications
that have been submitted currently
running or completed the applications
manager is responsible for accepting job
submissions negotiating the first
container for executing the application
specific application master and
restarting the application master
container on failure let's discuss how
each component of the resource manager
work together the resource manager
communicates with the clients through an
interface called the client service a
client can submit or terminate an
application and gain information about
the scheduling queue or cluster
statistics through the client service
administrative requests are served by a
separate interface called the admin
service through which operators can get
updated information about the cluster
operation in parallel the resource
tracker Service receives node heartbeats
from the node manager to track new or
decommission nodes the nm liveliness
monitor and nodes list manager keep an
updated status of which nodes are
healthy so that the scheduler and the
resource tracker service can allocate
work appropriately the application
master service manages application
masters on all nodes keeping the
scheduler informed the a.m. liveliness
monitor keeps a list of application
managers and their last heartbeat times
to let the resource manager know what
applications are healthy on the cluster
any application master that does not
send a heartbeat within a certain
interval is marked as dead and
reschedule
to run on a new container before Hadoop
2.4 the resource manager was the single
point of failure in a yarn cluster the
high availability or age a feature adds
redundancy in the form of an active
standby resource manager pair to remove
the single point of failure resource met
AJ is realized through the active
standby architecture at any point of
time one of the RMS is active and one or
more RMS are in standby mode waiting to
take over should anything happen to the
active the trigger to transition to
active comes from either the admin
through the command line interface or
through the integrated failover
controller the RMS have an option to
embed the zookeeper based active standby
elector to decide which RM should be
active when the active goes down or
becomes unresponsive another RM is
automatically elected to be the active
note that there is no need to run a
separate zk FC demon-like in HDFS
because the active standby elector
embedded in RMS act as a failure
detector and a leader elector the second
element of yarn architecture is the
application master the application
master in yarn is a framework specific
library which negotiates resources from
the RM and works with the node manager
or managers to execute and monitor
containers and their resource
consumption while an application is
running the application manager manages
the application lifecycle dynamic
adjustments to resource consumption
execution flow faults and it provides
status and metrics the application
master is architected to support a
specific framework and can be written in
any language it uses extensible
communication protocols with the
resource manager and the node manager
the application master can be customized
to extend the framework or run any other
code because of this
the application master is not considered
trustworthy and is not run as a trusted
service in reality every application has
its own instance of an application
master
however it's feasible to implement an
application master to manage a set of
applications for example an application
master for pig or hive to manage a set
of MapReduce jobs the third element of
yarn architecture is the node manager
when a container is leased to an
application the node manager sets up the
containers environment the environment
includes the resource constraints
specified in the lease and any kind of
dependencies such as data or executable
files the node manager monitors the
health of the node reporting to the
resource manager when a hardware or
software issue occurs so that the
scheduler can divert resource
allocations to healthy nodes until the
issue is resolved the node manager also
offers a number of services to
containers running on the node such as a
log aggregation service the node manager
runs on each node and manages the
activities such as container lifecycle
management container dependencies
container leases node and container
resource usage node health and log
management and reports node and
container status to the resource manager
a yarn container is a collection of
specific set of resources to use in
certain amounts on a specific node it is
allocated by the resource manager on the
basis of the application the application
manager presents the container to the
node manager on the node where the
container has been allocated thereby
granting access to the resources now
let's discuss how to launch the
container the application manager must
provide a container launch context or
CLC this includes information such as
environment variables dependencies on
the requirement of data files or shared
objects prot to the launch security
tokens and the command to create the
process to launch the application this
ILC supports the application master to
use containers this helps to run a
variety of different kinds of work from
simple shell scripts to applications to
a virtual operating system
owing to yarns generic approach a hadoop
yarn cluster runs various workloads this
means a single hadoop cluster in your
data center can run MapReduce storm
spark Impala and more broadly there are
five steps involved in yarn to run an
application first the client submits an
application to the resource manager then
the resource manager allocates a
container then the application master
contacts the related node manager then
the related node manager launches the
container and finally the container
execute the application master in the
next few screens you will learn about
each step in detail users submit
applications to the resource manager by
typing the Hadoop jar command the
resource manager maintains the list of
applications on the cluster and
available resources on the node manager
the resource manager determines the next
application that receives a portion of
the cluster resource the decision is
subject to many constraints such as
queue capacity access control lists and
fairness when the resource manager
accepts a new application submission one
of the first decisions the scheduler
makes is selecting a container then the
application master is started and is
responsible for the entire lifecycle of
that particular application first
it sends resource requests to the
resource manager to ask for containers
to run the applications tasks a resource
request is simply a request for a number
of containers that satisfy resource
requirements such as the following
amount of resources expressed as
megabytes of memory and CPU shares
preferred location specified by hostname
or rack name priority within this
application and not across multiple
applications the resource manager
allocates a container by providing a
container ID and a host name which
satisfies the requirements of the
application master after a container is
allocated the application master asks
the node manager managing the host on
which the container was allocated to use
these resources to launch an application
specific task this task can be any
process written in any framework such as
a MapReduce task the node manager does
not monitor tasks it only monitors the
resource usage in the containers for
example it kills a container if it
consumes more memory than initially
allocated throughout its life the
application master negotiates containers
to launch all the tasks needed to
complete its application it also
monitors the progress of an application
and its tasks restarts failed tasks in
newly requested containers and reports
progress back to the client that
submitted the application after the
application is complete the application
master shuts itself and releases its own
container though the resource manager
does not monitor the tasks within an
application it checks the health of the
application master if the application
master fails it can be restarted by the
new resource manager in a new container
thus the resource manager looks after
the application master while the
application master looks after the tasks
Hadoop includes three tools for yarn
developers yarn web UI hue job browser
yarn command-line these tools enable
developers to submit monitor and manage
jobs on the yarn cluster yarn web UI
runs on 8088 port by default it also
provides a better view than hue however
you can't control or configure from yarn
web UI you will see a demonstration on
yarn web UI in the later screen the huge
job browser allows you to monitor the
status of a job killer running
and view logs you will understand this
by viewing a demonstration in the later
screen most of the yarn commands are for
the administrator rather than the
developer a few useful commands for
developers are as follows to list all
commands of yarn type the command line
shown on-screen to print the version
type the command line shown on the
screen to view logs of a specified
application ID type the command line
shown on your screen using yarn web UI
hugh job browser and yarn command line
in this demo you will learn how to work
on yarn web UI hugh job browser and yarn
command line you will use the file word
count dot py which is a python file this
file will calculate the number of times
each word appears you will view how to
execute this file with the help of yarn
type spark
- submit
let's define the master as yarn - client
followed by the name of the Python file
you need each word count which is
present in the word directory loud Acre
Kb
click job browser while the program is
running on the terminal let's see how
the steps executing in yarn appear in
hue
you will notice the Python word count
status as running it will also show you
the status of map and reducer click ID
you can also click on ID option to view
more details of a job
you can also click on kill option to end
this job from UI
you
once you enter the application ID you
can view the running metadata of that
yarn in this page you can also view the
information such as start time end time
and the amount of memory per second that
a running program consumes let's go back
to the terminal
you
you can check similar details in web
user interface which you just viewed
from yarn resource manager you need to
navigate to localhost colon 8080 eight
slash cluster you will again notice the
running ID
click this ID to view more details
you will be able to view information
such as user name of the program
application type current state of the
running program and a few additional
information you can also scroll through
the accepted jobs currently running jobs
finished jobs and a few more details so
this is another web UI page in which you
can monitor your yarn progress once this
program is complete you can view the
output in the terminal it displays the
number of times each word appears this
brings you to the end of this demo in
this demo you learnt the steps to
calculate the word count for a file you
have also learned the steps to monitor
your yarn progress in a web user
interface hey want to become an expert
in Big Data then subscribe to the simply
learned Channel and click here to watch
more such videos to nerd up and get
certified in Big Data click here