Apache Hadoop Cluster Setup | Apache Hadoop Tutorial For Beginners | Hadoop Training | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
in this demo you will learn how to set
up Apache Hadoop cluster into our Linux
machines
first set up the machines in which
Apache Hadoop cluster are installed
both the machines can ping each other
and has secure shell ssh connection
between them then you need to edit the
hadoop config files to set up a cluster
Apache Hadoop cluster is a core
distribution of Hadoop therefore edit
the Hadoop config files manually if you
are using a vendor specific distribution
like cloud era or Hortonworks some of
the configuration steps will be
performed by pluster management solution
like cloud era manager or Ambari once
your machines are loaded log into your
machines
you
first configure the files in one machine
and then copy the files into the other
machine and start configuring in machine
m1 first login to your terminal using su
- HD user
you
this will act as an admin from my
cluster once you log in get your Java
path using the terminal type echo Java
underscore home then you will get your
Java path
you
now you will get the Hadoop path using
the code CD - USR - local - Hadoop
then type ls' using which you can know
what Hadoop directory contains to locate
the config files
type ls' etc' slash Hadoop to locate
binary files type ls' bin and to find
startup script type LS s bin these
scripts will be used to start Hadoop
cluster
now type cd' etc' slash Hadoop and then
type PWD this will take you into USR
slash local slash Hadoop slash etc'
slash hadoop the first file that should
be edited is hadoop - env SH which is a
Hadoop environment in order to edit the
file go into VI editor using command VI
Hadoop - env SH at the bottom you can
find export Java underscore home equals
dollar sign curly braces Java underscore
home closed curly braces which is a Java
path which is already set if you changed
the Java packages later you need to
change it here as well when you scroll
down you can find export Hadoop
underscore heap size which indicates the
amount of ram that will be used to show
the name node similarly you can setup
the amount of RAM for every daemon to be
displayed on screen give the heap size
as 200 which is 200 MB when you scroll
down you can see the log details like
where your logs process IDs will be
stored once you edit the Java path and
heap size save the file and close it
you
next edit the core - site XML file in
this file you can find the HDFS path
where the name node will run and what
port it uses in order to edit the file
type VI core - site dot XML under
configuration start entering the details
type property
enter named FS default FS close name
you
then give the value by typing value HDFS
colon slash slash m1 : 9000 close value
if you want to keep a note give it as
description
this is my HDFS path and location of
name node closed description
you
then close the property with closed
property default value name note is 9000
or 8020 and for resource manager is 9001
or 8021 the next file to edit is HDFS
using VI HDFS - site dot XML under
configuration give a couple of
properties using property
enter named DFS replication close name
if you do not give this command cluster
will take the default replication then
give the value to close value
enter close property which will be the
number of data nodes but the maximum
value that you can give should be the
number of data nodes that are present in
order to give second property which will
store the metadata type property
enter named DFS name note name dir close
name
enter value
/ simply el / name clothes value
enter close property you give this
command in order to save the metadata
into the drive if you do not give this
command metadata gets stored into a temp
directory so if it gets stored into a
temp file when you restart the system
the data will be deleted the next
property is for data node for this type
property enter name DFS data node data
dir close name
enter value / simply L / data one close
value
enter close property
the next property is used for secondary
checkpointing for this type property
enter name DFS name node HTTP - address
closed name
enter value
M 1 : 5 0 0 7 0 clothes value
enter close property give another
property which is a secondary name node
by typing property enter name DFS name
node secondary HTTP - address
clothes name enter value M 3 : 5 0 0 9 0
clothes value
enter close property by giving this the
secondary name note we'll start on
another machine then save the file
the next file to edit is MapReduce file
by vi MapReduce - site XML template and
click enter in this case you need to
enter framework in order to do that type
the following commands in configuration
property
enter named MapReduce dot framework dot
name close name
enter value
yarn clothes value enter clothes
property and click Save
the next file to edit is VI yarn - site
XML this file is to know the resource
manager list of classes that the
framework will use in order to enter the
commands type the following commands in
configuration property enter name if you
don't know the property names you can
find it in the hadoop website
by default the resource manager starts
where the name node starts
you
yarn dot resource manager dot address
clothes name enter value M 1 : 9001
clothes value enter clothes property the
next property is for resource tracker
property
enter name
yarn dot resource manager dot resource -
tracker dot address
you
clothes name
enter value M 1 :
80 31 close value
enter close property the next property
is for classes you should give it as
property
enter name yarn node manager dot aux -
services dot Map Reduce
underscore shuffle dot class
you
clothes name value org apache hadoop
mapreduce shuffle handler
you
clothes value enter clothes property
the next property is for ox
you should give it as property enter
name yarn node manager dot ox - services
clothes name enter value MapReduce
underscore shuffle clothes value enter
clothes property in order to know the
location of data nodes edit the slave
files by VI slaves delete if there are
any existing entries and type m1 enter
m3 and save these are the two machines
where you intend to run our data nodes
and node manager once you are done with
configuring all the above files you will
now copy the configured files into other
machines in order to copy the files to
another machine type SCP Hadoop - Ian vs
h HD user at m3 colon slash user slash
local slash Hadoop slash etc' slash
Hadoop
the next file is the core - site file
for this type SCP core - site XML HD
user at M 3 colon slash user slash local
slash Hadoop slash etc slash Hadoop the
next file is the HDFS - site file for
this type SCP HDFS - site XML HD user at
M 3 colon slash user slash local slash a
dupe slash etc slash Hadoop to copy our
yarn - site file for this type SCP yarn
- site dot XML HD user at M 3 colon
slash user slash local slash Hadoop
slash etc slash Hadoop the last file
which you copy is the MapReduce site
file for this type SCP MapReduce - site
XML template HD user at M 3 colon slash
user slash local slash Hadoop slash etc
slash Hadoop if you intend to start
resource manager in some other machine
you need to copy the slaves file for
this type SCP slaves HD user at M 3
colon slash user slash local slash
Hadoop slash etc slash Hadoop with this
you complete the copying of files into
other machines in order to check from
other machines log in to other machines
and type the initial commands for user
login and Hadoop files
to change the directory type CD etc
/ Hadoop enter LS to check the config
files
in order to look into the files type VI
Hadoop - env SH and check all the
configured files in the second machine
you can delete our name node as it is
not required in this machine
you
under VI HDFS - site XML you need to
change the subdirectory data 1 - data -
as you intend to work on another machine
you
after HTTP port you have to give the
secondary name node here in this machine
so give it as property
enter name
DFS nay
no dot checkpoint dot dir
clothes name enter value simply L / s NN
FSI clothes value
enter close property the secondary name
node will store the name node edits in
order to give a command type property
enter named DFS name note checkpoint
edits dir
clothes name enter value
simply l /
snn edits
clothes value enter clothes property
in order to mention the checkpoint type
property enter name DFS name node
checkpoint period
clothes name enter value 600
clothes value enter clothes property
save the file
once you have checked all the files you
are ready to create the cluster before
that one more important point is when
you work on Hadoop you need to create a
parent directory mentioned in the config
files
you
in order to do that sudo make directory
slash simply L and hit enter
you can also change the owner by typing
sudo CH
- capital r HD user : Hadoop dot simply
L and save the file in order to check
the ownership type LS - all slash
you can also change the ownership in
another machine that is the m1 machine
using the similar command
you
always remember that the first demon
that turns up his name node and to bring
up name node metadata is mandatory type
HDFS in the terminal to get all the
possible options which are available
with HDFS so in order to check the name
node type as per the options as HDFS
name node - format by doing this the
initial formatting is done
and the metadata is also created in the
path which is mentioned in HDFS - site
file
you
if there is any issue with respect to
packages type yum update
you
when the packages are getting updated
you can open up a new terminal in
machine m1 and log in as root when you
give a command iptables
- flush it flushes all the rules which
might prevent communicating between the
two machines when you type service
iptables stop this stops if there is any
firewall in connection
finally when you type check config
iptables off this makes sure the IP
tables are turned off when the machines
are turned on follow the same commands
in other machines as well
you
in this demo you have learned how to set
up Apache Hadoop cluster into our Linux
machines Hey want to become an expert in
Big Data then subscribe to the simply
learn Channel and click here to watch
more such videos to nerd up and get
certified in Big Data click here