Easiest way to install / setup hadoop | Hadoop tutorial Video Lecture | Hadoop Tutorials: Brief Introduction - Software Development

Hi welcome to the lesson setting up hadoop
At this point in time, I have installed Ubuntu a fresh
on my VMBox
and I have downloaded this Document
Setting up Hadoop 2 Made Easy on my desktop
I have attached the copy of this document with this lesson
As well this video is downloadable and if you wish to download it.
I'll follow this document completely
to show that this document has all the necessary steps
and it is pretty simple for you to install hadoop
So we have installed Ubuntu
and we now start out with installing Java
I'll copy this command
And open the terminal.
And paste the command
It asks for a password as I am using 'sudo'.
And Java gets installed
So as a next step
I type in Java --Version
to see if the Java got installed properly
So it has.
In the next step we would install openssh and create a new username and group.
To install openssh, I'll just copy paste the command.
And the installation happens.
And this installs the open ssh.
Then we would create another username and group
and use that username to run our hadoop jobs.
We do so because while installation there are sometimes
a few very annoying permission errors which can come up.
In order to avoid it, we would setup a new user account
So first we create a group Hadoop
So I'll copy paste this command.
Then we'll adduser hduser to this group
by using the command adduser.
Here we enter the password we want for the new account hduser.
Then we add this user to Sudo group as well.
And as the document says, I'll just re-login to the newly created account.
So I'll logout now.
And now I login to the newly create account hduser.
Now I need to copy the document.
So I go to Computer.
Then home
then nj
and then the downloads where I had the copy
and then copy it back to my download folder in hduser
and look for the next command.
Next we setup a password less ssh
so that we do not need to keep entering the password at the start of ever hadoop demon,
So I open the terminal.
Then I paste the command.
Then I again paste the next command.
And now I'll do ssh localhost which won't prompt me for a password.
Okay I'll do ssh localhost once more.
and now it prompts me for nothing.
Now the next step is to download Hadoop
So I open my web browser and google search download hadoop.
And in the first link itself you'll see the link to apache hadoop.
Then click on Download.
Then download a release now.
Then click on the suggested mirror site.
And then look for stable folder.
And then download the tar.gz file.
It will take a little while to download and I'll forward the video here.
So now the file has been downloaded.
Now next we go to downloads folder.
So I copy and paste the command.
Okay there is a capital 'D' in downloads and I'll change the documentation.
And now again copy paste.
Now we untar the file and in the same command move
the unzipped to the folder /usr/local
So I copy the command and paste it.
Type in my password and
And now the untarring happens.
Now next we move into the folder /usr/local
So I copy paste the command.
Then we would change the name of the folder to hadoop 2.x to hadoop
so we need not type in the complete version number again and again.
Then we change the ownership of the folder to the hduser
to avoid the annoying permission issues.
If you see, hadoop folder is not owned by hduser, but after I run this command
And now it would show that it is owned by Hduser.
And now we would setup a few variable in files.
First of all we edit bashRC
Bashrc is the script
that runs as soon as you start the terminal on linux.
in this script we will add a few variables which would set a few system variables
which would be used by Hadoop scripts.
So the variables are Java_Home, Hadoop_Install
and we would add the bin folder in the active path
so the scripts are available
and then we need to set these variables
as they would be used by hadoop scripts.
So I'll just goto folder and goto home
Here I would goto view and click on show hidden files.
And now here you can see the .bashrc script.
.bashrc script run with every start of the terminal
and .profile is another script that runs when the system starts.
So we can use either of these scripts to set our variables.
and it would work fine.
So in this video I am using bashrc file
So I open the bashrc file.
And then I copy
And paste the variables at the end.
Now next we edit the script hadoop-env.sh.
This is Hadoop scripts which sets all the environment related variables used by Hadoop.
So for that I goto system level.
Then goto usr
and then to local
then to hadoop
basically we have installed our hadoop here.
And now we would goto etc folder
and then hadoop
hadoop and this folder has all the configuration files for the hadoop installation.
Next I open hadoop-env.sh
Here you see the JAVA_HOME variable.
Next I'll goto the directory where Java is installed.
So it is in /usr
/lib
/jvm
and I'll go in one more folder.
and then I would type PWD to get the correct present working Directory.
It is very important to copy the correct path and avoid typo
and I'll copy and paste it in the hadoop-env script.
It is this way we tell hadoop where Java is installed and can be found in the system.
Next we restart the terminal so that the changes in bashrc
are refreshed with the fresh login in the terminal.
And now if we type in hadoop version now
then we would be able to see the version number.
Next we update a few Hadoop Configuration files
starting out with: core-site.xml.
So the hadoop configuration files are xml files and we specify important attributes
, in this case fs.default.name
This property specifies host and port pair where the namenode services would be running.
Next we update yarn-site.xml
So again we specify a couple of YARN properties in this file.
Next we update mapred-site.xml
For that I will copy the file mapred-site.xml template
and rename it as mapred-site.xml.
So here we specify the framework which we are using.
I have specified yarn here.
Classic can also be specified to run it in the classic mapreduce mode.
Next we update hdfs-site.xml with all these values.
Basically dfs.replication sets the replication of block to 1
as we are installing this in a pseudo-distributed manner.
and so we do not want replication of the blocks.
Then we mention the folders that would be used by datanode
and the name node demons in order to maintain the filesystem.
We do it by specifying the properties dfs.namenode.name.dir
and dfs.datanode.data.dir
Please note that we haven't created these directory paths till now.
By executing the next command, that is, formatting the name node
hadoop demons would automatically create these directory paths.
So Next we execute the format command.
I did not get the message that I was looking for so I would do it once again.
And now I type Y as a confirmation
and I got the successfully formatted message.
Next I'll start the demons.
So I type in start-dfs.sh
And it start the namenode and data node and Secondary namenode.
Please observe the log file paths for each of these demons.
If there is any error and one of the demons doesn't come up the right way,
you would have to look at it logs to see the error.
JPS shows the demons that are running
Then start-yarn.sh start the resource manager and node manager.
Now when I do JPS I see all the demons.
If you see all the 5 demons running here,
please accept my congratulations in installing hadoop.
Installing hadoop is a little tricky and takes time.