Big Data Hadoop Tutorial For Beginners | What Is Big Data? | Big Data Tutorial | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

let me introduce myself i am german i
have about 20 years of experience in
software industry work with lot of
product companies earlier and last four
years i am associated with big data work
doing a lot of consulting and providing
solutions to the corporates were seeking
for solutions in this space using these
technologies so that's pretty much brief
background about myself i also do a lot
of corporate trainings across the globe
I have a train about 2,000 people ok so
that's a brief bio about myself now what
we will have in place for you we will
basically I'll just let me walk you
through the agenda so anyway before we
go to the agenda let me only tell you so
these are the two basic concepts which
everyone is I think aware of or would
have heard of and we really want to
understand two basic questions related
to them what is Big Data and why it is
important and the second question is
what is Hadoop and why Hadoop is chosen
to be the technology to work with big
data okay and then we will also have a
small demo plan for you at the end of
the session where we will help you
understand how to work with HDFS a
distributed storage of Hadoop ok so
let's get started so first let's
understand what is Big Data
ok and for that we basically want to
understand these three basic parameters
of Big Data I'm sure many of you would
have heard about it
so Big Data basically is formally
defined by IBM as the company we'll talk
about what is Big Data
so big data has been defined formally by
IBM IBM is the first company who has
formally define what is Big Data so they
have defined three basic parameters with
respect to Big Data which is volume
variety and velocity ok and let's let's
try to understand each one of them so
what is volume as the name suggests
volume stands for huge amount of data
that means we can talk about terabyte
and petabytes of data that's the kind of
volume we are talking about what is
velocity where ITIN velocity so let us
understand that as well velocity means
the rate at which data comes to your
system or data gets generated into your
system okay so let's take two scenario
to understand that in the first scenario
let's say if I say I have one terabyte
of data which comes to my system every
day right so I am generating huge amount
of data with almost every day which
means the velocity is very high to
generate such huge volume of data that
is for scenario in the second scenario
if I say that I am generating one
terabyte of data but I am gen I'm taking
one year time to generate it so that
means the rate at which I'm generating
such a huge volume of data is very low
right it's not very high so when you
have a velocity which is not very high
and when you are generating such a huge
volume of data with a very low speed
then it doesn't really make sense to use
systems like Hadoop right but the moment
when you say you have high volume of
data and you are also generating it very
there piddly very fast then you need
systems like Hadoop to handle that kind
of heavy load okay and the third
dimension is basically a variety so
variety stands for the different forms
of data so any data you can by the way
categorize into three different buckets
the structural data unstructured data or
semi structured data right so let me
give you example of all of them what is
structured data structured data means
any data which where you can apply
schema definition and example could be
the data which you store it in normal or
DBMS systems normal database right in
the form of table rows and columns that
is what we define it as a structured
data what is unstructured data it's a
reverse definition of structured data
which means a data where you cannot
impose any schema definition that is
what we mean by unstructured data and
the example is log files
log which is generated from you know
application servers video files images
photos when we write emails to friends
the text message or
in fact the subject line as well as the
body message of the main email which is
also uh unstructured data so any simple
text can also be categorized as an
unstructured data example and the third
type of data is called semi structured
data example is basically XML file or
JSON files which is not storing data in
a completely structured or unstructured
format and hence we call it as a semi
structured data so in a nutshell if I
have huge volume of data which which I
am generating at a very high speed high
velocity and it can be of it is coming
in any form like structure
non-structural semi structure then we
call it as a big data so that's the
formal definition given by IBM ok now
let us understand more about some of the
applications of big data and we really
want to understand the importance of big
data so why big data and hadoop why
someone needs to really choose this
carrier in big data space so our team
has done a good analysis on that and
they found out some interesting steps
about it from recognized websites which
says as you can see from the screen it
says that the salary train is very high
in big data compared to last couple of
years and we are comparing it with some
of the high-performing technologies
right like Java and.net so cellular in
training big days it big data is
definitely it's much higher than other
technologies ok there are some other
steps published by Forbes article Forbes
insights which says 66% of the
organization report that big data and
analytics initiatives have had a
significant measurable impact on the
revenues which means they get profits
and revenues by using such kind of
technologies and there is another
article published by one of the reputed
consulting firm McKenzie which says buy
there will be a shortage of 1.7 million
people with big data skills in US alone
which means there will be lot of work
and a lot of demand which is growing in
this space and it's definitely one of
the area which someone
look into someone should bill carrier
into right because that has a definitely
very very bright future and a lot of
opportunities out there okay so that's
basically some of the you know important
facts which helps us understand the
importance of big data as well okay now
let's talk about couple of case studies
and understand how big data has been
used by them all right so what we really
want to understand is in fact in the
actual training will be covering all the
case studies which are mentioned here as
you can see here okay but for today in
the interest of time we would be able to
see only a couple of case studies so
let's take examples from let's say
social media social media is definitely
one of the interesting use case because
a lot of people are using it so for
example Facebook or Twitter right so we
are let's say using Facebook and Twitter
every day to do lot of social networking
and how does let's say Facebook
generates revenue or how does it use
this technology to generate a revenue so
I'll spend two minutes on understanding
that I am sure many of you are aware
that Facebook uses does a lot of
advertising based on which it generates
revenue but how does it actually do that
advertisement so there are two forms of
advertisements which are very prevalent
in the market one is called mass
advertisement one is called targeted
advertisement so mass advertisement
means let me give you an example so
let's say two people sitting in
different cities like let's say one
person is sitting in Bangalore one
person is sitting in let's say Mumbai
and both of them are actually watching
the television at the same time both of
them are watching live cricket match and
there may be a possibility that in
between there is some advertisement
which comes and both of them will have
to see that ad unless they switch off
the channel so what I'm trying to
highlight a point is they are not given
an option where they can actually choose
not to see this ad right which means
advertisement industry who is
broadcasting this adds to the users they
don't really care whether user is
interested to view this head or not
right that means it is called master
there is another type of advertisement
which is called targeted advertisement
and it is actually being mainly promoted
by Facebook with the help of Big Data
technology so what they have done they
are using big data technologies to
analyze the data which is there with for
every user so every user is registered
on the Facebook right and actually they
are doing a lot of activities on
Facebook so what happens here is when
they are doing all these activities
actually our Facebook register remembers
or analyze your recent activities and
based on that they do something called
sentiment analysis to find out what are
the likes and dislikes of this user and
accordingly they actually generate or
they basically you know show different
ads which are relevant to your interest
area on your Facebook page right the
sentiment analytics is an area and
machine learning right and there'll be
algorithms available so they run these
algorithms on a massive Hadoop clusters
which they have on in-house to analyze
this data to basically recommend
interesting ads okay now how does
Facebook makes revenue out of it so when
you show an ad on a Facebook page when
you just click on that ad which is being
shown on your Facebook page Facebook's
your nips array when you just by
clicking it even if you don't know after
that but just you click on that still
Facebook make some money if you let's
say well click on that and if you do
some transition Facebook makes a major
share of revenue so there are various
contracts based on that Facebook makes a
revenue but this is how basically
Facebook is using Big Data technologies
to generate revenues right that's one
interesting use case twitter is also
doing similar thing to analyze to is
data to come up with interesting
recommendations with respect to which
products you want to buy and all that
there is another use case which I want
to talk about from the retail industry
I'm sure everyone does lot of online
shopping right everyone is doing
shopping with let's say flip card or
Amazon all of these companies are
actually doing something called
recommendations so when you let's say
you have done some shopping of books on
Amazon and the next time when you log in
to Amazon you will see that there is a
section on the page which says that
these are the books which are
recommended
to you right and even if you have let's
say not bought any book but you have
always browsed a particular book or
authors books right then also phase
sorry Amazon remembers your past
browsing history your click click stream
data it is called quick stream analysis
and based on that they basically you
know analyze this data and recommend you
and relevant products related products
to you right now the state says that
chances are that the hit ratio is 60 to
70% which means six out of seven times
six out of ten times a user who clicks
on those products from the
recommendation section will end up
buying that product right which is a
very good marketing technique this is
called passive marketing with which they
are actually increasing the revenue
right so that is also one area where
Facebook Amazon generates a huge amount
of money for because of that okay now
actually similar to that we are also
going to talk about couple of it other
interesting you know technical points
about Hadoop I will not probably take up
any other case studies in the interest
of time so we just want to understand
these two case studies to understand how
other companies are using big data
technology to solve some problems now
let us understand more about you know
Hadoop as a technology itself and what
kind of advantage it provides all right
so what is heard by the way Hadoop
actually is a is a framework it's an
open source framework written in Java
okay which supports processing of large
data sets in a distributed computing
environment and as well as it also
supports storing of large data sets in a
distributed manner okay so let me let me
tell you some of the use case I will
open another screen and show you that
okay so before we understand how do I
really want to understand one important
thing which is related to distribute it
for her system so Hadoop by the way is
using a distributed file system to store
the data all right
and it has a lot of data so if you see
there is a chart which says how big data
has been growing actually so the curve
is like that so initially this is all
about you know 1990 is 95 99 then about
0 5 mm 5 10 15 and so and so forth so
this is one of the use case which is
published in fact one of the chart or
analysis which is published by one of
the reputed organizations which talks
about how big data has been growing over
the years and as you can see till late
1990s the data volume was very less but
from early 2000 onwards the data volume
has been growing exponentially
the reason being internet as a primary
driver a lot of people do a lot of
activities on internet which is why it
is growing very fast ok so - now
initially people never had a challenge
in terms of storing this data because it
was very less they could store it in a
single machine but now because the data
volume has been growing they have a
challenge in terms of storing the data
and they need a system which can support
such kind of storage massive storage
right so they can store it in a single
machine so they need multiple machine to
store the data and that's where
distributed file system came into
existence so let us understand more
about why distributed file system or
what sort of advantage it provides so I
have to use case for you let's say the
first scenario is I have one single
machine which has for our channels and
100 Mbps speed and if I store one
terabyte of data in that machine and now
if I want to let's say read one terabyte
of data from that machine how much time
it is going to take ok so there is a
formula available if you put all these
values it will say it is going to take
approximately 45 minutes of time which
is very very slow right so can we do
something better yes what if I will have
multiple machines each of them having
the same capacity but now I will store
this 1 terabyte of data in a distributed
manner which means some portion I will
store it here some
I'll store it in this machine so
likewise I am dividing your data and
storing it in a distributed manner in
different different machines with that
scenario if I want to read this data I
can read all the parts parallely and
hence I can complete a processing of one
terabyte of data in a distributed and a
parallel manner within four point five
minutes of time so I if I have ten
machines okay so basically with that
approach I am able to increase the
performance ten times which is a massive
improvement on the performance and that
is the fundamental advantage which is
provided by distributed file system
okay so distributed file systems major
advantage is this and Hadoop is also
using internally distributed file system
to store the data which is called HDFS
all right so let's talk more about some
of the advantages that Hadoop provides
so any distributed file system will be
basically considered as a master/slave
architecture you have a master and
multiple slave machines okay and let's
assume that we have one terabyte of file
which is stored in this distributed file
system so it is stored in let's say
different different machines so each
part of this file is physically
distributed in four machines okay now
what happens here when you are storing
this file right so there is let's say a
virtual layer which is created which
which basically is is referred as a
distributed file system distributed file
system is nothing but it's a virtual
file system so you store this file in
that virtual file system but internally
it is physically divided into four
different parts and it is stored in the
four physical machines which are running
underneath right master does not store
any data it just have a metadata which
means this file has how many parts and
which part of this file is stored with
which slave machines is with flayman
means which slave - and so and so forth
this is called metadata information okay
so basically
this is the high-level overview of a
distributed file system how a
distributed file system looks like okay
now let me give you some scenarios here
so let's say in the first scenario there
is a client who wants to read these file
so he will always have to talk to the
master because master knows where this
file is stored in which all parts it is
located right in which old machine so
client will always talk to the master to
read the information about the file
right read the data about the file let's
say when kind wants to read a file
during that time one of the machine goes
down in this cluster when I say cluster
it is nothing but its set of machines
connected over a network okay so if one
of the machine goes down then obviously
client cannot read one of the part
called Part B from this file which means
client will read incomplete data which
is not an acceptable scenario right so
what heard of does it it uses as a
distributed file system it actually
supports a feature called replication
because of this replication fee factor
you can actually configure it two to
three or whatever because of that every
block or every copy of this block will
be replicated two times in two different
in all the machines okay now what
happens in this case even if one machine
goes down I can still have a copy of
Part B available with one of the other
machines so that plant can still read
Part B even in case of failure this is
called high availability of the data and
hadoop guarantees high availability of
the data even in case of failures
because of replication and there is
something called RAC awareness policy
which it follows so the second advantage
is basically let's say in terms of
processing of the data so Hadoop
actually guarantees faster processing of
the data the big data right why does it
so so why does it do that way and and
how does it achieve that fast
performance so let us understand that so
I'll give you very simple differences
between traditional distributed system
and Hadoop right so let's take consider
a scenario where there is a client who
wants to let us say processor data so he
will submit proof
to the masternode who wants to process
all these one terabyte of file right now
when a program is submitted to the
master what happens here any I'm first
talking about the traditional systems
right how traditional systems used to
process the data so when you submit a
program the master knows that where this
file has been divided and which all
parts are stood in which machine so it
will bring all these parts of the file
to the master machine it will bring all
the parts to here and then here it will
combine the data and then program will
go and process this data so what is
happening here big you of you are
transfer you are actually trying to
process one terabyte of data so you are
actually sending almost one terabyte of
data over a network which means it is
occupying lot of network bandwidth that
is first thing and the second thing is
after transferring all the data it is
accumulating everything in one place and
then it is trying to process that data
which means it is actually not doing
parallel processing right so there are
multiple flaws with this approach
one is it is occupying lot of network
bandwidth second is it is not able to
achieve parallel processing so because
of this flaws that it has it is not
really getting a good performance of
processing data in a distributed manner
what Hadoop has done it has actually
done it has been in fact reverse the
entire processing scenarios and it has
done something in a very reverse way so
let me explain you that so instead of
doing it that way
what Hadoop does is it will actually
send instead of transferring data to the
machine where program is running it will
do the reverse very transfers
computation to the machine where data is
stored okay so it will be something like
this it creates multiple instance of the
program and it sends it
to the different machines where the data
is stored now what happens every
instance of the program is going to
process each part of this file p1 is
going to process a p2 is going to
process B b3 is going to process C and
so and so forth the advantage here is
you are not transferring lot of data or
a network so you're saving network
bandwidth and the second thing is you
are actually doing a parallel processing
here because all of these parts data I
can process it in parallel E so it is
generating an intermediate output by
processing every part of the data like
this once it completes this processing
this is not your final stage it has just
completed in T in you know intermediate
blocks it has just processed individual
blocks data now after this what happens
Hadoop's
follows another step where it actually
combines all this output data ok it
combines all this data and send it to
one of the machine where it runs a
process called reducer which process
this data and generates output final
output ok so this is how Hadoop
completes the entire processing now
let's not worry about what is reducer
and all that thing that we are going to
discuss in the actual training sessions
where we will have more detail
discussions on that but this is
basically an approach that Hadoop
follows where the fundamental principle
is it transfers computations to the
machine where program data is stored
okay and with this approach it is able
to generate 10 times faster performance
because it achieves parallel processing
and it saves lot of network bandwidth ok
this approach is called map and reduce
ok and it is using one design principle
called divide and conquer which many of
you are already aware of right which is
used in many data structure problems
like merge sort and quicksort so divide
and conquer means what you have a bigger
problem to solve
you first divide that problem into
smaller tasks you first independently
and parallely solve those parallel solve
those independent tasks the smaller task
and whatever output is
generated you then combine them and
process it one more time to generate
final output that is called divide and
conquer and the exit pattern has been
followed by map and reduce also okay and
that is why it is achieving 10 times
faster performance than the traditional
system so that is one of the fundamental
advantage that Hadoop provides now let
us also discuss some processes which are
there in the Hadoop cluster so by the
way there are there is a master node and
there are slave machines the master node
will have a processes running like name
node and a job tracker and the same node
will have a process is running like data
on an elastic this entire for machine
cluster is called a Hadoop cluster
so here name node and data node are the
two Java programs or we can call it as a
process which are basically there to do
the HDFS operation which means it helps
you to store data store big data in
Hadoop cluster and job tracker and TAS
tracker are another processes another
Java programs which are there to help
you do MapReduce kind of operations
execute map and reduce tasks basically
process they are dealing with processing
of the big data
so overall SD FS + MapReduce together is
called Hadoop so you should remember
Hadoop definition in this way if you
remember it this way you will never
forget what is Hadoop okay so this is
the Hadoop framework that is being
defined there is also one more
interesting component Kalyan which is
being added in the latest release of
Hadoop
but we will definitely talk about it
during the actual training sessions not
now okay so these are some of the
advantages that Hadoop has and and and
now I would like you to go through the
history of Hadoop
so we have in fact understood as of now
what is Big Data why big data is
important we also covered some case
studies we understood the importance of
how to do we covered some advantages
that Hadoop provides and I will now
brief you with the history of Hadoop so
here we have let's say about in 2003
time frame Google has published a paper
a white paper
GFS and MapReduce okay from that point
of time there there was a guy called
that cutting in fact he still and so he
is basically the one who was implemented
this concept initially he started with a
project called much were implemented
this later on he was hired by Yahoo and
they started a project where they
implemented this concepts from scratch
and later on they named it as Hadoop and
then later on you know that they that
became the top-level Apache project so
yeah who has you know given it as an
open-source project and since then it is
managed by Apache okay and since then
obviously after 2009 timeframe and there
are regular contributors to Hadoop
framework like you have Facebook cloud
era Yahoo these are the companies who
are regularly contributing to Hadoop
okay
by the way cloud era is a company which
is started by the same guy called
necrotic who's the father of holo so
that's the how the history has been a
war for 400 okay now what we can do is
is basically let me show you a demo one
of the interesting demo that we have so
basically there is a there is a cloud
setup which is there with which is
basically provided by simply none so let
me first brief you this head weren't
ages before we actually go to the demo
right when I say advantages it is not
related to technology but it is related
to the simply learn as a company right
why someone really needs to go for a
training from simply and only and I find
it very interesting because there are a
lot of key advantages that simply learn
weights which you don't find it anywhere
which is hundred plus hours of training
in Big Data Hadoop with the latest
version of photo which is 2.7 so they
have actually formed a cluster in cloud
right nobody needs to install Hadoop on
your machine right any participants
doesn't need to really worry about his
hardware configurations and lot of you
know compatibility issue with his OS
versions and all that
now that it's no more there they have a
latest sort of version they have a
Hadoop clusters running in cloud which
has multi node cluster you will actually
get a get basically access to those
multi node clusters for you know when
you are enroll for this training and you
can play with it you can really get a
feel of working on a live cluster right
and obviously we will train you during
the actual training how to access the
cluster and how to run MapReduce
programs how to access different
components how do basically execute hive
queries or pick scripts or as base
commands using that cluster so a lot of
stuff stuff will be handled using the
live cluster and apart from that the
important part is it's the latest
version so you'll actually get to see
the latest API is in latest framework
related components with the help of that
cluster you will definitely get a hint
on project execution with cloud lab so
cloud lab is an environment which will
be provided by simply learn to each
participants right it's there is a
single pass so you attend multiple
batches on weekends or weekdays and
there is a time schedule given on the
right hand side right so you don't need
to worry about which match to attend
based on your comfort time you just
enroll for it and you can attend it if
you miss any bitch in any session you
can go to another batch holes for no
problem with that and there is a
interesting facility available from
simply learn which is a very unique
feature simply learn hands which is
called single parts single course pass
basically so if you take a pass if you
enroll for let's say one of the course
we did a Hadoop developer course you are
entitled to actually access any other
live in basic course from any other from
from simply learn only write and any
other l basic course which is running
you are entitled to access that freely
you don't really need to pay any money
for that it's like pay for one and get
access to everyone else write everything
else so that's the kind of you know
offer that is there and you can actually
get more details about it by contacting
simpler itself right they will they
simply learn coordinators will
help you get more clarity on that apart
from that there is an on-demand support
with dedicated diamond simulant
community there is something called
simply talk which is a very famous
community in simply learn lot of people
pose their queries with respect to any
issue that they are facing and there
will be experts who are there to help
them guide them so there is a very
active community which actually helps
clarify your doubts you also will get a
three-month project certification at the
end of the training they will be given a
project to work on and once you complete
it there will be a hands-on work
experience certificate which is given by
simply none which is a very valuable
asset for their big data carrier there
is a training provided by industry
experts and definitely which will help
we look and you know help you gain
knowledge in this field with with with
sharing you know their real-world
knowledge on that so certainly definite
it's the best platform you can get to
learn and definitely build a career in
Big Data space this is the certain is
the how the certification looks like for
big data and hadoop developer right so
simply learn is helping you in building
a big data carrier path it basically if
you enroll for one of the course you are
actually entitled to enroll for all of
this course as you can see here on the
screen right free of course cost so this
is actually you know single pass feature
single course pass which feature right
so basically you can get more details
about it from the simulator itself but
it is actually helping you to be a big
data architect hadoop architect right
well knowing all the technologies like
no sequel SPARC Impala Scala and and and
all the other stuff in advanced
analytics as well right so these are
some of the key and unique feature
simply none has so let us come back to
the demo as I just told you that there
is a cloud lab which is set up by the
simple N and as you can see here in my
screen I have log already logged into
the cloud lab so I'm sure that many of
you may not really get enough idea about
you know how to access this or you know
what I'm doing
here but if you don't really get much
idea don't worry at all we are just
trying to help you understand how to
play with Hadoop cluster so first thing
is you can just check here the Hadoop
version so you can see here it's the
latest heard of version which is
basically install in this cloud lab okay
apart from that you can check the source
code which is installed here on Hadoop
so I will basically will help you
understand where to go for this cluster
how to play with this Leicester how to
see the required property details of
this cluster right it's little early to
demonstrate all this but I will be
showing you one interesting concept
which is called as DFS and for that I
want you to understand a you know a high
level how does it as DFS looks like so
let me first explain you that ok let's
first understand what is a distributed
file system or RS DFS looks like so
let's say I have this cloud lab ok which
is connected over a network which means
you will have different machines
connected together right of course one
of them will be masters and others are
slave nodes right so as we discuss there
is a name node running on master and a
turn-on running on the slave machines ok
so this name nor and 'eternal are
nothing but they are Java programs which
are running on this machines ok and
these are the Java programs which are
part of Hadoop framework now let's
assume that you you you have so when you
install all these programs it actually
makes one virtual layer
okay this is the virtual layer which is
referred as HDFS Hadoop distributed file
system
and here in this virtual lair let's say
if you want to really type some you know
come on or if you want to let's say you
have a file of one terabyte size and I
want to store this file in this HDFS
then as a user what I will do I will use
a Hadoop command I'll just say Hadoop FS
- put I'll type source location and the
destination path okay now when you
execute this command okay when you type
this command which is basically simple
command to copy a file from a source
location to this HDFS behind the scenes
these processes which are running as
part of Hadoop cluster what they do is
they take your file distribute that file
into multiple blocks and it will be
storing this this data into let's say
these three machines each part of the
file will be physically divided and it
will be stored in the three different
machines but as a user I let's say I
would have given some path like this
under data directory I have stored this
file F not tht
so the user I think I have stored it
here but actually we had the since it is
being divided into multiple such parts
we call it as blocks so they are
basically stored in multiple blocks
right but we are not aware of it as a
user we think it is entire one terabyte
is stored in a single machine but this
is just a virtual path behind the scenes
it is stored in this three machine three
blocks now if I really want to access
this I will just use this virtual path
I'll just say how-do professor - cat cat
is a command to read the data of this
file once you do this it will basically
again go ahead and read all the contents
of this blocks and it presents you this
data again I am thinking as a user that
I am going to read this data one
terabyte data single location from
single machine but actually it is
basically going and reading all the
blocks from different machines and then
giving you the data so this is what we
mean by distributed file system right
the
is why it is referred as a virtual file
system so all the location that you
create in HDFS are virtual paths they
are just metadata information behind the
scenes Hadoop distribute and physically
divide your data in many machines in the
form of blocks okay so let me show you
the same demo here so that will
understand it better so you can access
as DFS using this photo by FS - LS route
when you type that you will be exceeding
different directories this is actually a
root directory of SDF s under which you
can see the art of directory is created
for user ok I can actually go under user
directory by the way every user all of
you when you are registered in training
you'll be given your own home directory
which will be under user directory so I
have logged in as this unique user so I
should be you know there will be a death
date recreated like this so I will go
inside that Hadoop FS - LS under user
I'll go inside that so I could see there
is a simple file which I have created
some time back that is why it's shown
that there is a file available now we
will do one more exercise let me create
one more file let's say I want to create
as DFS test file dot txt okay and I want
to write something here hi all welcome
to Purdue world ok now when I this file
I have created under a local file system
this is not as DFS ok but I want to copy
this file in SD FS so this is how I do
it don't worry about the command right
now but just understand the concepts so
let's say under you know a home
directory I want to copy this file so I
don't need to type this entire location
by default it will consider it as that
directly on me now if I do this Hadoop
FS - LS on this home directory of my
user I will basically see there is SDF
estees file also available here right so
it gets copied in HDFS but how do I see
how many blocks are there for this file
so you can actually see it like this
no preface - unless there is a fsck come
on again you know it's little early for
you to understand them but it's fine I
will show you some information about it
DFS taste for a lot tht so you just type
the path of the file type the files
block so it will show you how many
blocks are there this is the block ID so
each file is divided this file is
divided into one block because it has a
very small data it just created one
block and it is replicated three time I
just talked to you about replication
factor right for high availability of
the data so it is divided in three
replicas and they are available in these
three machines much in one machine too
and there is one more machine it should
be somewhere here right this is the one
so basically yeah here it is so it's
actually three replicas copies which is
created in three different machines
right and you can go and physically
check if you are in it if you have an
admin access you can go and check these
blocks as well which will have your data
right so this is how we can play with
Hadoop cluster and we can play with HDFS
again it's very early stage to
understand it at a very high level so I
if you don't really get it much it's
fine absolutely fine I will be teaching
you in a very very detailed manner but I
just want to I wanted to have you feel
of this cluster that's why I have shown
this demo ok so that's all I had let's
all work that's what I wanted to cover
so thank you so much
now I'm open for Question and Answer hi
Jai I mean it's a wonderful session and
my question is yeah my question is how
can we set up our own enrollment in our
in our own machine is the possible or is
only cloud lab as possible yes so
actually it is a right for the
simplicity sake simply line has actually
provided a cloud lab right but in case
if you have if you really want to set up
in your own environment we will help you
do that as well
I read 180 days access would be provided
my question is after 180 days what would
be our suppose you want to practice
something so how could it is very very
very practical questions but I think by
after 180 days you'll be an expert that
is my guarantee but anyway if you really
want it if you really want to you know
practice it it really makes a lot of
sense and that is why it's really
important that you should have a cluster
in in your machine see the reason why
simple inland has come up with this
cloud leverage one important reason is
we have been actually training like a
cross I mean VI in fact I myself have
trained about 2,000 people and I have
faced this many times people come up
with different configuration in their
machines then it becomes very difficult
to you know help them understand that it
the Hadoop will not be able to install
on that machine and they need to upgrade
that so sometimes it is not really
practically possible for them to do that
and that's where we had lot of
challenges so that's where simply learn
as a company decided that we don't
really want to trouble our participants
and let's make a cloud environment so
anybody can actually learn and access
this environment during the training and
even after that you have like lot of
days excess so that's why it is
available right but and I'd of course
it's a multi node cluster field right in
if you install it in a single machine it
will be a single node cluster but if you
install it in in a multiple node this is
like the real production type twister
that is available so it's it's like both
both options are there so it's you are
free to install and use both the both
the ways okay so we live sheltered
instructions also how to set up a yes
yes yes once you once you enroll the
training and if you happen to be in my
class I will definitely share on the
instructions okay so we have one more
question from you Manchu do we need to
have Java knowledge for learning hey
they're learning this yes I'm a new so
it's a like a very generic question that
a lot of people will ask well answer is
no and yes as well which is that is
I said this because it is it is it is a
very valuable advantage if I mean you
will get if you have a Jaguar knowledge
that doesn't mean that you know if you
don't have Java knowledge you cannot
attend her dog training no that's not
correct if you have any other
programming background even if it is
Python or C or C++ it's absolutely fine
you can go ahead and attend hello right
because in Hadoop there is in fact not a
barrier on the technology or in fact the
programming language
however since Hadoop is written in Java
if you have a prior working knowledge on
Java it will help you it is it says
evaluate because you can understand some
of the API is very easy that's the only
reason why I said that it's it's
evaluate otherwise there is no mandatory
requirement that you need to know Java
one of the other important thing if you
don't know Java simply learn when you
enroll for this course there is a free
course available self-paced learning
course on Java you can get more details
from simply learn so you can parallel
and Java as well you don't know text you
know not expect it to be an expert on
that but you can actually you know get
your doubts clarified using that
sessions okay great
he is asking does hadoop have a lot of
java programming no office it doesn't
have in fact heard okay a lot of other
ecosystem components like high peak and
always doesn't need java they are
developed for the people who don't want
to write java programs they are
developed for as a high level components
where you can write SQL queries you can
write some scripts to analyze the data
okay what you are also free to write lot
of Java programs using MapReduce okay
thank you we have one more question from
Guruji what is the level of knowledge
would I get at the end of this training
is it basic intermediate advanced level
of big data in a row it is right from
basic to advanced and now Ravi final
question how much programming knowledge
is required to learn her OOP that's what
were we so very basic if you are yeah
you know if you have done any
object-oriented programming before when
I say object-oriented programming you
know how to write classes how to create
objects how do you know
call functions or some basic programming
which I am assuming all of us mr. who
are certain engineers would know a bit
on that which is more than enough and
and you might be actually surprised but
this is definitely a reason why I'm
saying that it is more than enough if
you just need to know basic knowledge of
how to write simple programs more than
enough you don't need to be an expert
right you should be able to understand
the concept that's more than enough okay
one more question of visas asking
MongoDB or Cassandra combine or work
with HDFS or not yes so you can make
both of them talk to HDFS technically it
is possible okay what is the scope of
Hadoop admin very very very valuable
actually course and very valuable scope
as a carrier opportunity because
everybody just need to understand
logically that you know everybody is
trying to move towards big data they are
trying to use Hadoop in house all the
companies and penny when you have a
Hadoop cluster definitely you need
admins to manage that to monitor that to
actually troubleshoot it right
so in fact Hadoop admins are more
critical roles than Hadoop developers
you will find a lot of developers but
you don't find many admins that is the
actual my real experience which I am
talking about so definitely it's a
wonderful opportunity if you are
interested in admins ok so we have a
final question from amongst you do we
need to learn all these technologies to
be a Hadoop expert or from job X
perspective so you might show actually
from the job perspective as I said you
don't really guarantee in such a way
that you know you learn all this in you
get a job or you just learned one and
you will get a job there is always a
probability but what I am trying to
highlight a point is in a bit data
Hadoop developer course you will be
given a thorough knowledge on Hadoop
which is a core Hadoop which includes
HDFS MapReduce and yon if you just
understand that very very in very
detailed manner which will be shampoo
which will be definitely going to cover
which is more than enough to get a good
job in any hadoop establishments yes
however there are other component which
are also being taught in the training
like high paying edge base and all that
so if you learn that it
good if you have the knowledge it's good
but not all the companies use all of
them some companies you'll have some
companies may not use hype some families
you speak so it's all you know variable
so that's why it is important to make
your base strong in core Hadoop of
course ecosystem components are
important but depends on which company
you are going and joining you know they
made use it they may not use it so you
should not really be an expert on those
technologies you should either be jack
of all but you should have a fundamental
strong knowledge and corrado thank you
hope you have enjoyed today's session
and thank you everyone for your time
thank you and thanks for coordinating
we really appreciate so hope to see you
in the actual training have a good day
you