Big Data Framework | Big Data Hadoop Tutorial For Beginners | Big Data Hadoop Training | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
before the year 2000 data was relatively
smaller than it is currently however
data computation was complex
all data computation was dependent on
the processing power of the available
computers later as data grew the
solution was to have more computers with
large memory and fast processors however
after 2000 data kept growing and the
initial solution could no longer help
over the last few years there's been an
incredible explosion in the volume of
data
IBM reported that two point five
exabytes or 2.5 billion gigabytes of
data was generated every day in 2012
here are some statistics indicating the
proliferation of data from Forbes
September 2015
40,000 search queries are performed on
Google every second up to 300 hours of
video is uploaded to YouTube every
minute in Facebook 30 1.25 million
messages are sent by the users and 2.7 7
million videos are viewed every minute
by 2017 nearly 80% of photos will be
taken on smartphones by 2020 at least a
third of all data will pass through the
cloud a network of servers connected
over the Internet by the year 2020 about
1.7 megabytes of new information will be
created every second for every human
being on the planet data is growing
faster than ever before so is there a
solution to manage this ever-growing
data yes you can use more computers
instead of one machine performing the
job you can use multiple machines this
is called distributed system let's look
at an example to understand how a
distributed system works suppose you
have one machine which has 4 input
output channels the speed of each
channel is 100 megabytes a second and
you want to process one terabyte of data
on it how much time do you think it will
take to process the data it will take 45
minutes for one
to process one terabyte of data now
let's assume one terabyte of data is
processed by 100 machines with the same
configuration how much time do you think
it will now take to process the data it
will take only 45 seconds for 100
machines to process one terabyte of data
distributed systems take less time to
process Big Data now let's look at the
challenges of a distributed system since
multiple computers are used in the
distributed system there are high
chances of system failure there is also
a limit on the bandwidth programming
complexity is also high because it is
difficult to synchronize data and
process did you know the solution to
these challenges
Hadoop so what is Hadoop Hadoop is a
framework that allows for the
distributed processing of large data
sets across clusters of commodity
computers using simple programming
models it is inspired by a technical
document published by Google the word
Hadoop does not have any meeting Doug
cutting
who discovered Hadoop named it after his
son's yellow colored toy elephant let's
discuss how Hadoop resolves the three
challenges of the distributed system
such as high chances of system failure
limit on bandwidth and programming
complexity the four key characteristics
of Hadoop are economical reliable
scalable and flexible its systems are
highly economical as ordinary computers
can be used for data processing it's
reliable as it stores copies of the data
on different machines and is resistant
to hardware failure it's easily scalable
both horizontally and vertically a few
extra nodes help in scaling up the
framework it's flexible and you can
store as much structured and
unstructured data as you need to and
decide to use them later
traditionally data was stored in a
central location and it was sent to the
processor at runtime this method worked
well for limited data however modern
systems receive terabytes of data per
day and it is difficult for the
traditional computers or relational
database management system our DBMS
to push high volumes of data to the
processor Hadoop brought a radical
approach in Hadoop the program goes to
the data not vice-versa
it initially distributes the data to
multiple systems and later runs the
computation wherever the data is located
let's discuss the difference between
traditional RDBMS and Hadoop with the
help of an analogy you would have
noticed the difference in an eating
style of a human being and a tiger a
human eats food with the help of a spoon
where food is brought to the mouth
whereas a tiger brings its mouth towards
the food now if the food is data and the
mouth is program the eating style of a
human depicts traditional RDBMS and that
of a tiger depict Hadoop Hadoop has an
ecosystem which has evolved from its
three core components processing
resource management and storage in this
topic you will learn the components of
the Hadoop ecosystem and how they
perform their roles during big data
processing the Hadoop ecosystem is
continuously growing to meet the needs
of big data let's understand the role of
each component of the Hadoop ecosystem
it's comprised of the following 12
components Hadoop distributed file
system HBase scooped flume spark Hadoop
MapReduce Pig Impala hive cloud eras
search Susie Hugh you will learn about
the role of each component of the Hadoop
ecosystem in the next screens however
you will learn about yarn and its
architecture in the next lesson only
Hadoop distributed file system let's
understand the meaning and importance of
HDFS HDFS is a storage layer for Hadoop
suitable for distributed storage and
processing that is while the data is
being stored it first gets distributed
and then it is processed HDFS provides
streaming access to filesystem data file
permission and authentication HDFS uses
a command line interface to interact
with Hadoop
so what stores data in HDFS it is the
HBase which stores data in HDFS HBase is
a no sequel database or non relational
database HBase is important and mainly
used when you need random real-time read
or write access to your big data it
provides support to high volume of data
and high throughput in an age base a
table can have thousands of columns we
discussed how data is distributed and
stored now let's understand how this
data is ingested or transferred to HDFS
it is done by scoop scoop is a tool
designed to transfer data between Hadoop
and relational database servers it's
used to import data from relational
databases such as Oracle and my sequel
to HDFS and export data from HDFS to
relational databases if you want to
ingest event data such as streaming data
sensor data or log files then you can
use floom floom is a distributed service
that collects event data and transfers
it to HDFS it is ideally suited for
event data from multiple systems after
the data is transferred in the HDFS it
is processed one of the framework that
processes data is spark spark is an open
source cluster computing framework it
provides up to 100 times faster
performance for a few applications
within memory primitives as compared to
the two stage disk based MapReduce
paradigm of Hadoop SPARC can run in the
Hadoop cluster and processes data in
HDFS it also supports a wide variety of
workload which includes machine learning
business intelligence streaming and
batch processing SPARC has the following
major components as shown in the diagram
spark core and resilient distributed
data sets or RDD SPARC sequel SPARC
streaming machine learning library or M
Lib and graphics SPARC is now widely
used and you will learn more about it in
subsequent lessons
doop mapreduces the other framework that
processes data it is the original Hadoop
processing engine which is primarily
Java based it's based on the map and
reduced programming model many tools
such as hive and pig are built on
MapReduce model it has an extensive and
mature fault tolerance built into the
framework it is still very commonly used
but is losing ground to spark after the
data is processed it is analyzed it can
be done by an open source high level
data flow system called pig it's used
mainly for analytics pig converts it
scripts to map and reduce code thus
saving the user from writing complex
MapReduce programs ad-hoc queries like
filter and join which are difficult to
perform in MapReduce can be done easily
using pig you can also use Impala to
analyze data it is an open source high
performance sequel engine which runs on
Hadoop cluster it is ideal for
interactive analysis and has very low
latency which can be measured in
milliseconds Impala supports a dialect
of sequel so data in HDFS is modeled as
a database table you can also perform
data analysis using hive it is an
abstraction layer on top of Hadoop it's
very similar to Impala
however it's preferred for data
processing and extract transform load
also known as ETL operations Impala is
preferred for ad-hoc queries hive
execute queries using MapReduce however
a user need not write and encode in
low-level MapReduce hive is suitable for
structured data after the data is
analyzed it is ready for the users to
access what supports the search of data
it can be done using cloud areas search
search is one of cloud areas near
real-time access products it enables
non-technical users to search and
explore data stored in or ingest it into
Hadoop and HBase users do not need
sequel or programming skills to use
cloud areas search because it provides a
simple full-text interface for searching
another benefit of cloud air is search
compared to standalone search solutions
is a fully integrated data processing
platform cloud air is search uses the
flexible scalable and robust storage
system included with CD 8 or cloud eras
distribution including Hadoop this
eliminates the need to move large data
sets across infrastructures to address
business tasks Hadoop jobs such as
MapReduce Pig hive and scoop have
workflows Guzzi is a workflow or
coordination system that you can use to
manage the Hadoop jobs
Luzi application lifecycle is shown in
the diagram as you can see multiple
actions occur between the start and end
of the workflow another component in
Hadoop ecosystem is hue hue is an
acronym for Hadoop user experience it is
an open source web interface for Hadoop
you can perform the following operations
using hue upload and browse data query a
table in hive and Impala run spark and
pig jobs and workflows search data Hugh
makes a dupe easier to use it also
provides sequel editor for hive Impala
my sequel Oracle post gray sequel spark
sequel and solar sequel we will learn
more about hue in our future lessons
after a brief overview of the twelve
components of the Hadoop ecosystem we
will now discuss how these components
work together to process big data there
are four stages of big data processing
ingest processing analyze access the
first stage of big data processing is
ingest the data is ingested or
transferred to Hadoop from various
sources such as relational databases
systems or local files as discussed
earlier in this lesson you know that
scoop transfers data from our DBMS to
HDFS whereas flume transfers event data
the second stage is processing in this
stage the data is stored and processed
we discussed earlier that the data is
stored in the distributed file system
HDFS and the no sequel distributed data
HBase spark and MapReduce perform the
data processing the third stage is
analyzed here the data is analyzed by
processing frameworks such as pig hive
and Impala pig converts the data using
map and reduce and then analyzes it hive
is also based on map and reduced
programming and is most suitable for
structured data the fourth stage is
access which is performed by tools such
as hew and cloud areas search in this
stage the analyzed data can be accessed
by users Hugh is the web interface where
at cloud eras search provides a text
interface for exploring data Hey want to
become an expert in Big Data then
subscribe to the simply learned Channel
and click here to watch more such videos
centered up and get certified in Big
Data click here