Hadoop Ecosystem Tutorial | Hadoop Ecosystem Components Overview | Hadoop Tutorial | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
the Hadoop ecosystem is continuously
growing to meet the needs of big data
let's understand the role of each
component of the Hadoop ecosystem it's
comprised of the following 12 components
Hadoop distributed file system HBase
scooped flume spark Hadoop MapReduce Pig
Impala hive cloud eras search Susie Hugh
you will learn about the role of each
component of the Hadoop ecosystem in the
next screens however you will learn
about yarn and its architecture in the
next lesson only Hadoop distributed file
system let's understand the meaning and
importance of HDFS HDFS is a storage
layer for Hadoop suitable for
distributed storage and processing that
is while the data is being stored it
first gets distributed and then it is
processed HDFS provides streaming access
to filesystem data file permission and
authentication HDFS uses a command-line
interface to interact with Hadoop so
what stores data in HDFS it is the HBase
which stores data in HDFS HBase is a no
sequel database or non relational
database HBase is important and mainly
used when you need random real-time read
or write access to your big data it
provides support to high volume of data
and high throughput in an H base a table
can have thousands of columns we
discussed how data is distributed and
stored now let's understand how this
data is ingested or transferred to HDFS
it is done by scoop scoop is a tool
designed to transfer data between Hadoop
and relational database servers it's
used to import data from relational
databases such as Oracle and my sequel
to HDFS and export data
from HDFS to relational databases if you
want to ingest event data such as
streaming data sensor data or log files
then you can use floom floom is a
distributed service that collects event
data and transfers it to HDFS it is
ideally suited for event data from
multiple systems after the data is
transferred in the HDFS it is processed
one of the framework that processes data
is spark spark is an open source cluster
computing framework it provides up to
100 times faster performance for a few
applications within memory primitives as
compared to the two stage disk based
MapReduce paradigm of Hadoop spark can
run in the Hadoop cluster and processes
data in HDFS it also supports a wide
variety of workload which includes
machine learning business intelligence
streaming and batch processing SPARC has
the following major components as shown
in the diagram spark core and resilient
distributed data sets or RDD spark
sequel spark streaming machine learning
library or EMM lib and graphics spark is
now widely used and you will learn more
about it in subsequent lessons Hadoop
MapReduce is the other framework that
processes data it is the original Hadoop
processing engine which is primarily
Java based it's based on the map and
reduced programming model many tools
such as hive and pig are built on
MapReduce model it has an extensive and
mature fault tolerance built into the
framework it is still very commonly used
but is losing ground to spark after the
data is processed it is analyzed it can
be done by an open source high level
data flow system called pig it's used
mainly for analytics pig converts it
scripts to map and reduce code thus
saving the user from writing complex
MapReduce programs ad-hoc queries like
filter and join which are difficult to
perform in MapReduce can be done easily
using Pig
also use Impala to analyze data it is an
open-source high-performance sequel
engine which runs on the dupe cluster it
is ideal for interactive analysis and
has very low latency which can be
measured in milliseconds Impala supports
a dialect of sequel so data in HDFS is
modeled as a database table you can also
perform data analysis using hive it is
an abstraction layer on top of Hadoop
it's very similar to Impala
however it's preferred for data
processing and extract transform load
also known as ETL operations Impala is
preferred for ad-hoc queries hive
execute queries using MapReduce however
a user need not write any code in
low-level MapReduce hive is suitable for
structured data after the data is
analyzed it is ready for the users to
access what supports the search of data
it can be done using clutter is search
search is one of cloud eras near
real-time access products it enables
non-technical users to search and
explore data stored in or ingest it into
Hadoop and HBase users do not need
sequel or programming skills to use
cloud areas search because it provides a
simple full-text interface for searching
another benefit of cloud era is search
compared to standalone search solutions
is a fully integrated data processing
platform cloud eras search uses the
flexible scalable and robust storage
system included with CD 8 or cloud eras
distribution including Hadoop this
eliminates the need to move large data
sets across infrastructures to address
business tasks Hadoop jobs such as
MapReduce Pig hive and scoop have
workflows Guzzi is a workflow or
coordination system that you can use to
manage the Hadoop jobs
Luzi application lifecycle is shown in
the diagram as you can see multiple
actions occur between the start and end
of the workflow
another component in Hadoop ecosystem is
hue hue is an acronym for Hadoop user
experience it is an open source web
interface for Hadoop you can perform the
following operations using hue upload
and browse data query a table in hive
and Impala run spark and pig jobs and
workflows search data Hugh makes a dupe
easier to use it also provides sequel
editor for hive Impala my sequel Oracle
post gray sequel spark sequel and solar
sequel we will learn more about you in
our future lessons after a brief
overview of the twelve components of the
Hadoop ecosystem we will now discuss how
these components work together to
process Big Data
there are four stages of big data
processing ingest processing analyze
access the first stage of big data
processing is ingest the data is
ingested or transferred to Hadoop from
various sources such as relational
databases systems or local files as
discussed earlier in this lesson you
know that scoop transfers data from our
DBMS to HDFS whereas flume transfers
event data the second stage is
processing in this stage the data is
stored and processed we discussed
earlier that the data is stored in the
distributed file system HDFS and the no
sequel distributed data HBase spark and
MapReduce perform the data processing
the third stage is analyzed here the
data is analyzed by processing
frameworks such as pig hive and Impala
pig converts the data using map and
reduce and then analyzes it hive is also
based on map and reduced programming and
is most suitable for structured data the
fourth stage is access which is
performed by tools such as Hugh and
Cloud areas search in this stage the
analyzed data can be accessed by users
Hugh is the web interface where as cloud
areas search provides a text interface
for exploring data
Hey want to become an expert in Big Data
then subscribe to the simply learned
Channel and click here to watch more
such videos centered up and get
certified in Big Data click here