Big Data Tutorial For Beginners - Lesson 1 | Big Data Introduction | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

hello and welcome to Big Data Hadoop and
spark developer course offered by simply
learn this lesson is an introduction to
Big Data and Hadoop ecosystem now what
you will learn in this session after
completing this lesson you should be
able to first understand the concepts of
Big Data and its challenges explain
Hadoop and how it addresses those
challenges and finally you should be
able to describe the Hadoop ecosystem in
the first topic of this lesson you will
get an overview of Big Data and Hadoop
so first let's look at the evolution of
Big Data before the year 2000 the data
volumes and our databases tended to be
relatively slow but relatively small
compared to what they are currently
however the processing was still fairly
complex and all the computations were
dependent on a single server or a small
cluster of database servers and as the
volume of the data grew we needed to
upgrade the server add more processors
or add more memory however after the
year 2000 we saw the data explosion and
our single server hardware was no longer
keeping up with the growth of the data
now let's look at the data explosion
that happened in the recent decade and a
half
back in 2012 IBM reported that two and a
half exabytes of data that's two and a
half billion gigabytes was generated
everyday and that was in the year 2012
now looking at more reasons to
sticks that's reported from reported by
Forbes last year 40,000 search queries
were performed by Google every second up
to 300 hours of video are being uploaded
to YouTube every minute in facebook 31
of the quarter million messages are sent
by the users and two and three-quarter
million videos are viewed every minute a
year ago they estimated that by now 80%
of all photos will be taken by
smartphones and by year 2020 at least a
third of all data will pass through
cloud and about 1.7 megabytes of new
data will be created every second for
every human being on the planet so
that's the scope of the data explosion
that we saw in the past decade and a
half so what's the solution for managing
this enormous volume of data and the
solution is with distributed systems so
instead of a single large server we can
have lots of lots of machines connected
together in the cluster and sharing the
workload and that's what is called the
distributed system now let's look how
distributed system works let's consider
an example suppose we have a single
machine that has for i/o channels and
the speed of each channels is 100 MB per
second so if I want to process one
terabyte of data that will take us 45
minutes now if we have a hundred machine
cluster connected together it will take
only 45 seconds to solve the same
problem so that's how distributed
systems are help us process big data
so next let's look at the challenges
faced by big data systems one is the
high chances of system failure if we
have a lots and lots of machines
connected together in the cluster if one
machine fails that can bring down the
whole cluster second is the limit on the
bandwidth as all those machines need to
be connected together on a network and
finally high complexity of programming
because now what we not only have to
worry about solving the problem at hand
but also have to manage all those
machines on the cluster that have to be
synchronized and work together now
Hadoop is the platform that was designed
specifically to address those challenges
what is Hadoop now Hadoop the official
definition Hadoop is a framework that
allows for distributed processing of
large data sets across clusters of
commodity computers using simple
programming models originally the
platform was inspired by a technical
document published by Google years ago
the word Hadoop by itself does not
really have any meaning doc cutting one
of the original developers of Hadoop
named the system after his sons to a
yellow elephant so that's how the name
came about let's look at the key
characteristics of Hadoop platform there
are four of them Hadoop is reliable
scalable flexible and economical now it
is reliable because every piece of data
in Hadoop clustered is redundant and is
copied across multiple machines on the
cluster so if one machine
down the data will still be available in
the cluster will continue functioning it
is economical because Hadoop runs on the
cluster of commodity hardware machines
there is no specialized hardware
involved it's all cheap cheap machines
the reason Hadoop is scalable is because
in order to scale it both horizontally
and vertically all you need to do is
just plug in additional nodes into the
cluster and it is flexible because it
can store virtually unlimited volumes of
data and it can handle both structured
and unstructured data so those are the
key kudzu characteristics next let's
compare Hadoop to traditional or DBMS
platforms let's let's consider an
analogy let's compare dining habits of
humans and tigers so as humans in order
to consume our food we bring it into our
home we cook it we serve it and then use
utensils to bring food to the mouth now
Tiger on the other hand tiger brings his
mouth to the food so if we look at
traditional database platforms versus
Hadoop in traditional platforms in order
for data to be processed it has to be
retrieved from the disk from storage and
then pulled into the centralized
processing agent and that single
processing module becomes a bottleneck
while with Hadoop what happens is that
the data processing program itself is
sent to wherever data is located so
that's the revolutionary idea behind the
Hadoop platform so next let's take a
look at
several youth cases of Hadoop and see
what similarities and what common trends
we can find and first use case is a bank
using risk modeling so originally a bank
had multiple data warehouses one for
each department and each one performed
its own risk analysis now after bringing
in Hadoop the company was able to
combine all their data on a single
platform and run more reliable risk
analytics and that amount produced
better models another example is a
recommendation agent for online dating
now the challenge here is if we look at
database problems we solve in a
traditional database world we would
usually have a a single item compared to
a large data set across one or maybe a
handful of parameters here if you think
about online dating recommendation agent
you basically have to compare everyone
to everyone across lots of different
parameters so that is a big processing
challenge another example threat
analysis so let's say a company wants to
monitor for a criminal behavior like a
fraud and abuse so the challenge here is
that a single event by itself may not be
meaningful and indicative for for
anything in particular
it is when we identify a trend when we
identify multiple similar events or
dependent events then they become
another example let's say a company
wants to measure public perception of
their organization and for that purpose
they monitor multiple social media
platforms and continually monitor each
of those for opinions being expressed
online and ranks it as positive negative
or neutral so one of the challenges here
is streaming data and streaming
simultaneously from multiple sources
another example is a search quality
something like a search engine here you
have to you have to examine and rank an
enormous volume of data lots and lots of
materials published online and which we
need to analyze and rank in order to
optimize user search um so that I can
also add an example from my own recent
practice I am building a product for a
healthcare client which would allow them
to measure and compare the quality of
health care
so here the challenge is that the data
for millions of patients need to be
combined from multiple sources and the
statistical analyst in the complex
statistical algorithms need to be
applied in order to calculate the common
quality measures and compare them to
each other so next let's briefly look at
the core components of Hadoop later
we'll examine each of them in more
detail first one is a Hadoop HDFS which
which stands for Hadoop distributed file
system so that's the story
component of Hadoop that's where your
data is stored next is yarn yarn Stan's
happens to stand for yet another
resource navigator and that's basically
Hadoop's resource manager that's the
component that allows you to coordinate
jobs and tasks that are spread among the
many machines in the cluster and finally
the processing components the which are
hadoop mapreduce and spark where
MapReduce is the original processing
engine and spark is becoming more
popular in recent years we'll cover each
of this components in more detail in
later in the class so next topic on
introduction to Big Data and of
introduction to Big Data and Hadoop
lesson is Hadoop ecosystem a dupe has an
ecosystem that has evolved from the
three core components that we've just
discussed processing resource management
and storage so on this topic we will
learn about each of those components and
how they work together as a single equal
system so looking at Hadoop's core
components HDFS which we already covered
distributed file system now the data
that resides on Hadoop can can reside
either in raw files on HDFS we can be
storing in one of several no sequel
databases one of them is HBase and we'll
cover it later in more detail
yarn will really briefly covered which
is the
resource manager and the data processing
agents MapReduce and spark now before
the data gets into Hadoop before it's
stored on the distributed file system
data needs to be ingested from somewhere
so we'll cover two of the data ingestion
options available Hadoop which is group
and flu next we'll explore the tools
that are available to us for analytics
and those are Pig hive and Impala and
we'll also cover data exploration and
end-user tools like you and the cloud
era surge and finally we'll briefly talk
about Guzzi which is a workflow manager
that helps us manage complex workflows
so next let's let's discuss in some
detail HDFS distributed file system
storage layer of hadoop now it's
suitable for distributed storage and
processing and data can be either stored
there raw or in in no sequel database
platforms now before the data is written
to the file system it gets automatically
distributed so any piece of data that is
ingested into Hadoop gets distributed
across all available not nodes in the
cluster and that process happens
automatically but to you as a user this
processing is entirely transparent and
you communicate with the HDFS vieira
command into
face it looks just like any regular file
system HDFS provides streaming access to
file system data and it supports the
authentication and file permissions just
like any other file system boot so how
is data stored in HDFS one option is to
store data in HBase which is a no sequel
database in other words it's a non
relational database we're now in no
sequel stands for not only sequel HBase
is used to provide random real-time read
and write access to big data and it
provides support for high volume and
high throughput being a no sequel
database a single age-based table may
have thousands of columns it is more of
a logical structure which is not
directly related to how the data is
actually stored on baby
so a data ingestion tools one of the
popular data ingestion tools is scoop
and scoop is designed to transfer data
between Hadoop and relational databases
like Oracle sequel server my sequel and
so forth and scoop works is
bi-directional you can use it either to
ingest data into Hadoop or to export
data from Hadoop to relational databases
flume is another option for data
ingestion and the it works very similar
to scoop main advantage your flume is
that is suitable for streaming data so
that's what it's ideal for let's talk
about spark so spark is a cluster based
computing frameworks and it is extremely
fast it can provide up to hundred times
faster performance than MapReduce based
framework and what makes park so
exceptionally fast is that it is works
exclusively in memory there is the there
is no disk storage in spark so there is
no retrieval from disk step well you use
spark for analysis spark support
supports machine learning business
intelligence streaming and batch
processing the components of spark quad
core and resilient distributed data sets
or our DG's spark sequel spark does
support sequel spark streaming machine
learning and graphics in addition to
sequel by the way Scala is a very
popular option with spark that's what we
are by the way using in our current
project it is a similar similar
framework to sequel but it is a lot more
flexible MapReduce so MapReduce is
another option for a processing
framework and this is actually the
original framework that came with Hadoop
and it is and it is in Java based
framework and it works as a map and
reduce paradigm those are basically Java
Java based
fairly sophisticated programs many
Hadoop tools like hive and pig are built
on top of metric use so they basically
provide a simpler interface internet
reduce the low power goes to slating in
the course so now that we covered the
processing component I will let's talk
about components that can be used for
data analysis one of them is Pig and egg
is a scripting interface on top of
MapReduce now the advantage of pig is
that it's much simpler MapReduce
programs can be a quite voluminous can
can include hundreds of lines of code
well pig allows you to perform your
tasks with a fairly simple scripting
interface and simple commands like
filter and join can take a lot of code
with Java based MapReduce but just a few
lines with pig now impala another option
we can use for analytics impala is a
sequel based framework it is uses a
dialect of sequel and the main advantage
of it is that a sequel is familiar to
most database developers the with Impala
you have very very low latency it can
support multiple users and it's and it
provides a high performance sequel
another option is high
so just like Impala hi if this also a
sequel based framework it's not as fast
as in power but it does provide better
support for sequel it supports a lot
more sequel operations than Impala does
now it is similar to Pig in a way that
it is also an interface on top of
MapReduce so whenever you create a
sequel statement in hive when you submit
it it gets translated into a MapReduce
job and then executes it so it is a
two-step process which makes it slower
than Impala but it is more flexible and
hive is typically used for batch
processing like ETL jobs so next we'll
talk about data access tools one of them
is cloud era search it comes packaged
with cloud air distribution of a dupe
and it works against through HDFS files
or you can use it against the data
stored in HBase cloud era search enables
non-technical users to search data
available in Hadoop so it requires no
programming skills it's very simple
intuitive interface next
ponent will discuss is boozy boozy is a
workflow manager and coordination system
that is used to manage group jobs when
you have complex tasks that need to be
built into workflows when you need to
manage dependencies
that's what woozi is used for it
it creates workflow so that the tasks
are well managed and your your processes
do not step on each other's toes so to
speak so that's what use is used for and
finally Hugh now Hugh is a stands for
Hadoop user interface and one way to
describe it it's really a portal into
the food world
it's a web-based interface and through
it you can access all tools and all
resources available in Hadoop you can
use it to execute queries in hive or an
Impala you can run
pegye jobs or spark processing jobs you
can search data so it's your basically
your single window into the Hadoop world
where you can access all of the
available resources
so to cover what we just learned the
four stages of data of data processing
in Hadoop it's data ingestion processing
analysis and access so for ingestion
with covered scoop and flu for data
processing we talked about HDFS Hadoop
distributed file system we've covered
HBase one of the no sequel databases
available in Hadoop we've talked about
MapReduce and spark we have MapReduce is
the original processing engine and spark
and much faster memory based processing
framework that's becoming more and more
popular in the recent years we discussed
several options of analytical tools that
can be used in Hadoop to remind you hive
and pig are the interfaces that sit on
top of MapReduce where Pig provides a
scripting interface and hive provides
sequel interface we also covered in
pollen which is another secret another
way to access a Hadoop be a sequel the
main difference with Impala is that
instead of translating sequel into
MapReduce jobs Impala goes directly to
the data and that makes it work much
faster
although hive is more flexible as it
supports more sequel features and the
hive would typically be used for ETL
jobs while you can use Impala for a
real-time data access and finally we've
also discussed and user access
tools like cloudy research which allows
you to search data available in Hadoop
and you the portal into Hadoop that
allows you to access any tool and any
resource available into so let's let's
look at the key takeaways what you
should be taking out of this class let's
go over the key points first Hadoop is a
framework for distributed storage and
processing core components of Hadoop
include HDFS for storage and yarn for
cluster resource management and
MapReduce or spark for processing a dupe
ecosystem includes multiple components
which support each stage of the big data
processing humans group to ingest data
HDFS and age-based to store data SPARC
and MapReduce to process data pic hive
and Impala for analysis human search you
can use to explore the data and easy to
manage the workflow of your Hadoop jobs
so that concludes our lesson
you