Hadoop vs Spark | Which One to Choose? | Hadoop Training | Spark Training | Edureka Video Lecture | Apache Spark: Master Machine Learning - AI & ML

hello everyone this is Rashmi from ed
Eureka and today's video is on Hadoop
vs. spark now as we know organizations
from different domains are investing in
big data analytics today they're
analyzing large data sets to uncover all
hidden patterns unknown correlations
market trends customer preferences and
other useful business information these
analytical findings are helping
organizations in more effective
marketing new revenue opportunities and
better customer service and they're
trying to get competitive advantages
over rival organizations and other
business benefits an Apache spark and
Hadoop are the two of most prominent big
data frameworks and I see people often
comparing these two technologies and
that is what exactly we're going to do
in this video now we'll compare these
two big data frameworks based on
different parameters but first it is
important to get an overview about what
is Hadoop and what is Apache spark so
let me just tell you a little bit about
hadoo Hadoop is a framework to store and
process large sets of data across
computer clusters and Hadoop can scale
from single computer system up to
thousands of commodity systems that
offer a local storage and compute power
and Hadoop is composed of modules that
work together to create the entire
Hadoop framework these are some of the
components that we have in the entire
Hadoop framework or the Hadoop ecosystem
for example let me tell you about HDFS
which is the storage unit of Hadoop yarn
which is for resource management there
are different analytical tools like
Apache hive Pig no sequel databases like
Apache HBase even Apache spark in a
party Stone fits in the Hadoop ecosystem
for processing big data in real time for
ingesting data we have tools like flume
and scooped flume is used to ingest
unstructured data or semi structured
data whereas coop is used to ingest
structured data into HDFS if you want to
learn more about these tools you can go
to Eddy rei'kas youtube channel and look
for a Hadoop tutorial where everything
has been explained in detail now let's
move to spark up
a spark is a lightning-fast cluster
computing technology that is designed
for fast computation the main feature of
spark is its in-memory cluster computing
that increases the processing of speed
of an application far perform similar
operations to that of Hadoop modules but
it uses an in-memory processing and
optimizes the steps the primary
difference between MapReduce in Hadoop
and spark is that MapReduce uses
persistent storage and spark uses
resilient distributed data sets which is
known as rdd's which resides in memory
the different components in spark are
the spark core engine the spark core is
the base engine for large-scale parallel
and distributed data processing further
edition libraries which are built on top
of the core allowed diverse workloads
for streaming sequel and machine
learning spark core is also responsible
for memory management and fault recovery
scheduling and distributed and
monitoring jobs in a cluster and
interacting with the storage systems as
well next up we have sparked streaming
spark streaming is the component of
spark which is used to process real-time
streaming data it enables high
throughput and fault tolerant stream
processing of live data streams we have
sparked sequel spark sequel is a new
module in spark which integrates
relational processing with sparks
functional programming API it supports
querying data either via sequel or via
the hive query language for those of you
familiar with our DBMS spark sequel will
be an easy transition from your earlier
tools where you can extend the
boundaries of traditional relational
data processing next up is graphics
graphics is the spark API for graphs and
graph parallel computation and thus it
extends the spark resilient distributed
datasets with a resilient distributed
property graph
next is spark MLM for machine learning
Emma lip stands for machine learning
library spark Emily is used to perform
machine learning in Apache spark now
since you've got an overview
of both these two frameworks I believe
that the ground is all set
to compare apache spark and hadoop let's
move ahead and compare apache spark with
hadoop on different parameters to
understand their strengths we will be
comparing these two frameworks based on
these parameters let's start with
performance first spark is fast because
it has in-memory processing it can also
use disk for data that doesn't fit into
memory sparks in memory processing
delivers near real-time analytics and
this makes spark suitable for
credit-card processing system machine
learning security analytics and
processing data for IOT sensors now
let's talk about Hadoop's performance
now hadoop was originally designed to
continuously gather data from multiple
sources without worrying about the type
of data and storing it across
distributed environment and MapReduce
uses batch processing MapReduce was
never built for real-time processing
main idea behind yarn is parallel
processing over distributed dataset the
problem with comparing the two is that
they have different way of processing
and the idea behind the development is
also divergent
next ease of use SPARC comes with a
user-friendly API is for Scala Java
Python and spark sequel SPARC sequel is
very similar to sequel so it becomes
easier for sequel developers to learn it
SPARC also provides an interactive shell
for developers to query and perform
other actions and have immediate
feedback now let's talk about Hadoop you
can ingest data in Hadoop easily either
by using shell or integrating it with
multiple tools like scoop and flu and
yarn is just a processing framework that
can be integrated with multiple tools
like hive and fake for analytics hive is
a data warehousing component which
performs reading writing and managing
large data set in a distributed
environment using sequel like interface
to conclude here both of them have their
own ways to make themselves user
friendly now let's come to the cost
Hadoop and spark are both Apache
open-source projects so there's no cost
for the software cost is only associated
with the infrastructure both the
products are designed in such a way that
it can run on commodity hardware with
low TCO or total cost of ownership
well now you might be wondering the ways
in which they are different they're all
the same
storage and processing in Hadoop is disk
based and Hadoop uses standard amounts
of memory so with Hadoop we need a lot
of disk space as well as faster transfer
speed Hadoop also requires multiple
systems to distribute the disk
input/output but in case of Apache spark
due to its in-memory processing it
requires a lot of memory but it can deal
with a standard speed and amount of disk
as disk base is a relatively inexpensive
commodity and since spark does not use
disk input/output for processing instead
it requires large amounts of RAM for
executing everything in memory so spark
systems incurs more cost but yes one
important thing to keep in mind is that
sparks technology reduces the number of
required systems
it needs significantly fewer systems
that cost more so there will be a point
at which spark reduces the cost per unit
of the computation even with the
additional RAM requirement there are two
types of data processing batch
processing and stream processing batch
processing has been crucial to the big
data world in simplest term batch
processing is working with high data
volumes collected over a period in batch
processing data is first collected and
processed and then the results are
produced at a later stage and batch
processing is an efficient way of
processing large static datasets
generally we perform batch processing
for archived datasets for example
calculating average income of a country
or evaluating the change in e-commerce
in the last decade now stream processing
stream processing is the current trend
in the big data world need of the hour
is speed and real-time in
Meishan which is what stream processing
does batch processing does not allow
businesses to quickly react to changing
business needs in real-time stream
processing has seen a rapid growth in
that demand now coming back to Apache
spark versus Hadoop yarn is basically a
batch processing framework when we
submit a job to yarn it reads data from
the cluster performs operation and write
the results back to the cluster and then
it again reads the updated data performs
the next operation and write their
results back to the cluster and so on on
the other hand SPARC is designed to
cover a wide range of workloads such as
batch application iterative algorithms
interactive queries and streaming as
well now let's come to fault tolerance
Hadoop and spark both provides fault
tolerance but have different approaches
for HDFS and yarn both master daemons
that is the name node in HDFS and
resource manager in the arm checks the
heartbeat of the slave daemons the slave
daemons are data nodes and node managers
so if any slave daemon fails the master
daemons reschedule all pending and in
progress operations to another slave now
this method is effective but it can
significantly increase the completion
time for operations with single failure
also and as Hadoop uses commodity
Hardware another way in which HDFS
ensures fault tolerance is by
replicating data now let's talk about
SPARC as we discussed earlier our Dedes
or resilient distributed data sets our
building blocks of apache spark and
rdd's are the one which provide fault
tolerant to spark they can refer to any
data set present an external storage
system like HDFS HBase shared file
system etc they can also be operated
parallely rdd's can persist a data set
in memory across operations which makes
future actions 10 times much faster if
our DD is lost it will automatically get
recomputed by using the original
transformations
and this is how spark provides fault
tolerance and at the end let us talk
about security while Hadoop has multiple
ways of providing security Hadoop
supports Kerberos for authentication but
it is difficult to handle nevertheless
it also supports third-party vendors
like LDAP for authentication they also
offer encryption HDFS supports
traditional file permissions as well as
access control lists Hadoop provides
service level authorization which
guarantees that clients have the right
permissions for job submission SPARC
currently supports authentication via a
shared secret SPARC can integrate with
HDFS and it can use HDFS ACLs or access
control lists and file level permissions
Spahr can also run on Yarn leveraging
the capability of Kerberos now this was
the comparison of these two frameworks
based on these following parameters now
let us understand use cases where these
technologies fit best use cases where
Hadoop fits best for example when you're
analyzing archive data yarn allows
parallel processing over huge amounts of
data parts of data is processed
parallely and separately on different
data nodes and gathers result from each
node manager in cases when instant
results are not required now hadoop
mapreduce is a good and economical
solution for batch processing however it
is incapable of processing data in real
time use cases where SPARC fits best in
real time big data analysis real time
data analysis means processing data that
is getting generated by the real time
event streams coming in at the rate of
millions of events per second the
strength of SPARC lies in its abilities
to support streaming of data along with
distributed processing and SPARC claims
to process data hundred times faster
than MapReduce while ten times faster
with the disks it is used in graph
processing SPARC contains a graph
computation library called graphics
which
simplifies our life in memory
computation along with inbuilt graph
support improves the performance of
algorithm by a magnitude of one or two
degrees over traditional MapReduce
programs it is also used in iterative
machine learning algorithms almost all
machine learning algorithms work
iteratively as we have seen earlier
iterative algorithms involve
input-output bottlenecks in the
MapReduce implementations MapReduce uses
coarse grain tasks that are too heavy
for iterative algorithms spark caches
the intermediate data set after each
iteration and runs multiple iterations
on the cache data set which eventually
reduces the input/output overhead and
execute the algorithm faster in a
fault-tolerant manner so at the end
which one is the best the answer to this
is Hadoop an Apache spark are not
competing with one another in fact they
complement each other quite well
Hadoop brings huge datasets under
control by commodity systems and spark
provides a real-time in-memory
processing for dos datasets
when we combine Apache sparks ability
that is the high processing speed and
advanced analytics and multiple
integration support with Hadoop slow
cost operation on commodity hardware it
gives the best results Hadoop
complements Apache spark capabilities
spark cannot completely replace a do but
the good news is that the demand of
spark is currently at an all-time high
if you want to learn more about the
Hadoop ecosystem tools an Apache spark
don't forget to take a look at the Eddie
rei'kas youtube channel and check out
the big data and hadoop playlist thank
you for watching this video happy
learning I hope you have enjoyed
listening to this video please be kind
enough to like it and you can comment
any of your doubts and queries and we
will reply them at the earliest do look
out for more videos in our playlist and
subscribe to any Rica channel to learn
more happy learning