Spark Tutorial For Beginners | Big Data Spark Tutorial | Apache Spark Tutorial | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
spark as a data processing framework was
developed at UC Berkeley's pamphlet by
Mattea haria in 2009 in 2010 it became
an open source project under a berkeley
software distribution license in the
year 2013 the project was donated to the
Apache Software Foundation and the
license was changed to Apache 2.0 in
February 2014 spark became an Apache
top-level project by November 2014 spark
was used by the engineering team at data
bricks a company founded by the creators
of Apache spark to set a world record in
large-scale sorting now data bricks
provides commercial support and
certification for taking the spark
programming test at present spark exists
as a next-generation real-time and batch
processing framework let's try to
understand what batch and real-time
processing mean we will use this
information in the subsequent slides to
compare spark with MapReduce both of
which are data processing framework in
Hadoop in case of batch processing a
large amount of data or transactions are
processed in a single run over a time
period the Associated jobs generally run
entirely without any manual intervention
additionally the entire data is
pre-selected and fed using command-line
parameters and scripts in typical cases
batch processing is used to execute
multiple operations handle heavy data
load generate reports and manage data
workflow which is offline an example is
to create daily or hourly reports to aid
decision making on the other hand
real-time processing occurs
instantaneously on data entry or command
receipt it needs to execute within
stringent response time constraints an
example is fraud detection the need for
spark was created by the limitations of
MapReduce which is another data
processing framework in Hadoop let's see
what these limitations are MapReduce
is suitable for batch processing where
data is processed as a periodic job thus
it takes time to process data and
provide results if the data is high
depending on the amount of data and the
number of nodes in the cluster to
complete a job it just takes minutes to
process the data however it is not a
good choice for real-time processing
MapReduce is also not suitable for
writing trivial operations such as
filter and join to write such operations
you might need to rewrite the jobs using
the MapReduce framework which becomes
complex because of the key/value pattern
this pattern is required to be followed
in reducer and mapper codes MapReduce
doesn't work so well with large data on
the network the reason is that it takes
a lot of time to copy the data which may
cause network bandwidth issues it works
on the data locality principle and hence
works well on the node where the data
resides MapReduce is also unsuitable for
online transaction processing or OLTP
which includes a large number of short
transactions since it works on the batch
oriented framework it lacks latency of
seconds or sub seconds additionally
MapReduce is unfit for processing graphs
graphs represent the structures to
explore relationships between various
points for example finding common
friends in social media sites like
Facebook Hadoop has the Apache
giraffe library for such cases it runs
on top of MapReduce and adds to the
complexity another important limitation
is its unsuitability for iterative
program execution some use cases like
k-means need such execution where data
needs to be processed again and again to
refine results MapReduce runs from the
start every time as it is stateless
executor SPARC is an open source cluster
computing framework which addresses all
of the limitations of MapReduce it is
suitable for real-time processing
trivial operations and processing larger
data on a network it is also suitable
for OLTP graphs and iterative execution
compared to the disk-based two-stage
MapReduce of Hadoop spark provides up to
100 times faster performance for a few
applications within memory primitives
fast performance makes it suitable for
machine learning algorithms as it allows
programs to load data into the memory of
a cluster and create a data constantly a
spark project comprises various
components such as spark core and
resilient distributed data sets or rdd's
spark sequel spark streaming machine
learning library or ml lib and graphics
let's discuss the components of spark
the first component spark core and rdd's
are the foundation of the entire spark
project they provide basic input/output
functionalities distributed tasks
dispatching and scheduling let's look at
RDD closely our DDS are the basic
programming abstraction and is a
collection of data that is partitioned
across machines logically rdd's can be
created by applying coarse-grained
transformations on the existing rdd's or
by referencing external datasets the
examples of these transformations are
reduce join filter and map the
abstraction of rdd's is exposed
similarly as in process and local
collections through a language
integrated application programming
interface or API in Python Java and
Scala as a result of the RDD abstraction
the complexity of programming is
simplified as the manner in which
applications change rdds is similar to
changing local data collections the
second component is spark sequel which
resides on the top of spark core it
introduces schema RDD which is a new
data abstraction and supports semi
structured and structured data schema
RDD can be manipulated in any of the
provided domain-specific languages such
as java scala and python by the spark
sequel spark
will also support sequel with open
database connectivity or Java database
connectivity commonly known as ODBC or
JDBC server and command-line interfaces
the third component is spark streaming
spark streaming leverages the fast
scheduling capability of spark core for
streaming analytics ingesting data in
small batches and performing RDD
transformations on them with this design
the same application code set written
for batch analytics can be used on a
single engine for streaming analytics
the fourth component of spark is machine
learning library also known as ML Lib it
lies on top of spark and is a
distributed machine learning framework
ML Lib applies various common
statistical and machine learning
algorithms with its memory based
architecture it is nine times faster
than the Apache mahute Hadoop disk based
version in addition the library performs
even better than Val Powell wabbit or VW
the VW project is a fast out of core
learning system sponsored by Microsoft
the last component graphics also lies on
the top of spark and is a distributed
graph processing framework for the
computation of graphs it provides an API
and an optimized runtime for the prey
ghul abstraction fraggle is a system for
large-scale graph processing the API can
also model the prego abstraction we
discussed earlier that spark provides up
to 100 times faster performance for a
few applications within memory
primitives let's discuss the application
of in-memory processing using column
centric databases in column centric
databases similar information can be
stored together and hence data can be
stored with more compression and
efficiency it also permits the storage
of large amounts of data in the same
space thereby reducing the amount of
memory required for performing a query
it also increases the speed of
processing in an in
memory database the entire information
is loaded into memory eliminating the
need for indices aggregates optimized
databases star schemas and cubes with
the use of in-memory tools
compression algorithms can be
implemented that decrease the in-memory
size even beyond what is required for
hard disks users query allowed in memory
is different from caching in memory
processing also helps to avoid
performance bottlenecks and slow
database access caching is a popular
method for speeding up their performance
of a query where caches are subsets of a
very particular organized data which are
already defined within memory tools data
analysis can be flexible in size and can
be accessed within seconds by concurrent
users with an excellent analytics
potential this is possible as data lies
completely in memory in theoretical
terms this leads to data access
improvement that is 10,000 to 1 million
times fast when compared to a disk in
addition it reduces the performance
tuning needed by IT professionals and
therefore provides faster data access
for end-users with in memory processing
it is possible to access visually rich
dashboards and existing data sources
this ability is provided by several
vendors in turn in memory processing
allows end users and business analytics
to create customized queries and reports
without any need of extensive expertise
or training we have already discussed
that SPARC provides performance which in
turn offers developers a rewarding
experience SPARC is chosen over
MapReduce mainly for its performance
advantages and versatility
apart from these another critical
advantage is its development experience
along with language flexibility SPARC
provides support to various development
languages like Java Scala
and Python and will likely support R as
well in addition SPARC has the
capability to define functions in line
with the temporary exception of Java a
common element in these languages is
that they provide methods to express
operations you
lambda functions and closures using
lambda closures you can use the
application CoreLogic to define the
functions inline which helps to create
easy to comprehend codes and preserve
application flow let's look at MapReduce
in the Hadoop ecosystem the Hadoop
ecosystem which allows you to store
large files on various machines uses
MapReduce for batch analytics that is as
easy as it is distributed in nature on
the other hand Apache spark supports
both real-time and batch processing in
Hadoop third-party support is also
available for example by using ETL
talented tools various batch oriented
workflows can be designed in addition it
supports pig and hive queries that
enable non Java developers to use and
prepare batch workflows using sequel
scripts you can perform every type of
data processing using spark that you can
execute in Hadoop for batch processing
spark batch can be used over hadoop
mapreduce
for structured data analysis spark
sequel can be implemented using sequel
for machine learning analysis the
machine learning library can be used for
clustering recommendation and
classification for interactive sequel
analysis spark sequel can be used
instead of Impala in addition for
real-time streaming data analysis spark
streaming can be used in place of a
specialized library like storm spark has
three main advantages which are provide
speed capability combines of various
processing types and supports Hadoop the
feature of speed is critical to process
large data sets as this implies the
difference of waiting for hours or
minutes and exploring the data
interactively spark has extended the
MapReduce model to support computations
like stream processing and interactive
queries supporting run computations in
memory with respect to speed also its
related system is more effective when
compared to MapReduce to run complex
applications on a disk this adds to the
speed capability of spark spark covers
various workloads that require different
district
systems such as streaming iterative
algorithms and batch applications as
these workloads are supported on the
same engine combining different
processing types is easy SPARC is
normally required in the production of
data analysis pipelines the combination
feature also allows easy management of
separate tools SPARC is capable of
creating distributed datasets from any
file that is stored in the Hadoop
distributed file system or any other
supported storage systems you must note
that SPARC does not need to do it just
supports the storage systems that
implement the api's of Hadoop and
sequence files Parque Avro text files
and all other input output formats of
Hadoop now the question is why
unification matters unification not only
provides developers with the advantage
of learning only one platform but also
allows users to take their apps
everywhere
the graphic shows the apps and the
systems that can be combined with spark
a spark project includes various closely
integrated components for distributing
scheduling and monitoring applications
with many computational tasks across a
computing cluster or various worker
machines the spark core engine is
general-purpose and fast as a result it
empowers various higher-level components
that are specialized for different
workloads like machine learning or
sequel these components can interoperate
closely another important advantage is
that it integrates tightly allowing you
to create applications that easily
combine different processing models an
example is the ability to write an
application using machine learning to
categorize data in real-time as it is
ingested from sources of streaming
additionally it allows analysts to query
the data which results through sequel
moreover data scientists and engineers
can access the same data through the
Python shell for ad hoc analysis and in
standalone batch applications for all
this the IT team needs to maintain one
system only
Hey
want to become an expert in Big Data
then subscribe to the simply learned
Channel and click here to watch more
such videos centered up and get
certified in Big Data click here