What Is Apache Spark | Apache Spark Tutorial For Beginners | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
Apache spark is an open-source cluster
computing framework that was initially
developed at UC Berkeley in the ampulla
as compared to the disk-based
two-stage MapReduce of Hadoop spark
provides up to 100 times faster
performance for a few applications
within memory primitives this makes it
suitable for machine learning algorithms
as it allows programs to load data into
the memory of a cluster and query the
data constantly as shown on the screen a
spark project contains various
components such as spark core and
resilient distributed data sets or rdd's
spark sequel spark streaming machine
learning library or ml library and
graphics
spark was started at uc-berkeley amp lab
by Mattea haria in the year 2009 it was
in 2010 when it was open sourced under a
BSD license the project was then donated
to the Apache Software Foundation and
the license was changed to Apache 2.0 in
the year 2013 in the month of February
2014 spark became an Apache top-level
project then in November of the same
year it was used by the engineering team
at data bricks to set a world record and
large-scale sorting now data bricks
provides commercial support and they
provide certification for it
at present spark exists as a
next-generation real-time and batch
processing framework the graphic on the
screen also displays the journey of
spark
the components of a spark project are
listed on the screen the first component
is spark core and rdd's which is the
foundation of the entire project it
provides basic input/output
functionalities distributed tasks
dispatching and scheduling rdd's is the
basic programming abstraction and is a
collection of data that is partitioned
across machines logically these can be
created by applying coarse-grained
transformations on the existing rdd's or
by referencing external datasets the
examples of these transformations are
reduce join filter and map the
abstraction of rdd's is exposed
similarly as in process and local
collections through a language
integrated api in Python Java and Scala
as a result the complexity of
programming is simplified as the manner
in which applications change rdd's is
similar to changing local data
collections spark sequel is a component
lying on the top of spark core an
introduces schema RTD which is a new
data abstraction and supports semi
structured and structured data this
abstraction can be manipulated in Java
scalar and Python by the spark sequel
provided a domain-specific language in
addition spark sequel supports sequel
with ODBC JDBC server and command-line
interfaces the next component smart
streaming leverages the fast scheduling
capability of spark core for streaming
analytics ingest data in small batches
and performs RTD transformations on them
with this design the same application
code set that is written for batch
analytics can be used on a single engine
for streaming analytics machine learning
library lies on the top of spark and is
a distributed machine learning framework
with its memory based architecture it's
nine times faster than the Apache mahute
Hadoop disk based version in addition
the library performs even better than
the valve pal wabbit in addition it
applies various common statistical and
machine learning algorithms the last
component graphics also lies on the top
of spark and is a distributed graph
processing framework for the computation
of graphs it provides an API and an
optimized runtime
for the pre --gel abstraction the API
can also model this abstraction hey want
to become an expert in Big Data then
subscribe to the simply learn Channel
and click here to watch more such videos
to nerd up and get certified in Big Data
click here