PySpark Training | PySpark Tutorial for Beginners | Apache Spark with Python | Edureka Video Lecture | Apache Spark: Master Machine Learning - AI & ML

[Music]
in a world where data is being generated
at an alarming rate the correct analysis
of the data at the correct time can be
very useful now one of the most amazing
framework to handle big data in real
time and perform analysis is Apache spar
and if we talk about the programming
languages being used nowadays for
different purposes
I'm sure Python will top this chart as
it is being used almost everywhere so
guys this is Chris Lane from Aida Rica
and today we will be discussing about PI
spark and explore the benefits of with
the world that are the Python and Spa
now let's have a quick look at the
agenda I'll start off by talking about a
party spark its features and the various
use cases of spark in the industry then
we'll understand why python is best
suited to work with spark and moving
forward we'll discuss the PI spark
training provided by Ed Rekha we'll
understand some of the important
features like spark oddity spark context
and also spark streaming along with some
examples and practicals and finally I'll
finish off this video by showing you
guys some demos on PI spa so talking
about the features of apache spark
starting off with the most important
feature that is speed it is almost
hundred times faster than the
traditional data processing tools and
frameworks it has powerful caching the
simple programming layer provides
powerful caching and this persistent
capabilities now coming to deployment a
party spot can be deployed through
Miso's or Hadoop wire yarn or via sparks
on Cluster the most important feature
that help spark achieved the fast speed
and real-time computation and low
latency is the use of in-memory
computation the lazy valuation of
transformations the directed acyclic
graphs and much more
the spark is polyglot which means it can
be programmed in various languages like
Python Scala Java and heart and it is
one of the reasons why Apache spark has
taken over machine learning and
exploratory analysis now let's have a
look at the various companies that use a
party spot here we have a Yahoo Alibaba
Nokia Netflix NASA data Prix which is
the official enterprise distributor of
hwacha spark we have Trip Advisor and we
have eBay so you can see spark has been
used in the industry a lot
let's have a look at the various
industry used cases first of all
starting with health care as healthcare
providers look for normal ways to
enhance the quality of health care pata
spark is slowly becoming the heartbeat
of many healthcare applications many
healthcare providers are using a
participation tree cause along with the
past clinical data - RFI which patients
are likely to face health issues after
being discharged from the clinic this
helps hospitals prevent hospital
readmissions as they can deploy home
healthcare services to the identified
patient saving on costs for both the
hospitals and the patient Apache spark
is used in genome sequencing to reduce
the time needed to process UNAM data it
took several weeks to organize all the
chemical compounds with genes but now
with apache spark on hadoop it just
takes few hours now coming to finance
banks are using a participant and
analyze the social media profiles call
recordings complaint logs he means the
forum discussion to gain insights which
can help them make right business
decisions for credit risk assessment
targeted advertising and customer
segmentation what are the financial
institutions that has retail banking and
brokerage operations is using a party
spark to reduce its customer churn by
25% the financial institution has
divided the platform's between retail
banking trading and investment however
the bank wants a 360-degree view of the
customer regardless of whether it is a
company or an individual to get the
consolidated view of the customer the
bank uses Apache spark as the unifying
layer the spark helps the bank automate
analytics with the use of machine
learning by accessing the data from each
depository for the customers not talking
about media Apache spark is used in the
gaming industry to identify patterns
from the real-time in-game events and
respond to them to harvest lucrative
business opportunities like target
advertising auto adjustment of gaming
levels based on complexity
player retention and many more now con
vive is another company averaging about
four million videos per month it uses
Apache spark to reduce customer churn by
optimizing video streams and managing
live video traffic thus maintaining are
consistently smooth high quality viewing
experience now you all might have heard
about Netflix Netflix users are patches
for real-time stream processing provide
online recommendation to its customer
streaming devices at Netflix send events
with capture all member activities and
play a vital role in personalization it
processes 450 billion events per day
which flow to the server-side
application and are then directed to
Apache Kafka now coming to the retail
and e-commerce industry one of the
largest e-commerce platform Alibaba
let's solve the largest apache spark
shops in the world in order to analyze
hundreds of petabytes of data on its
e-commerce platform so the spark jobs
that perform feature extraction on image
data then for several weeks
millions of merchants and users interact
with Alibaba ecommerce platform each of
this interaction is represented as a
complicated large graph and a party
spark is used for fast processing of
sophisticated machine learning on this
data now eBay also uses Apache spark to
provide targeted offers enhance customer
experience and optimize overall
performance Apache spark is leveraged at
eBay through Hadoop Yan Yun manages all
the clusters resources to run the
generator tasks and eBay spark users
leverage the Hadoop cluster in the range
of 2000 notes 20000 course and 100 TB of
RAM through yon now finally coming to
the travel industry Trip Advisor a
leading travel website that helps plan a
perfect trip is using Apache spark to
speed up its personalized customer
recommendation Trip Advisor uses Apache
spark to provide advice to millions of
travelers by comparing hundreds of
websites to find the best hotel prices
for its customers the time taken to read
and process the reviews of the hotels in
a readable format is done with the help
of Apache spark now we all know Oba
every day at this multi national online
taxi dispatch company gathers terabytes
of event data from its mobile users by
using kafka spark streaming and SD FS to
build a continuous ETL pipeline uber can
convert a structure event data into
structured data as it is collected and
then use it for further and more complex
analysis so as I was talking about spark
being a polyglot earlier it basically
means that programming in spark can be
done in various languages like Scala
Python and ha now you may ask which one
should I go
I choose to begin with so spark was
developed in skaila
that you should know it was a default
language in which spark was developed it
is very similar to Java but the recent
emergence of data analytics and machine
learning has made it difficult for Scala
to keeper so instead spark came up with
a Python API to use Python so let's have
a look at the reasons why one should go
for Python to begin with firstly it is
easy to learn for programmers Python is
comparatively easy to learn because of
its syntax and standard libraries
moreover it is dynamically typed
language which means our IDs can hold
objects of multiple types now we will
discuss more about our it is coming up
in this video not it's portable and it
can be used with various operating
system like Windows Solaris Linux we
have the be US PlayStation and the Mac
OS and lastly Scylla does not have
sufficient data science tools and
libraries like Python for machine
learning and natural language processing
SPARC ma lib the machine learning
library has only fewer ml algorithms but
they are ideal for big data processing
in summary we can say that Scala lacks
good visualization a local data
transformation tools along with highly
used machine learning libraries now
Eureka as we know provides a detailed
and comprehensive training on Apache
spark in Python that is the PI SPARC
developer certification training now
this course is designed to provide
knowledge and skills to become a
successful SPARC developer using Python
you will get in-depth knowledge of
concepts such as Hadoop distributed file
system the Hadoop cluster flu scoop
Apache Kafka you learn out the api's and
the libraries which spark offers such a
spark streaming MLF spark sequel and
this by smart developer course is an
integral part of a big data developers
career path this course is designed to
provide knowledge and skills to become a
successful Hadoop and spark developer
and would help to clear the spark and
Hadoop developer which is the cca1 75
examination this course has in total 12
modules with one bonus module and it
focuses unclouded as Hadoop and spark
developer certification training now
coming to module one which is the
introduction to Big Data Hadoop a spark
in this model you will understand Big
Data the limitation of existing
solutions for big data problems how
Hadoop solves the big data problem the
Ecosystem Hadoop architecture HDFS rack
of it has had replication you will learn
about the Hadoop cluster architecture
important configuration files in Hadoop
cluster and you will also get an
introduction to spark why it is used and
understanding the difference between
past processing and real-time processing
now coming to module 2 which is
introduction to Python for Apache spark
at the end of this model you will be
able to define Python understand
operands and expressions you will be
able to write your first Python program
understand command-line parameters and
flow control you will understand how to
take an input from the user and perform
operations on it and you'll also learn
about numbers strings tuples less
histories and sets now coming to module
3 which is basically the functions oops
modules error and exceptions in Python
this module you will learn how to create
general pattern scripts how to address
the errors and exceptions in the code
and finally how to extract and filter
content using the red X now module 4 is
deep dive into a party SPARC framework
in this module you will understand
Apache spark in-depth and you will also
learn about the various spark components
you will be creating and running various
spark application and at the end you
will learn how to perform data injection
using scoop now coming to model 5 which
is playing with spark oddities in this
module you will learn about spark
oddities which are the resilient
distributed data sets and other oddity
related manipulations for implementing
business logistics like the
transformations actions and the
functions performed on a DD moving on to
model 6 which is the radar instant
sparks equal in this model you will
learn about spark sequel which is used
to process structured data with sequel
queries you will learn about the data
frames and the data sets in spark sequel
along with different kind of sequel
operations performed on the data frames
you will also learn about the spark and
hive integration now module 7 which is
machine learning is in spark MLM in this
module you will learn about why machine
learning is needed different machine
learning techniques and algorithms and
their implementation using the spark ml
lip now its model is steve dive into
spark MLM and this model you will be
implementing various algorithms
supported by a machine learning library
which is the ML Lib such as linear
regression decision tree render forest
and many more now coming to moral line
which is
understanding Apache cop-con Apache flew
in this model you will understand Kafka
and Kafka architecture afterwards you
will go through the details of copper
cluster and you will also learn how to
configure different types of copper
cluster at last you will see how
messages are produced and consumed using
Kafka API is in Java you will also get
an introduction to Apache flew its basic
architecture and how it is integrated
with Apache Kafka for event processing
now model 10 is Apache spark streaming
in this model you will work on a
participants streaming which is used to
build scalable fault tolerant streaming
application you will learn about these
streams and the various transformations
performed on the streaming data you will
get to know about commonly used
streaming operators such as sliding
window operator and the stateful
operators now model 11 is Apache spark
streaming data sources in this model you
will learn how the different streaming
data sources such as Kafka and flume and
at the end of this module you will also
be able to create a spark streaming
application now model 12 is the in-class
project now this project will comprise
of all that we have learned till now
that is Hadoop spark Kafka flu and much
more and as a bonus we have another
model which is the graphics in this
module you will be learning the key
concepts of spark graphics programming
concepts and operations along with the
different graphics algorithms and their
implementations now that we have seen
the training structure offered by Erica
let's understand what exactly pi spark
is now a party spark is an open source
cluster computing framework for
real-time processing developed by Apache
spark foundation and PI spark is nothing
but the Python API for Apache spa now
let's have a look at the various
components of the spark ecosystem the
core engine of the entire spark
framework provides utilities and
architecture for other components spark
streaming enables analytical and
interactive ask for live streaming data
the ml live bitches machine learning
library of Spa it is built on top of
SPARC to support the various machine
learning algorithms graphics
computational engine which is similar to
cry off combined state apparel and craft
parallel concepts SPARC R is the package
for our language to enable our user to
leverage
power from the our shell now finally we
have the pie spar the API developed to
support Python as a programming language
force path now PI spark shell links the
Python API to the Sparco and initializes
the spa context the spark context is the
heart of any spark application spark
context sets up internal services and
establishes a connection to a spark
execution environment the spark context
object in the driver program coordinates
all the distributed processes and allows
resource allocation the cluster managers
provides executors which are JVM
processes with large spark context
object sends the application to the
executors and then spark context
executes these tasks in each executors
now when you have installed spark on
your system just by typing PI spark you
can enter the spark shell and it looks
something like this just make sure that
all the demons or spark are running in
the background now that you have entered
the spark shell it is important to know
about rdd's when it comes to iterative
distributing computing that is
processing data over multiple jobs in
computation we need to reuse or share
data among multiple jobs in earlier
frameworks like Hadoop there were
problems while dealing with multiple
operations we need to store the data and
some intermediate stable distributed
storage such as HDFS the multiple i/o
operations make the overall computation
job slower and the replication says
serialization which in turn makes the
process even more slower already's try
to solve all these problems by enabling
fault tolerant distributed in memory
computation now heartediy is nothing but
resilient distributed datasets
since our IDs are created over a set of
transformation it locks those
transformations rather than actual data
it is a distributed memory abstraction
which lets programmers perform in memory
computation or large clusters in a
fault-tolerant manner they are read-only
collection of objects partition across a
set of machines that can be rebuilt if a
partition is lost our release can be
created for multiple data sources like
Scala collection the local file system
we have the Hadoop HDFS Amazon s3
HBase table and so many more now there
are several operations performed on
rdd's which are the transformations
actions and the
options so let's have a look at the few
transformations and actions you can do
the programming in shell or you can
integrate your Jupiter notebook with
SPARC so that every time you start the
SPARC shell you are automatically
redirected to the trip returned notebook
now personally I find the Jupiter
notebook a little bit easier to use
because of the color combination and all
that but it comes down to your choice
so let me just start my VM so before
starting the spark shell you need to
check whether all the demons of spa are
running
as you can see worker is running but
still I'll stop and start it once again
now the stop - all that Sh will stop all
the spark Beeman's
now to restart all the DMS all you need
to do is change the stop to start
so now if I check whether all the demons
are running or not
you can see the worker and the master is
running
so let me start the spark shell just by
typing PI spark I'll enter into spark
shell and I will be redirected to the
Jupiter notebook
so as you can see it has automatically
opened the true Pathan notebook for me
so all I will do is select a new
notebook of Python to listen a meter
Zarate practice
so first we'll see how to take a text
file as an input so as you can see I
have created a RDD and we have used the
SC dot eps file so one important thing
to note down here is that by default the
path you provided is for the SC FS path
so in order to take the file from your
local file system you need to mention
the file colon and to forward slash now
to have a look at the output of the
first five elements all we need to do is
enter our daily dot take five as you can
see I have the output first five output
let me mention as you one two three four
five now suppose if you want to convert
all the words in the Oddity to the
lowercase and split those lines of the
document using this space for that will
create a function to do both the task
I'll show you how it's done
so as you can see in this function I
have taken the input in lines and I have
converted it to dot lower and dot split
will split the lines and I've created
another oddity which is the RET one so
now if we take a look at the already one
you can see here all the words are in
lower cases and everything is split now
the output of this is so long so if you
want to flatten the output will use the
function or rather I'll say will use the
transformation flat map so let me show
you the flat mark transformation so here
I am going to create a daily to which
will take the initial already and cover
it into a flat map function and then
we'll see the output of the first five
elements as you can see we have the
output as the Apache flat map output now
suppose I want to remove the stop words
in my file we need to define the list of
stop words in a variable called stop
words now Here I am selecting only a few
words in the stop word list instead of
all the words now applying filter to the
RTD to I looking at the first and output
and comparing it with the initial one we
can see we have removed though for all
these stop words which I have mentioned
are not here now after getting the
result into our d3 we want to group the
words in the early d3 based on the
letters that they start with for example
I want to group each word of the
auditory based on the first three
characters all I've got to do is use the
group biotransformation or the group by
action I should say as you can see we
have the alo and alo ng along now what
if I want to work with the samples
instead of the full data now the sample
transformation help us in taking samples
instead of working on full data the
sample method will return a new oddity
containing a statistical sample of the
audition RDD we can pass the arguments
inside as the sample operation so as you
can see in the are d3 door sample I have
given the parameters false 0.4 and 42
now false is basically the with
replacement parameter it is used to
choose the sample with or without
replacement so I've given a false now a
fraction is 0.4 which means we want to
choose to forty percent of the data in
the RDD and the seed which is 42 is used
for the reproduced of the results
now if you want to join the two pair
oddities based on the key we have the
joint transformation which helps us do
the right thing first we create the
two-sample key value pair additives
which are the sample1 and sample2 so now
that we have created the sample1 and
sample2 applying a joint transformation
on sample1 and sample2 will give us the
desired output so as you can see here we
are joining sample 1 to sample 2 and we
are taking the first two output I'll
show you as you can see we have the
output here you a UT you have the a T
and T and we have the 8 T and P now if
you want to find out the sum of all the
numbers and an R DT the solution to
demonstrate this I will first create an
oddity from the list of number from 1 to
10,000 and then we'll reuse the reduce
action and pass a function through it
which is the lambda x and y such that
the output is X plus y and a reduced
action is used for aggregating all the
elements of Oddity by applying pair wise
use of function so now that we have seen
some of the transformations and actions
which we can perform on our ID let's
understand one of the use cases which we
have here which is the NBA used case now
we are using the data from the basket
world reference so for that I'll open a
new notebook and I'll name it as NBA
stats or enviar practice now here we are
using the data from the basketball
reference we read in the seasonal total
starts for every player since the 1979
to the 1980 season into this park data
frame using the PI Spock
so let me rename it as in the practice
so as you can see here we have data
and we are using the red dot option with
header true and info schema true you can
see here I have taken the input data
file from the local file system which is
the season total start CSV so now we can
also view the column names or our data
frames for that we are going to print
the data frame dot columns so I can see
we have the underscore Co which is the
first column name we have the player
position age team ID the ggs MP F G just
a three point the first point to the
second point where the total points and
we have the years now using our data
frame we can view the top 10 players
sorted by number of points in an
individual season the notice we use the
two pandas function to retrieve our
results the cross one result looks
cleaner for display than when we are
using the tape function
so as you can see the output is much
cleaner and in a proper format
now next using the data frame
domain-specific language is the DSL we
can analyze the average number of three
points attempt for each season scaled to
the industry standard which is the 36
minutes the per 36 minute metric
provides an estimate of a given player
stats projected to 36 minute an interval
corresponding to an approximate full NBA
game with adequate rest while also
allowing comparison across players that
play a different number of minutes so as
you can see here we are using the group
Y we are using the aggregation we are
using the Select we using the order by
now we compute this metric using the
number of three-point field goal
attempts which is the FG 3a and the
minutes played which is the MP so as you
can see here we are taking the sum of
the FG 3a and you are adding with the
column sum of the MP which is the
minutes played
now that we have a graded our data and
computed the average attempts per 36
minutes for each season we can query our
result into the pandas data frame and
plot it using the matplotlib
so as you can see we are imploding the
matplotlib library here let us just
compute and plot it
now we can see a steady rise in the
number of three points attempt since the
shots introduction in the 1979 to 1980 s
along with a blip in the number of
attempts during the period in the mid
90s when the NBA moved the line in a few
feet now we can fit a linear regression
model to this curve to model the number
of shots attempt for the next five years
of course this assumes a linear nature
of the rate of increasing attempts is
likely an assumption but still let's do
the analysis firstly we must transfer
our data using the vector assembler
function into a single column where each
row of the data frame contains a feature
vector this is the requirement for a
linear regression API in MLM we first
build the transformer using our single
variable wire which is the year and
transform our season total using the
transformer function next we'll build a
rain elevation model object using our
transform data now next we want to apply
our train model object to our original
training set along with five years of
future data containing this time period
we build a new data frame transform it
to include a feature vector I then apply
our model to make a prediction so as you
can see here we are applying the model
for those 1979 to 80 season through 2020
and 21 season and we are building the
test data frame and applying the linear
regression model so now when we want to
plot our result let's see what the
result tells us so as you can see we
have fitted a linear regression model to
this one and the prediction is still
2020 now we have saved this image in our
local file directly which is the home
iterator downloads image section I'll
show you that I'll go to the files in
the downloads in the images of the
regression so you can clearly see here
we have the linear regression line and
we have the future prediction here
according to our model we created just
now earlier so this is it guys I hope
this session was helpful and I hope it
helped you a lot in understanding the PI
spark all the features the reason why to
choose Python training provided here at
Erica and I hope you enjoyed this video
thank you I hope you have enjoyed
listening to this video please be kind
enough to like it and you
can comment any of your doubts and
queries and we will reply them at the
earliest do look out for more videos in
our playlist and subscribe to Eddie Rica
channel to learn more happy learning