Big Data Analytics With Spark | Big Data and Spark Tutorial | Spark for Beginners | Edureka Video Lecture | Apache Spark: Master Machine Learning - AI & ML

hello everyone welcome to the Apaches
park so before we proceed first of all I
would like to give a quick introduction
about myself and then we will proceed
with the discussion folks so folks my
name is Mahesh I have got extensive
experience in big data to mine across
what across multiple industries I have
worked with travel airlines security
networking telecom banking and many
other domains many other industries
including oil and oil and gas applying
the big data analytics across different
verticals so I've been working in this
industry like since more than six years
specific to be digital mine and my core
skill sets include basically the Hadoop
and then the ecosystem technologies
spark and in fact I am as well a data
miner using the mahute and our kind of
mission learning tools okay so that's a
quick introduction about myself
I hope my versatile domain experience
could help you all in terms of
understanding that you will use cases
and the big dated of mine itself so I
would at the moment I would like to
quickly start with the discussion so
apologies for not taking your
introductions at this point in time okay
so in this quick webinar today now in
this one our discussion today what are
we going to learn let's have a quick
look at the agenda for today first we
will try to understand what is Apache
spark why there is a need for a new
framework when already a permanent
framework existing and solving industry
problems for big data sets
for example Hadoop so why there is
another technology called as Apache
spark so we'll try to understand that
and then how spark fits into Hadoop
ecosystem so is it's so many of you are
a bit confused or I personally interact
with many new delibird so basically they
get confused with their spark could
replace Hadoop or whether it's it's an
complementary software along with the
Hadoop so let's have a quick look at how
actually spark fits into Hadoop
ecosystem
then why spark for big data analytics
why specifically we need to choose our
relay embedded analytics so here the
topic will get more interesting so
that's where I try to explain you
theoretically as well as try to show you
certain examples which will help you out
in clearly distinguishing between the
classical way of processing the data on
Hadoop versus the spark then sparks
popularity so some of the theoretical
slides which basically gives you an idea
on how this park of future looks like
and if you really adopt this park from
here onwards so how would we
how would be where killer part look like
so we will try to understand that and
finally will trying to run a specific
complex use case on top of a spark and
then we'll try to understand how simple
is it it is to implement the code in
spark by solving a really complex use
case and we'll probably in the word
discussion at the point in time so does
that sounds like a good plan for for one
hour discussion folks just say a certain
or even smiley should work excellent
okay
I'm sure you guys are quite enthusiastic
to go and start discussion okay so let's
see so here is the quick introduction
about this part framework so it's
basically talks about it's a simple
definition about the Apache spark by the
way it says that Apache spark is a
general-purpose data processing engine
of with in-memory computing and then
spark provides API first color Java
Python and arm which makes spark widely
adopted for data processing requirements
so let's try to understand each one of
these definitions in detail so to stop
it it says that apache spark is a
general-purpose data processing engine
with in-memory computing capability so
what does it mean why am i referring to
referring spark as a general-purpose
data processing engine so why am i
calling it as an engine in the first
place so let's discuss a few points
about that so folks this park is one of
the very interesting frameworks
available today in the big data
ecosystem which could work as a
pluggable engine for multiple underlying
data sources you can plug-and-play spark
quite quickly with most of the
underlying data sources available today
for example one of the most prominent
big data source is basically HDFS Hadoop
distributed distributed file system so
you can easily integrate spark with the
loop and then quickly run your
applications on top of spark rather than
depending on tau or depending on her
loops MapReduce directly
and as I mentioned it supports multiple
data sources you could plug and play
with additional data sources for example
no sequel databases like MongoDB or even
Cassandra are many would have many more
dead no sequel databases available today
and it's not limited to no sequel
databases of course you can even connect
it with relational databases for example
like my sequel and you can analyze the
data quite quickly compared to that of
the primitive engines available to
process the data in those data sources
ok so we will try to understand why it's
very fast it's because spark has got
ability to process the data in memory so
that's the beauty of spark we'll go into
the details and more into more details
in few other slides so why I'm calling
it as an in-memory computing system and
how does it benefit for an end user or
for the developer at least in our point
of view ok so back in 2010 when I have
been extensively working with Hadoop
iDEN they were discussions about the
in-memory processing frameworks for
large data sets like say data relay data
residing on top of her duper so if you
ask my genuine opinion at that point in
time I had was I was really confused
that whether is it really possible to
process such massive data sets sitting
on top of Hadoop with an in-memory
processing system is it really true or
is it going to happen even though there
were a lot of discussions around that in
2010 then I started finding out a
technology called a spark
somewhere in between 2011
so this framework has got a very
interesting internal mechanism with
which it could be able to load the data
across the cluster of machines in memory
and then process the data quite
efficiently compared to that of the
classic MapReduce programming paradigm
okay said there are too many technical
aspects now so let's quickly move on and
try to understand each one of those
definitions indeed okay so before that
even spark provides API for Scala Java
Python and even now with the latest
releases in spark like 1.4 1.5 spark has
built very efficient API is for example
data frames and some functions with
which you could be able to easily
liveries on the are functions and could
run on top of spark so with the
implementation of the data frames and
some other functions we have built as
part of the our module so let's move
further and try to understand that so
then SPARC is intended to enhance not
replace the Hadoop stack okay so before
going further so there are a few
questions which I would like to take at
this point in time and then we'll move
forward so Ganesha says I have spoke it
so I think there are some specific
questions related to their working their
current situation so I would like to on
address such questions at the end of the
discussion Ganesh Manish were sorry and
then Pollux is actually in memory
computing system most of the time we
lost the processing data and so weaken
validate our
when developing a code so how could we
resolve this sure so I will definitely
address this question at the end of the
discussion public so I'm not going to
take questions which are not related to
the context so which could easily
disturb the session but don't worry I'll
definitely help you out in answering
those questions at the end of the
discussion
okay so Vijay says how does learning
spark help senior QA professionals say
the key okay so I think there are some
very very interesting questions from
public and some others so I will pick up
such questions at the end of the
discussion so please bear with me till
we finish up with this game folks okay
I'll give you maybe roughly about 15 to
20 minutes time to answer your questions
okay so spark is intended to enhance not
replace the Hadoop stack so this is very
very important and crucial statement to
understand for all of you it's because
many of you in fact even in the forum's
I continuously I frequently interact
where I see the questions related to
whether spark and replace the ribbon
which is totally wrong because Hadoop
has caught its popularity it's not only
because of the MapReduce programming
paradigm it's mainly because of its
underlying distributed file system which
is the most stable open source
technology available with us today with
which you could have scale infinitely
across any number of nodes over the
cluster with the Hadoop distributed file
system and with the advent of generation
2 lot of things lot of issues have been
resolved and it has become more stable
and more scalable and many more features
have been introduced
like for example multi-tenancy
so spark on the other hand does not have
its wound storage system as of today so
that's very very important to understand
spark is mainly designed to be a compute
engine which could easily be plugged to
any underlying data source and can
process the data quite quickly and
efficiently so that's the whole idea
behind this power framework so then what
is the purpose often of actually moving
on to spark rather than some of the
conference in head open so let's discuss
what that so folks many applications
since like last I would say last decade
have been implemented using Hadoop
MapReduce framework so it went it worked
very well
it was quite efficient and work very
well for large citizens because her loop
was in hilly purely designed for batch
processing applications and which fits
very well with such kind of environment
but the growing number of communities on
top of Hadoop user communities on top of
Hadoop how started leveraging and Hadoop
in different aspects so they started
treating her loop as alarmed
architecture the process streaming they
turned on the batch processing data they
started considering Hadoop for real-time
applications as well so that's where the
introduction of so many ecosystem
technologies has come into picture and
existence ok so then this parks
MapReduce on the other hand is quite
performance computing framework
computing framework as compare to that
of the hadoop mapreduce directly it's
because the underlying implementation is
entirely different the way it does the
MapReduce spark also has got MapReduce
Hadoop by default has somehow produced
framework so both the implementations
are entirely different conceptually both
of both of them are same because of the
underlying implementation is totally
different you could actually achieve
high performance on top of this Park for
certain applications the performance of
spark could go beyond hundred times than
that of the Hadoop MapReduce folks so
that's why even today I personally see
many of the domain many of the companies
they started totally replacing their
existing hadoop mapreduce - sparks
MapReduce even though there are some
consistency related issues which have
been solved which have been getting
result quite quickly because of the
number of user community support
provided for spark especially because of
the performance this part MapReduce ills
the spark is designed to read so in
essence so spark MapReduce is quite
efficient and spark is not only meant
for MapReduce it has got some another
very interesting modules which we'll be
discussing in the moment which also
makes the spark more popular and widely
adopted framework are the future
framework okay so spark is designed to
read and write data to HDFS as well as
other storage systems such as CSV files
Amazon s3 and no sequel databases so as
we discussed say it's supposed many
underlying data sources
okay so what makes park suitable for big
data analytics so folks if you consider
the classical Hadoop world so basically
here if you have gotten the ability to
process RM lays the data wire MapReduce
but with the advent of generation 2 with
the introduction of an framework which
is said to me at another resource
negotiator so with the introduction of
an framework
even though Hadoop opens up the channel
for multiple types of applications to
run on top of Hadoop other than the
MapReduce MapReduce just become the part
of a part of it so it actually and
actually allows you to run other types
of workloads like graph processing
applications are say streaming
applications are some messaging
applications so different types of
applications are now supported on top of
Hadoop and makes her lips tool to
continue in the in the open source world
so okay so now that we have got an since
now that an allows you to run different
types of applications on top of Hadoop
then why we are saying that spark is
suitable is most suitable for big data
analytics it has an interesting
description here
so SPARC is not only about MapReduce
folks so keep in mind that spark has got
different modules which are packaged in
a single framework called a spark so
what are all those other modules which
are actually packaged as part of the
spark spark MapReduce is by default and
then
Sparks graphics which allows you to
write grab specific applications on top
of spark spark streaming module which
allows you to integrate streaming
applications with spark quite easily and
efficiently even though there is slight
difference between other streaming
applications and spark streaming module
which I will LLL discuss both in a
moment so spark streaming spark graphic
spark Emily Emily package is another the
most prominent framework nowadays
because of the adoption and because of
the number of algorithms been
implemented in sparks Emma live machine
learning library package so then finally
last but not the least in a spark sequel
which allows sequel developers to
quickly able to analyze the massive data
set sitting on top of any data source
where this Park sequel quite quickly are
quite efficiently so these are different
modules bundled together as a single
framework so how does it matter
so Hadoop having an in place supporting
all types of applications on the other
hand spark being a single framework
having all types of workloads integrated
together how does it matter tree
developer to a company it's very very
crucial to understand so folks for you
you know need to learn multiple other
frameworks laying around the hoop
ecosystem just for example if you want
to do spark cycle programming or if you
want to do graph analysis if you want to
do if you only apply most complex
mathematical models if you want to
say for instance process the streaming
data so you no need to learn multiple
technologies you no need to worry about
how to integrate those technologies with
your Hadoop platform to be able to
seamlessly walk for you so that's where
SPARC gains more even more popularity so
you know to learn such different
frameworks rather you just learn single
framework all a spark so you are done so
you you be able to run any of these
types of workloads because of all these
modules being bundled together as a
single framework so let's move on and
see some other aspects of spark so spark
simplifies data analysis so we will see
so this is also very very important to
understand folks why it simplifies data
analysis it's because the next couple of
slides will give you very good
understanding and white simplifies the
data analysis for example hadoop
mapreduce being the Java framework
you had to write tons of lines of code
to implement simple application in
MapReduce in Hadoop MapReduce on the
other hand SPARC being implemented in
Scala Scala is the is the language in
which spark framework has been built it
allows you to implement complex
applications in very few lines of code
especially because of its scalable
computing capabilities this color
provides said that's the key
differentiator for hadoop mapreduce and
then this parks MapReduce so I'll show
you I'll show you practically how many
lines of codes are reduced while you
implement the code in spark
then spark provides built-in libraries
to do advanced analytics so it has got
built-in mission learning library
package which has got implementation for
different types of algorithms like super
waste learning techniques it has got
unsupervised learning techniques it is
good regression algorithms it has got
recommendation information retrieval
kind of algorithms and this and this
package is growing quite quickly because
of the number of number of developers
supporting that particular package and
in fact today we have got support for
our as well so not all the functions in
our are supported in spark but they
don't quite a number of functions which
are most popular which are today's
bottle in spark where oh okay so then
spark speaks more than one language so
this in the first slide we have seen
sparks of boats Java Python and even
spar and even Scala so what does this
mean to us basically so any developer
development community may not need to
learn a new technology to be able to
implement the code on top of spark
rather they can implement in their own
fell a well known technology for example
say Python if you are a Python developer
just don't worry just go ahead and
implement evil code in Python Westpark
if you are a Java developer just go
ahead and implement your code in Java so
if you are from Scala background so just
use color so it has got multilingual
support multi-language support
okay so then SPARC provides faster
results
so widest part provides pass faster
results it's because spark basically is
built around an interesting data
structure called as Rd be resilient
distributed data set so this data
structure in fact I would say most of
this park components has been actually
built around this data set folks this
data structure which actually allows you
to load massive data set into memory
with high reliability and did high
efficiency as well and lot of other
techniques actually follow that so
because of that because of the in-memory
computing capability spark is quite
efficient than that of any other
frameworks and this park allows you to
use different Hadoop vendors so as I
mentioned SPARC is not just supporting
Apache Hadoop Swiss park today supports
multiple third-party vendors available
over the market same map are say
cloudera say harden works even say
whoever distribution so most of you may
not be aware that there is something
called as who were her loops
distribution which is available in the
market so even SPARC is important in
that kind of distributions as well okay
so let's take a quick look about the
first point which I discussed so it is
quite simple to implement code in spark
why here is one slide folks so please
have a quick look on the slide could you
tell me how many lines of code has been
written in this light everyone on board
so this program is basically Worldcon
program which is a kind of a lowell
program
for hadoop mapreduce so approximately
you could imagine somewhere around like
40 to 50 lines of code has been
implemented in this light for loops
MapReduce the same functionality the
same program can be implemented in spark
let's see in how many number of lines in
the next slide so folks the same program
has been implemented in spark just in
three lines of code if you look at the
first three lines here so this basically
allows you to perform the word count
operation in just three lines why is
there such a drastic difference it's one
of the key reasons is mainly because of
this color programming language so as I
already mentioned color allows you to
implement highly scalable code and the
development time required for
implementing applications in Scala is
quite quite less than compare to that of
the Java programming language a session
okay so here is you know a quick
differentiation between our loops
MapReduce versus the sparks MapReduce
folks so this clearly processing data
with spark is much easier than MapReduce
and spark gives you the flexibility to
choose your favorite language as well so
it is not really required for you to
implement the code in only specific
language a scholar Archer or ro Java so
you can actually switch over any type of
language depending on how flexible you
were in which type of language so if you
observe carefully this light the top
three lines over here has been
implemented in scholar versus the down
three lines have been implemented in
Python okay
so what I will do at this point in time
so we have been talking about too many
conceptual things so I would like to
show you practically how spark is more
efficient than hadoop mapreduce just
quickly I'll run a simple application
the same simple word Konda application
on top of Hadoop versus on top of Scala
so let's have a quick look at our map
folks so what I'm going to do here
I'll quickly run my virtual machine and
I'll show you the installation directory
of spark or maybe I can skip that step I
can quickly start both the loop and
spark clusters and then run the word
count program agnus both these
frameworks and we'll literally try to
compare the performance between hadoop
mapreduce verses parts of MapReduce ok I
know there are plenty of questions of
floating around please uh please accept
my apologies
all those slides and then I'll give some
time to you to ask your questions so
I'll apply for your questions folks in a
moment okay so I'm just going into my
Hadoop so which is to that full at one
and then I'll quickly start my hurdle
cluster at the same time I'll stop my
start my spark cluster as well maybe
I'll take another 10 to 15 minutes to
continue with the discussion and then
probably I'll give the rest of the time
for the queue
the folks so I'm starting my spark
cluster now my hurry pleasure has
already been started so I can quickly
check I think almost all the services
are running quite successfully
I can see master work curvature from
spark and I can see the remaining
services are from Hadoop okay so let me
quickly run the word cone program so I
hope you everyone is comfortable with
familiar with word con program right so
basically word count program allows you
to count the words I'm sorry to count
the awards from the given file okay so
what I am doing here is basically I'm
running the Hadoop command I didn't
charge some wood Condor char and then
here is my class which does the work on
top raishin just the same goal i have
just put it on the slide and then they
were the input pattern in the output
path so let me change my output paths to
some other directory so now i'm running
this application the count do move to
counter words so i'll put curry already
exists okay no problem let me take
another nap or diet tree
excellent
so now my program has started running it
just complained I just shouted on me
saying that your output directory was
already existing so please remove the
directory so rather than removing my
output area okay I have specified a new
directory path I think we should suffice
our needs
so my mapper is my mapper successful
100% my reduce it is about just heart
which is still running I can see it's
still running my reducer also has been
successfully X
so folks if you carefully notice how
much time this application has taken to
complete so it's basically this
application started somewhere around
let's say somewhere around starting from
743 and it has completed attend home
like say 816 which is like cumulate you
how much time it took somewhere around
17 plus 16 say approximately 33 minute
33 seconds to complete this job on
hadoop mapreduce so let me try to run
the same word con program in in spark no
I'm trying to compare the performance
okay so now spark another beauty of
Sparky spacegate has caught an
interactive show with which you could
interactively run your applications no
need to run it remotely you can
interactively run your code and then
quickly either learn are just debug your
application folks again it is beneficial
for both of this Hido for learning or
for debugging your application so now I
launch that interactive shell okay so
with binge / spark sure okay so no it is
launching this partial now so once it is
launched I'll quickly run the program
for world countdown spark so in this
case just bear in mind one thing that
the data is being looked up from HDFS
directly and then the processing happens
on spark okay so I'm going to just run
three lines of code I have shown you
previously folks
okay so now that my spark shell has been
started successfully so let me quickly
run three lines of code so first I will
load the data into something called as
file oddity so in order to load that
I'll just copy and paste the command so
my part is incorrect so let me check my
path here so my path is basically this I
marked a slash word cone problem okay so
I'm running this code they just loaded
the file and then I will run the other
command which basically does the
counting of the words I'm not going to
go into the details of this but I'll
just quickly run the command folks okay
so the word count has been computed now
I'll just quickly print those word
counts onto the console
okay so now when I run this command it
basically starts the execution so first
column basically you need to have a
triggering point to kick off the project
or to kick off the application so now
this command basically starts the
application and let's see how much time
this is going to take okay so
essentially if I carefully notice even
from the beginning which is 1152 delay
and 11:57 so how many seconds it has
taken folks approximately five seconds
so this basically tells us practically
that the the performance of spark
MapReduce is very higher than the Hadoop
MapReduce framework maybe this is very
very small so you might ask me why do
you really care about 30 seconds versus
say 5 seconds does it really impact
anything in your from your side
no but you should think about this in
the terms of highly scalable
applications or high scale data sets
where you're talking about terabytes
then the amount of time taken will
become quite significant to meet the
soleus level a glooming agreements
defined by your business users okay
so again I don't want to actually
leverage use too much of business
terminology here here the intent is to
show quickly the difference between
these two frameworks okay
so let me come back up quickly and see
what else is remaining for today okay so
folks part is blazingly fast we have
seen that in fact here is an example
just have a quick look around the slide
it says that using spark on tune or six
easy two machines Amazon Elastic Compute
cloud machines we started who is that it
is basically the nulls in committed ever
PMC's from parties marco fond of data
breaks it's basically data breaks was
able to sort native bricks is basically
the key contributor behind the apache
spark framework folks so there they are
able to sort hundred terabytes of data
on disk in 23 minutes
in comparison the previous world record
set by her loops MapReduce used two
thousand one hundred verses - not six
machines and took 72 minutes so
previously Hadoop MapReduce took seven
to two minutes on 2,100 machines versus
two days Park took only 23 minutes which
is maybe somewhere around two bit third
of the time and look at the number of
machines it is nowhere comparable
maybe somewhere one by tenth I think
less than or less than less than less
number of notes
this means that spark started the same
way that three times faster using ten
times fear machines okay so that's
that's quite into do right and then
sparks equal so here are the different
modules which we have already discussed
spark has got different packages when it
sparks equal which allows you to quickly
run sequel queries on top of hi to
basically you know just leveraging the
hi metadata which is on high metal store
and then just runs the queries and hops
on top of spark to which you could
analyze the high data then even HDFS
data then ml loop basically allows you
to run any complex mathematical model
quite quickly and efficiently even
developers could be able to easily learn
and run the complex algorithms and
massive data sets folks so you know need
to be really a data scientist by nature
but if you if at all you are just from
development background still you are
able to use those algorithms quite
quickly the learning curve is quite less
even in MLM then graphics which allows
you to run the graph analysis on top of
Spartan spark streaming so here is what
I would like to differentiate between
the original streaming applications
between star spark streaming so the
quick difference is if you are talking
about palm framework which is quite well
quite familiar as the streaming
application streaming framework as it is
true even today so other sub-second are
even at a microsecond level you could
process the data using spark streaming
sorry i'm my apologies with storm
framework atom microsecond you could
process the data the moment the record
enters into the system you could analyze
that record with Strom framework but
with spark streaming framework there is
a slight different itchbay which
basically allows you to do
screaming in mini batches the mini batch
could be as less as 30 seconds are 15
seconds but still it is a mini batch
system it's not a pure streaming
application because folks bear in mind
one thing there are many applications
which require you to process the data at
a microsecond level okay so that's the
difference between spark shaming and
other streaming applications for example
like storm
so then spark multi-language support or
here all different languages since part
Python Java Scala sequel and all so you
can see the adoption of these languages
so basically Scala has got the higher
adoption few questions I have seen which
which language is good higher adoption
71% is easy since color Java is 31% and
58% is in Python it basically depends on
how many number of users are out there
who wanted to use park and then sequel
also then R as well so here is the
fantastic slide throughout this PPT I
like the slide the most because it gives
a very detailed understanding and how
spark has been exploited three Apache
sparks over two thousand fifteen thirty
one person days are still under
evaluation phase that's really good
folks for all of us because who ever
want to step into spar framework say yes
just be assured that you're caring is
quite secure especially because of the
adoption of the park today in the market
and then 20 percenters are planning to
use park already in 2015 13 percent is
are currently running spark in
production they are very less so it's
it's really good to us to know then ATT
person is if users choose part to
replace MapReduce and that's already we
discussed and we have seen the
difference seventy percent percentage of
users need
spark for even stream processing that's
also a good point you can see how how
crucial it is to even support many bad
kind of application than 78 percentage
of users need faster processing of
larger datasets so that's quite
intuitive as well than 62% of users load
data in this part with distributed file
system so here are some more empower
interesting topics like Scala is been
adopted by 88% java off by 44% and
python between this is 2% so the
language is being used so so this is
very very interesting slide folks it's
just just go through this slide whenever
you find time you'll definitely get very
good idea so I no need to really like go
through into each one of this
okay so spark use cases different
industries has got different
applications for spark different uses
scenarios for spark however a spark is
used for resolving witness problems like
recommendations then business
intelligence and the fraud detection so
when I say recommendation so basically
spark is being leveraged as a machine
learning tool for analyzing large data
sets so that's one of the key
applications via the machine learning
library package business intelligence
application so basically if you have got
visualization tools for example like say
tableau or say MicroStrategy or any
other bi tool basically can easily
integrate the BI tool with this power
framework who a spark sequel or any
other module fraud detection so again
it's a machine learning specific use
case so here are the adoption rate
across different techno domain aspects
so like others recommendation systems
data warehousing as a data warehousing
to be ITIL log processing fraud
detection and then user facing services
so this is just like a static slideshow
you guys can go through any point in
time who is using spark so here is the
top list of spark users don't worry
these are not the only users there are
plenty of them who are already being
adopted this part or at least playing
der up this park this park is here to
stay for quite a long time so now the
future is coming for spark the future is
awaiting for spark so spark is not one
of those here today gone tomorrow
spark is here to stay for for a
significant amount of time and it is
well worth to get your teeth into it too
in order to get some value out of your
data so this is this is in fact this is
statement this varies
I would actually hand hundred-person
dial I agree with this statement because
with my experience I personally travel
almost throughout the world
I even travel orbit in the impact region
quite quite a number of times so I
personally see many companies are really
trying to adopt this park framework
folks against a park platform is really
growing and here is the sum here is some
information about it and then let's try
to run a quick application on top of
spark so before we actually wrap up our
discussion maybe I'll take this couple
of minutes not more than that so folks
what I'm trying to do basically is okay
so otherwise I hope that said this
session basically gives you very good
significant idea on this park versus the
hadoop mapreduce so again taking another
program would be too heavy for today's
discussion so let devote this time for
any Q&A folks since we have already seen
a practical example between spark and
okay so folks oh so let me just quickly
go through this yeah I think probably I
can take your questions now so then I
can just finish off the remaining just
couple of slides so I have missed few
questions which has been posted
previously so please try to post once
again folks okay so angela is a pretty
key site of experience to go up for
hadoop linux ubuntu knowledge
programming knowledge what is there is
no experience of programming at all so
that's a good question either OC both I
would say Hadoop and then boot spark
really does not require you to you - how
are very good programming skills as long
as at least you were familiar with
sequel skills and the way the courses
has been designed at a direcor
is basically - they let you first female
with the programming and then and then
we'll go about doing programming and
then there are plenty of other modules
like if you know you're talking about
Hadoop there is picking high frame work
which which does not require any
programming skills in spark for example
spark sequel does not require any
programming knowledge however spark
course has been designed in such a way
that will have very fundamental
programming sessions maybe for first two
to three sessions will be dedicated for
programming concepts where we'll be
discussing about the basics of
programming and then we'll get into the
actual sparks discussion which allows
you to quickly add up this park so I
hope your question is too big I hope in
a way I have answered part of it so I
would like to discuss others questions
as well brinjal I hope you understand
that
so then Kansas dust particle run on
theta on HDFS R it takes help
of appreduce now it directly runs the
data on HDFS it has got its own
MapReduce kiddin okay
it does not leverage an Hadoop MapReduce
instead it uses its own MapReduce but it
still uses done framework to run its
MapReduce applications okay
does that answer your question Sam Mills
is since hurry pitched a loop in Java
what about compatibility comfort between
MapReduce code and Scala code very good
question so Scala and Java both are like
sister languages okay so in Scala you
could actually use Java code okay most
of the APS in Java can be used directly
in Scala Samuel which we can discuss in
this session again then what of the
pretty pieces still learn SPARC is
development background necessary I hope
I already answered this question to
Brazil I hope in a way that also super
question 2 so it's not mandated it to
help pre programming languages of
pre-requisite however the course itself
will help you to get familiar with the
programming and then we'll do the
programming in spark and and also we
have got set of modules which does not
at all require any programming
experience so generous is slow loop says
how is spark different from Hadoop of is
it's part different ok so I hope I have
already answered that questions were up
so Hadoop is basically it's coal
capability is on HDFS Erdem distributed
file system and it also has got
MapReduce but spark being the compute
engine you just instead of using Hadoop
MapReduce you will now live raisin spark
MapReduce and in Hadoop for doing other
type of applications you have to
leverage and other applications other
frameworks for example for streaming you
have to go with stomm for instance for
graph processing
you had to go with off so different
different framework see how to learn an
improvement on top of her
but in spark everything is packaged as a
single framework so there is there is
very less learning curve hope the sod I
hope that answer your question
so then satin arena says I am new
learner of spark researchers books so
books yeah I mean probably I can request
the support team to share the books
links to use at the now I'm a hockey
Monza's is Park similar to say P Hana in
memory technology so they are not really
similar as such at least in my point of
view I'm in or I am not very quite
familiar with this ASAP Hana
implementation I know how it works but
I'm not quite familiar with that
underlying implementation details
however that's a commercial tool and
then SPARC is an open-source technology
available for us today more then Alan
says are you going to take this position
yes absolutely
myself and their couple of their few
other faculties as well um okay if you
are interested in mind so probably you
can actually request the support team
they could be able to assist you when
I'm going to start the session okay I
hope you enjoyed today's description as
well then you know it says can I try
spark on my sequel in Windows platform
you can try that absolutely we know that
then this is easily of course a pretty
sight for spark not really it's not
really quite to learn hundred before you
lunge park there stand alone in essence
in in this course I'll give you in fact
some basics of Hadoop as well which
actually help you to distinguish between
her dependence Parker then Alan says
okay then no oh so I have already
answered that
Arden says I have ten years of
experience in Microsoft and want to
switch to Big Data since I will be
fresher what will be the prospects in
Big Data so Aaron if we are asking in
terms of like go whether you are you
your Friedman are to Big Data I would
say definitely know it will fit for even
the freshers especially because Big Data
world has just started even though I
have been working since six years and
big dated of mine actually the traction
towards the Big Data has been increased
quite significantly since last two years
and I see lot of companies are trying to
adapt to Big Data technologies for
various reasons so I'm sure in fact the
other reason other main contributing
factor is basically the amount of data
being generated in the next two decades
will be very very high for example with
the introduction of IOT Internet of
Things which is a major contributor for
data and the amount of data being
generated with the World Wide Web and
they were devices so there are a lot
more problems to be solved and at least
for next two decades will be the driller
it'll be for big data applications so
I'm sure that I hope that answers your
question and then Harry ah yes sure our
PowerPoint will be shared from the
support team for another
Ravi says I think we are already running
out of time maybe another two to three
minutes I'll take regarding your
questions and then probably I hope you
skipped drop decision folks then Ravi
says I'm in process of learning had you
been planning to move to her to do mine
next year as of now I'm learning Pig
framework I am looking at job
descriptions and sparkles Kurla is
showing in many job descriptions
I am a okay I'm basically Microsoft
developer and I thought Java will be new
start for me for Hadoop I want your
advice what language will be good Scala
versus Java for me both are new thank
you
okay so if I try to break down I think
it's a right time for you to move to Big
Data let's try searching for jobs maybe
starting immediately so who knows maybe
you can get into a role may be within
December also so certain times I mean it
is quite possible that people could
recruit even in December so again coming
back to the other question since you are
not from Java background see Ravi I had
to actually write code in C hash maybe
like last two weeks back coincidentally
so I'm from Java background like since
my day one not even before my career has
actually started I did not see much
difference between C hash and Java it's
true that it's quite similar at least a
vaguely I can answer that that both of
them looks alike to me
and even when I write applications in
other languages even I write programs in
Python I feel like as long as you're
comfortable in logical thinking language
is just a MIDI the image immaterial for
you you just have to understand the
syntax that's it but again coming back
to your specific question C hash in Java
looks pretty much alike and I don't find
difficulties in adopting to these two
technologies so I I'm sure that you
wouldn't be able to quickly learn Java
if you're from C hash development
background and on the other hand Scala
versus Java so since you are already
learning hadoop just continue with Java
just get some hands-on experience in
Java if you come for SPARC it is my
responsibility to make you familiar with
the Scala okay
Robbie doesn't answer the questions then
Theseus okay so I want to start my
Korean Analytics I'm new in this in this
in this in this industry so please
adjust me shall I proceed learning our
language our analytics or shall I start
learning scholar
okay so analytics so what do you mean my
analytics so there are different aspects
to it if you are talking in terms of
sequel a sequel blaz'd analytics see you
can actually go to spark sequel and if
you are good with sequel background and
that's fine if you want to do most
complex like complex analytics where
mathematical modeling and all
statistical analysis then is you have to
learn scholar if at all you want to
leverage and spark so spark has got
Emily package which allows you to run
most complex algorithms so there are
plenty of algorithms around there which
are I mean the most popular algorithms
are already available in sin spark so
which you could be able to quickly learn
it's it's not a big deal and then R is R
is like I don't want to really compare
our with any other language because R is
one of my favorite static statistical
tools and the maturity of R is quite
quite higher than any other open source
technology at least in my perspective in
big dated of mine so R is a different
story altogether say if you really want
to become a madman a statistician I
would suggest you to go continue with
our and then learn spark as well if you
are interested in running models and
larger assets okay
and then shirisha's which programming
language would be used in session is
color gibberish
and skylights a more prominent language
for implementing spark code okay so I
think we are already out of time maybe
I'll take a couple of questions so my
apologies if I couldn't take others
questions I hope you guys understand
because of the limited time we have
caught and then there are worth they
were around like 66 folks on the call I
hope you understand that Sam Mills is do
we need to complete the Hadoop admin and
Hadoop developer course before taking
this power course no it's not required
so a middle spark is a standalone
framework I can
I mean if it's it's our responsibility
to basically get you equipped with spark
directly so it's not hard it is not
mandatory
Danny Shui says I have a 16 gig ram and
I want to process 400 GB data house part
can do that in memory processing that's
a good question to answer a question of
quickly quick and dirty way it's
basically goes in hi creations Manish
were so most of the real-world
applications are like that not only
years so we're basically sperm goes in I
Croatian so it actually does this disk
reads and processes the data depending
on the amount of memory at how okay so I
hope that's pretty much it for today so
folks I hope you guys have enjoyed
today's discussion so here are just a
couple of slides so just references just
go through this links and then here is
the survey feedback see what's over
feedback is quite essential for us to
rectify ourselves our to adopt do our to
find him a specific to your needs
okay so then course details so folks
here at the coast details and probably
the upcoming batches are will be
starting quite soon
so get in touch with the support and
they should be able to assist you
support him you could reach out at spark
at a naked Ezio
our support at any record our CEO so
thanks for joining today's discussion
folks thanks for your valuable time to
join the webinar so have a good night
and have a nice day bye to all of you