Apache Spark Tutorial | Apache Scala Tutorial | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

hello and welcome to the introductory
lesson of the Apache spark in Scala
course offered by simply learning this
lesson will give you an overview of the
course its prerequisites and the value
it will offer you after completing this
course you will be able to explain the
process to install spark describe the
features of Scala discuss how to use RDD
for creating applications and spark
explain how to run SQL queries using
spark SQL discuss the features of spark
streaming explain the features of sparks
ml programming and describe the features
of graph X programming
via practice
work in Scala training course offered by
simply learner provides details on the
fundamentals of real-time analytics and
need of distributed computing platform
it will also explain Scala and its
features further it will enhance your
knowledge on the architecture of Apache
spark the course will also explain the
process of installation and running
applications using Apache spark further
it will enhance your knowledge on
performing SQL streaming and batch
processing finally it will explain
machine learning and the graph analytics
on the Hadoop data
the course is aimed at professionals
aspiring for a career in growing and
demanding fields of real time big data
analytics analytics professionals
research professionals IT developers
testers data analysts data scientists bi
and reporting professionals and project
managers are the key beneficiaries of
this course other aspiring sin students
who wish to gain a thorough
understanding of Apache spark can also
benefit from this course
fundamental knowledge of any programming
language is a prerequisite for the
course participants are expected to have
basic understanding of any database SQL
and query language for databases working
knowledge of Linux or UNIX based systems
is an added advantage for this course
although it is not mandatory let's
understand the value of Apache spark in
Scala to the professionals with the help
of this case study of Allah which is a
video services company that's based in
San Francisco and offers services to
various media organizations including
ESPN Bloomberg Yahoo Japan and a
Telegraph Media Group it offers
actionable analytics that allows them
analyzing the manner in which their
video content is being consumed in great
detail it's also enabling them
optimizing their delivery standards for
maximizing their revenues the analytics
engine processes of the organization
achieve more than 2 billion analytics
events every day these analytics are
obtained from about 200 million viewers
across the world watching videos on an
oil-powered player
for Allah was to deliver intelligence in
real time by mining the stored data the
data included personalized content
recommendations to clients and the
viewing patterns in addition the
organization needed to run real-time
interactive SQL query and MapReduce
master
this issue has been resolved by using
spark ayala can now perform real-time
analytics using it it has made a leading
organization of big data technology the
organization now has the power to expand
rapidly and offer real-time intelligence
that are based on huge datasets
there are seven core lessons in this
course apart from the current lesson
course overview I take a look at the
course map displayed on the screen in
addition to the lessons there are some
demos provided in the course to
facilitate a better understanding of the
concepts
concludes the course overview the next
lesson is introduction to spark all the
best
hello and welcome to lesson one of the
Apache spark in Scala course offered by
simply lending this lesson we'll explain
the need features and benefits of spark
it will also compare spark with the
traditional Hadoop ecosystem
this lesson you'll be able to describe
the limitations of MapReduce in Hadoop
compare batch versus real-time analytics
describe the application of stream
processing and end memory processing
explain the features and the benefits of
SPARC explain how to install SPARC as a
standalone user and compare SPARC versus
Hadoop ecosystem
we need a new generation of distributed
systems let's understand why nowadays
it's very rare that an organization
exists that depends on its centralized
computing irrespective of the fact there
are still many organizations that keep a
tight hold on their internal data center
and avoid any absolutely required data
distribution this sometimes happens
because of their heavy investments in
the infrastructure data centralization
isn't prevalent nowadays because of
various reasons one of those reasons is
a variety of client devices the number
and variety of these devices is
increasing each year leading to a
complex array of endpoints to be served
another reason is social mobile and
embedded technology as the amount and
variety of the collected data is
increasing exponentially also landscape
transformation and latency reduction is
causing data centralization to decrease
with a few exceptions like high
frequency trading or hft in which
physically locating servers in a single
location can lower latency leveraging
the distributed computing technology
with parallel processing techniques
transform the landscape and reduce
latency
MapReduce used in Hadoop this is not
suitable for many reasons such as it's
not a good choice when it comes to
real-time processing its bachelor
oriented because of which it's executed
as periodic jobs that take time to
process the data and provide results it
takes minutes to complete a job which
mainly depends on the data amount and
number of nodes in the cluster MapReduce
is also not suitable for writing trivial
operations such as filter and joined to
write such operations you might need to
rewrite the jobs using the MapReduce
framework which becomes complex because
of the key/value pattern this pattern is
required to be followed in reducer and
mapper codes in addition MapReduce
doesn't work well with large data on a
network it works on the data locality
principle and hence works well on the
node where the data actually resides
however it's not a good option when you
need to process a lot of data requiring
shuffling over the network the reason is
that it'll take a lot of time to copy
the data which may cause bandwidth
issues
MapReduce is also unsuitable with OLTP
that includes a large number of short
transactions since it works on the batch
oriented framework
it lacks latency of seconds or sub
seconds another limitation exists with
name node that tracks the metadata of
about 600 bytes per file as estimated by
Yahoo this means that in the case of too
many files there can be a problem with
name node
additionally MapReduce is unfit for
processing graphs graphs represent the
structure to explore relationships
between various points for example
finding common friends in social media
like Facebook Hadoop has Apache giraffe
library for such cases however on top of
MapReduce it adds to complexity another
important limitation is its
unsuitability for iterative execution of
programs some use cases like k-means
need such execution where data needs to
be processed again and again for
refining results MapReduce being a
stateless execution runs from the start
every time the features listed on the
screen show a comparison of batch and
real-time processing in case of the
enterprise to use cases in case of batch
processing a large amount of data or
transactions is processed in a single
run over a time period the Associated
jobs generally run entirely without any
manual intervention additionally the
entire data is pre-selected and fed
using command-line parameters and
scripts and typical cases it's used to
execute multiple operations handle heavy
data load reporting and offline data
workflow an example is to generate daily
or hourly reports for the purpose of
decision-making on the other hand
real-time processing takes place upon
data entry or command receipt
instantaneously it needs to execute on
response time within stringent
constraints an example missed fraud
detection note that Hadoop has different
subsystems like pre gal Grif s4 and
drill for a different business use case
it would be better to have just one
processing framework to solve all these
use cases
stream processing fits well for
applications showing three
characteristics let's first talk about
computer intensity which is defined as
the number of arithmetic operations per
global memory or input-output reference
today in various signal processing
applications this intensity is well
above 50 ratio 1 also it's increasing
with the complexity of algorithms the
next feature is data parallelism that
exists in a kernel when a function is
applied to an input streams records and
multiple records can be processed
simultaneously
this should happen without results
waiting from the previous records data
locality is the third feature that is a
particular type of temporal locality and
is general in media and signal
processing applications in which data is
produced in read once or twice and then
never again read intermediate streams
can capture this locality directly these
are the streams that are passed between
kernels and the data within kernel
functions and they do it using the model
of stream processing programming with
column centric databases coming the
similar information can be stored
together and hence data can be stored
with more compression and efficiency it
also permitted to store large data
amounts in the same space which thereby
reduce the memory amount required for
performing the query and also increase
the speed of processing in an in-memory
database the entire information is
loaded into memory eliminating the need
for indexes aggregates optimize the
databases star schemas and cubes with
the use of in-memory tools
compression algorithms can be
implemented that thereby decreased the
in-memory size even beyond what is
required for hard disks users querying
the data loaded into the memory is
different from cashing this also helps
to avoid performance bottlenecks and
slow database access caching is a
popular method for speeding up the
performance of a query where it caches
our subsets of very particular organized
data that are defined already within
memory tools the analysis of data can be
flexible in size and can be accessed
within seconds by concurrent users with
an excellent analytics potential
this is possible as data lies completely
in memory in theoretical terms this
leads to data access improvement that is
10,000 to 1 million times faster as
compared to a disk in addition it also
reduces the performance tuning need by
the IT folks and hence provides faster
data access for end-users within memory
processing it's also possible to access
visually rich dashboards and existing
data sources this ability is provided by
several vendors in turn this allows
end-users and the business analytics to
create customized queries and reports
without any need of extensive expertise
or training
catchy spark is an open source cluster
computing framework that was initially
developed at UC Berkeley in the amp lab
as compared to the disk-based
two-stage MapReduce of Hadoop spark
provides up to 100 times faster
performance for a few applications with
in-memory primitives this makes it
suitable for machine learning algorithms
as it allows programs to load data into
the memory of a cluster and query the
data constantly as shown on the screen a
spark project contains various
components such as spark core and
resilient distributed data sets or rdd's
spark sequel spark streaming machine
learning library or ml library and
graphics
the components of a spark project are
listed on the screen the first component
is spark core and rdd's which is the
foundation of the entire project it
provides basic input/output
functionalities distributed tasks
dispatching and scheduling rdd's is the
basic programming abstraction and is a
collection of data that is partitioned
across machines logically these can be
created by applying coarse-grained
transformations on the existing rdd's or
by referencing external datasets the
examples of these transformations are
reduced join filter and vamp the
abstraction of rdd's has exposed
similarly as in process and local
collections through a language
integrated api in Python Java and Scala
as a result the complexity of
programming is simplified as the manner
in which applications change rdd's is
similar to changing local data
collections spark sequel is a component
lying on the top of spark core it
introduces schema RTD which is a new
data abstraction and supports semi
structured and structured data this
abstraction can be manipulated in Java
scalar and Python by the spark sequel
provided a domain-specific language in
addition spark sequel supports sequel
with ODBC JDBC server and command-line
interfaces the next component spark
streaming leverages the fast scheduling
capability of spark core for streaming
analytics ingest data in small batches
and performs RTD transformations on them
with this design the same application
code set that is written for batch
analytics can be used on a single engine
for streaming analytics machine learning
library lies on the top of spark and is
a distributed machine learning framework
with its memory based architecture it's
nine times faster than the Apache
Macoutes Hadoop disk based version in
addition the library performs even
better than the valve hal wabbit in
addition it applies in various common
statistical and machine learning
algorithms the last component graphics
also lies on the top of SPARC and as a
distributed graph processing framework
for the computation of graphs it
provides an API and an optimized run
time for the pre glow abstraction the
API can also model this abstraction as
discussed spark was started at UC
Berkeley ant lab by Mattea haria in the
year 2009 it was in 2010 when it was
open sourced under a BSD license the
project was then donated to the Apache
Software Foundation and the license was
changed to Apache 2.0 in the year 2013
in the month of February 2014 spark
became an Apache top-level project then
in November of the same year it was used
by the engineering team at data bricks
to set a world record and large-scale
sorting now data bricks provides
commercial support and they provide
certification for it at present spark
exists as a next-generation real-time
and batch processing framework the
graphic on the screen also displays the
journey of spark
we have already discussed that spark
provides performance which in turn
provides developers and experience that
they won't forget easily spark is
considered over MapReduce mainly for its
performance advantages and versatility
apart from this another critical
advantage is its development experience
language flexibility is another
important benefit that we will discuss
here spark provides support to various
development languages like Java Scala
and Python and will likely support our
in addition that has the capability to
define functions in line with the
temporary exception of Java a common
element in these languages is that they
provide methods for expressing
operations using lambda functions and
closures using closures you can use the
application core logic to define the
functions inline which helps to create
easy to comprehend code and preserve
application flow the components of SPARC
execution architecture are listed on the
screen
it has SPARC submit script that is used
to launch applications on a spark
cluster it can use all cluster managers
supported by SPARC using an even
interface due to this it's not required
to configure your application for each
one particularly the next component is
SPARC applications these applications
run as sets of processes independently
on a spark cluster these are coordinated
by the SPARC context object in the
driver program which is your main
program SPARC context can connect to
different cluster managers which are
three types standalone Apache esos and
Hadoop yarn a standalone cluster manager
is a simple one that makes setting up a
cluster easy Apache meso this is a
general cluster manager that is also
capable of running service applications
and MapReduce on the other hand group
yarn is the resource manager in Hadoop -
the last component ec2 launch scripts
makes launching a standalone cluster
easy on Amazon ec2 the interaction of
these components is shown in the diagram
displayed on the screen
let's now talk about the next feature of
spark automatic parallelization of
complex flows it's your task to make the
sequence of MapReduce jobs parallel in
case of a complex pipeline here a
scheduler tool like Apache Ozzy is
generally required for constructing this
sequence carefully using spark the
series of individual tasks is expressed
in terms of a single program flow to
give a whole picture of the execution
graph to the system this flow is lazily
evaluated using this approach the course
scheduler can map the dependencies lying
between various application stages
correctly
this allows parallel adding the
operators flow automatically without any
intervention with this capability you
can also achieve a few optimizations to
the engine with less burden an example
of such a job is given on the screen
the screen also shows how this
parallelization works through the given
diagram
an important point about this structure
is that every application has its own
executor processes which run tasks in
various threads and stay to the duration
of the entire application while it
benefits in terms of scheduling and
executor sides by separating
applications it also implies that
without writing to an external storage
system you cannot share data across
applications of SPARC another feature is
the agnostic behavior of SPARC to the
cluster manager underlying till SPARC
can obtain executive processes and these
can connect its comparatively relaxed to
run it even on a cluster manager
supporting other applications such as
yarn and mesos note that the driver
program needs to listen and accept
connections coming from its executor all
the time in other words the driver
program must be accessible to the
network to be addressed by the worker
nodes the driver schedules the tasks on
the cluster therefore it should run in
proximity to the worker nodes if
possible on the same local network to
send remote requests to the cluster you
should open an RPC to the driver and let
it submit operations from neighborhood
this is better than running a driver far
through the worker nodes
as a developer
with MapReduce you generally get forced
to combine basic operations to make them
customer mapper reducer jobs
this happens because there is no
built-in feature that could streamline
this process therefore some developers
turn to the higher-level api's for
writing their MapReduce jobs these api's
are provided by frameworks such as cash
skating and Apache crunch on the other
hand spark provides a powerful and
ever-growing operator's library these
api's contain functions for the
operators listed on the screen these are
just a few examples there are above 80
operators available in spark while a few
of them provide you operations that are
equivalent to MapReduce operations
the others are high-level and allow you
to write much more precisely
note that when scripting frameworks such
as Apache Pig many high-level operators
are also available SPARC lets you access
them in the full programming language
context as a result you can use
functions classes and control statements
as in a typical environment of
programming
when it comes to speed spark has
extended the MapReduce model to support
computations like stream processing and
interactive queries the feature of speed
is critical to process large data sets
as this implies the difference of
waiting for hours or minutes and
exploring the data interactively spark
supports running computations in memory
also the related system is more
effective as compared to MapReduce when
it comes to running complex applications
on a disk these features add to the
speed capability of spark spark covers
various workloads that used to require
different distributed systems such as
streaming iterative algorithms and batch
applications as these workloads are
supported on the same engine combining
different processing types is easy
it's normally required in production
data analysis pipelines the combination
feature also allows easy management of
separate tools SPARC is capable of
creating distributed datasets from any
file that is stored in the Hadoop
distributed file system or HDFS or any
other supported storage systems you
should note that SPARC doesn't need
Hadoop it just supports the storage
systems that implement the api's of
hadoop and supports sequence files
parkette AVO txt files and all other
input output formats of hadoop now the
question is why unification matters the
unification not only provides developers
the ease of learning only one platform
but also allows users to take their apps
everywhere the graphic shows the apps
and systems that can be combined in
spark
a spark project includes various closely
integrated components for distributing
scheduling and monitoring applications
with many computational tasks across a
computing cluster or various worker
machines the sparks core engine is
general-purpose and fast as a result it
empowers various higher-level components
that are specialized for different
workloads like machine learning or
sequel these components can interoperate
closely another important benefit is
that it integrates tightly allowing to
create applications that easily combine
different processing models for example
ability to write an application using
machine learning to categorize data in
real time as it is ingested from sources
of streaming additionally it allows
analysts to query the data that's
resulted through sequel
moreover data scientists and engineers
can access the same data through the
Python shell for ad hoc analysis and in
standalone batch applications for all
this the IT team needs to maintain just
one system the
deployment modes of spark are explained
on the screen the standalone mode is a
simple one that can be launched manually
by using launch scripts or starting a
master and workers this mode is usually
used for development and testing spark
can also be run on the hardware clusters
that are managed by mesos running spark
in this mode has advantages like
scalable partitioning among different
spark instances and dynamic partitioning
between spark and other frameworks
running spark on yarn has all parallel
processing and all benefits of the
Hadoop cluster by running spark on ec2
you have key value pair benefits of
Amazon
the configuration options that can be
passed to the master and worker are
listed in the given table on the screen
this demo will show the steps to install
Apache spark on a Linux machine
in this demo you'll learn to install
Apache spark on a Linux machine as the
first step download the latest spark tar
file once the spark binary is downloaded
move the tgz file to the home directory
now on target by executing the given
command
you
now you need to add the spark
installation path in the bashrc file so
that you can run spark from anywhere in
Linux add the given lines in the bashrc
file by executing the given command
note that your bashrc file location
might be different based on your user
name and profile
you
now let's reload the environment
variable by executing the given command
type echo spark home to verify whether
the spark home environment variable has
been correctly defined or not go to the
s bin directory of the spark install
directory and type the given command to
start master and worker Damons process
you
the steps listed on the screen depict
house park applications run on a cluster
these applications run independently as
different processes sets on the cluster
the spark context object that lies in
your main program coordinates the
process for running on the cluster this
object can connect to different types of
cluster managers we have already
discussed the different types of cluster
managers this allocates resources across
applications once the system is
connected spark requires executor zon
nodes in the cluster executor czar the
processes that store data for
applications and run computations then
it sends the application code to
executives at the end spark context
sends tasks for executors to run
one of the tasks that is performed on a
spark cluster is submitting applications
of any type to a cluster this is done
using the spark submit script another
task is monitoring every driver program
has a web-based UI that usually lies on
port 4 0 4 0 it shows information about
the storage usage executor Zand running
tasks to access the same go to the given
URL you can also schedule jobs on the
cluster as spark provides control over
resource allocation both across and
within applications companies like NTT
DATA
Yahoo Groupon NASA Nokia and more are
using spark for creating applications
for different use cases these use cases
are stream processing of networked
machine data performing big data
analytics for subscriber personalization
and profile in the telecommunications
domain and executing the big content
platform which is a b2b content asset
management service that provides an
aggregated and searchable source of
public domain media live news feeds and
archives of content a few more use cases
are building data intelligence and
e-commerce solutions in the retail
industry and analyzing and visualizing
patterns and large-scale recordings of
brain activities
the Hadoop ecosystem allow storing large
files on various machines it uses
MapReduce for batch analytics that is
easy as it is distributed in nature in
Hadoop third party support is also
available for example by using ETL
talent tools various bachelor oriented
workflows can be designed in addition
that supports pig or hive queries that
non Java developers can use and prepare
batch workflows using sequel scripts on
the other hand apache supports both
real-time and batch processing you can
perform every type of data processing
using spark that you execute in Hadoop
for batch processing spark batch can be
used over Hadoop MapReduce for structure
to data analysis spark sequel can be
used using sequel for machine learning
analysis machine learning library can be
used for clustering recommendation and
classification for interactive sequel
analysis spark sequel can be used over
stringer Tezz and Impala in addition for
real-time streaming data analysis spark
streaming can be used over specialized
library like storm a summary of this
comparison is also given in the table
shown on the screen
let's summarize
topics covered in this lesson data
centralization is becoming less popular
because of various reasons such as
variety of client devices MapReduce and
Hadoop has many limitations such as it's
unsuitable for real-time processing
apache spark is an open source cluster
computing framework the components of a
spark project are spark core and rdd's
spark sequel spark streaming ml lib and
the graphics spark is popular for its
performance benefits over MapReduce
another important benefit is language
flexibility
mark execution architecture are sparks
submit script spark applications or
context cluster managers and ec2 launch
scripts the different advantages of
spark or speed combination unification
and Hadoop support the different
deployment modes of spark are standalone
onme sews on yarn and on ec2 companies
like NTT DATA Yahoo group on NASA Nokia
and more are using spark for creating
applications for different use cases you
can perform every type of data
processing using spark that you execute
in Hadoop
with this we come to the end of the
lesson one introduction to spark of the
Apache spark and Scala course the next
lesson is introduction to programming in
Scala hello and welcome to lesson two of
the Apache spark and Scala course
offered by simply learn this lesson
we'll explain the basic concepts of
programming in Scala
after completing this lesson you'll be
able to explain the features of Scala a
list of the basic data types and
literals used in Scala lists the
operators and methods used in Scala and
discuss a few concepts of Scala
Scala is a modern and multi-paradigm
programming language it's been designed
for expressing general programming
patterns in an elegant precise and
typesafe way one of the prime features
is that it integrates the features of
both object-oriented and functional
languages smoothly it's a pure
object-oriented language as every value
in it is an object the objects behavior
and types are explained through traits
and classes
it's also a functional language as every
function in it is a value by providing a
lightweight syntax to define anonymous
functions and provide support for
higher-order functions in addition the
language also allows functions to be
nested and provide support for currying
it also has features like case classes
and pattern matching model algebraic
types support in addition Scala is
statically typed being empowered with an
expressive type system the system
enforces the use of abstractions in a
coherent and safe way to be particular
this system supports various features
like annotations classes views
polymorphic methods compound types
explicitly type self references and
upper and lower type bounds when it
comes to developing domain-specific
applications
it generally needs domain-specific
language extensions scalloped being
extensible provides an exceptional
combination of language mechanisms due
to this it becomes easy to add new
language constructs as libraries scalo
supports the same data type as Java does
with the same precision and memory
footprint the table on the screen lists
and explains all the basic data types
used in Scala
these are byte short integer long float
double character string unit null
boolean any-any reference and nothing
the rules used by Scala for literals are
easy and intuitive let's first discuss
the integer and floating type literals
integer literals are of integer long
short and byte types they are available
in two forms decimal and hexadecimal an
example is given on the screen on the
other hand floating-point literals
contain decimal digits they may
optionally have a decimal point and may
optionally be followed by an E and an
exponent a few examples for defining
such literals are given on the screen
other types of literals are character
literals and string literals character
literals contain any Unicode character
between single quotes they are special
character literal escape sequences an
example is given on the screen
on the contrary string literals contain
characters surrounded by double quotes a
few examples to define these literals
are given on the screen
type of literals as a billion type that
has literals true and false a few
examples for defining these literals are
given on the screen one more type is
symbol literals which are written with
idents an ident can be any alphanumeric
identifier an example of it is given on
the screen
an operator is a symbol that conveys to
the compiler that it needs to perform
specific logical or mathematical
manipulations actually these operators
provide a nice syntax for general method
calls for instance consider the example
given on the screen
in this example the class integer
includes a method called plus that takes
an integer and provides an integer as a
result to invoke the plus method you
need to add two integers as depicted on
the screen
this demo will show the steps to declare
and use basic literals and the
arithmetic operator in Scala
let's first learn how to use basic
literals in Scala here we are going to
declare and use literals such as long
float character and string for this
we'll use the print in statement
we will declare the long variable based
on the value of the variable Scala
automatically detects the datatype to
define the float variable you need to
use the character F at the end of the
variable value doing so will let Scala
make this variable as float otherwise it
will be treated as a long variable if a
variables value contains only one
character then the Scala treats that
variable as the char datatype if a
variables value contains more than one
character then Scala treats that
variable as the string datatype
let's see how to print a string in the
console if a string contains double
quotes then we need to use double quotes
as an escape character
you
now let's understand how to declare and
use the boolean literal and use
different types of arithmetic operators
in Scala
Scala automatically detects a boolean
data type if the value is either true or
false let's now look at the use of
logical operators like addition
multiplication division and subtraction
Scala automatically selects the datatype
for the result
you
if you specifically want to specify your
data type as the long data type you need
to use the L character
you
this demo will show the steps to the
Clarion used a logical operator in Scala
in this demo you'll learn how to use the
logical operator in Scala here we're
going to use different logical operators
such as equal greater than and less than
in Scala
you can perform the bitwise operation in
Scala using the ampersand character
to perform an equality check you can use
the double equal to character
you can also use other logical operators
such as greater than greater than equal
to less than and less than equal to
you
many logical operators can be combined
together in a statement to
you
scallop provides a built-in mechanism
called type inference using which you
can omit certain type annotations
therefore for example it's generally not
required to specify the variable type as
the compiler can deduce it from the
initialization expression of the
variable in addition you can also omit
return types of methods as they
correspond to the body type and can be
inferred by the compiler an example to
define type inference is given on the
screen
for recursive methods the compiler can't
infer a result type the program example
shown on the screen will fail the
compiler for the same reason
when polymorphic methods are called or
generic classes are instantiated it's
not required to specify type parameters
the reason is that the compiler of Scala
will infer these missing type parameters
from the actual method or constructor
parameters and the context the example
given on the screen illustrates the same
concept in this program the last two
lines are equivalent to the code given
on the screen we're all infer two types
have been made explicit
in some scenarios using type inference
can become dangerous for example
consider the program given on the screen
this program does not compile as the
type inferred for variable object is
null because the only value of that type
is null
you can't make this variable refer to
another value
to understand the concepts of a mutable
and immutable collection let's compare
them through the table given on the
screen mutable implies that it's
possible to modify the collection and
concern
therefore to change a collection a and
every other reference to that collection
you can append elements using the given
operator on the other hand immutable
implies that the given collection object
never changes you would need to build
new collection objects using the given
operators this helps in concurrent
algorithm because it doesn't need
locking to add anything to a collection
however this feature can prove to be
very useful
it may come at the cost of some overhead
as compared to mutable collections
immutable collections are fully
persistent data structures the
difference between these collections is
very similar to what exists between
variable and value a mutable collection
that is bound to a value can be altered
however you cannot reassign the value on
the other hand an immutable collection
cannot be altered however if it is
assigned to a variable that variable can
be assigned to a collection built from
it using operations like +
in Scala a function is a first-class
value therefore like other values these
can be returned as a result or passed as
a parameter an example to define such
functions is given on the screen
note that higher-order functions are the
ones that take other function that
parameters or return them as results
an anonymous function is an alternative
of named function definitions for small
argument functions that tend to get
created by parameterization by functions
it's an expression that evaluates to a
function these are defined without
giving them any name an example of such
a function is given on the screen
an object is a named instance with
members like methods and fields in Scala
an object and its class have the same
name for instance in this example the
first line has the keyword object which
is being used to declare a new object
it's being followed by the name of the
object in the second line of value X is
being assigned to the object instance
note that the members of objects are
similar to those of classes and let's
now discuss some of the uses of objects
one of the primary uses to contain the
methods and fields that are independent
from any environment an example is math
this has the object name in the standard
library containing multiple fields and
methods depending only on arguments if
any provided to them objects are also
used to create instances of classes an
example is given on the screen
note that in Scala there are singleton
objects instead of static members which
is a class with only one instance and
can be created using the keyword object
in this way Scala is more object
oriented than Java
scowler does not include built-in
rational numbers however you can define
them using a class a class acts like a
blueprint of objects once a class is
defined objects can be created from this
blueprint using the keyword new for
instance in this example two variables x
and y and the method are being defined
the method isn't returning any value the
variables of a class are called fields
of the class and methods are termed as
class methods the name of the class acts
as a class constructor that can take
multiple parameters
this demo will show the steps to define
and use type inference functions
anonymous function and class in Scala
in this demo you'll learn how to define
and use type inference functions
anonymous function in class first we'll
learn how Scala can infer the type of
variable based on the type of operations
and data
you
now let's see how to define a
and Scala here the function name is some
integers which takes an integer
parameter and returns an integer value
you
in scala small argument functions can be
used as anonymous functions let's see
how
you
now let's see how to define a class and
implement some methods inside it
you
classes and Scala are static templates
that can be instantiated into many
objects at runtime here we are declaring
a class which takes two integer
variables as parameters and has a method
called addition which will increment
those two integers by one and print them
in the console
you
before you understand traits let's first
discuss the reason of its origin in
languages like Python and C++ you don't
require interface says they have
multiple inheritance which is subject to
the diamond issue what will happen if
you define a class inheriting two
methods with the same type signature
from two different classes an interface
is basically a set of properties and
methods that an implementing class needs
to have for avoiding the issues of
multiple inheritance the Java designers
instead decided to have single
inheritance with interfaces to make the
system a little flexible Scala exists
halfway between full multiple
inheritance and Java's single
inheritance with interface model and it
has traits traits allow you to recreate
interfaces as Java does and also lets
you go further using traits you can
recreate the interfaces and
implementation model of Java
the given example to create traits in
Scala the code below shows how you can
implement the student trait
skala includes a rich set of collection
library which are containers of things
these can be sequenced as linear sets of
items they may be one elements bound to
zero or have an arbitrary number they
can be lazy or strict too lazy
collections contain elements that may
not require memory until access for
example ranges in addition collections
can be mutable or immutable immutable
collections can contain immutable items
you should note that mutable and
immutable collections work better in
different problems in doubt you should
start with an immutable collection and
change it later if required
the most commonly used collections type
is lists list X as a linked list of type
X an example is given on the screen
another type is sets which is a
collection of pairwise various elements
of the same type an example is displayed
skala maps represent a collection of key
or value pairs consider the given
example
Skala two bowls can hold objects with
different types like an array or a list
an example of the same is given in
Scalla options option X gives a
container for 1 or 0 elements of a given
type consider the given example the most
direct method to diagnose all the
elements that are returned by an
iterator is by using a while loop as
explained through the given example
all operations on lists can be expressed
using the given methods a head is a
method that returns the first element of
a list the tail method returns a list
with all elements except the first the
third method is empty returns true if
the list is empty otherwise false the
example given on the screen shows the
use of these methods when this code is
compiled and executed it gives the
output shown
this demo will show the steps to use
different operations on the list data
structure
in this demo you'll learn how to use
different types of data structures such
as lists set map tuple option and object
in Scala
you
to define a list use the list keyword
and past elements as parameters
similarly you can use set and map see
the syntax used for defining a map
you
let's Scala tuple combines a fixed
number of items together so that they
can be passed around as a whole unlike
an array or a list a tuple can hold
objects with different types but they
are also immutable here's an example of
a tuple holding an integer and a string
an option is a container for zero or one
element of a given type an option can be
either some P or none object which
represents a missing value
let's create an object test that will
have a main method to print characters a
2d
you
as discussed maps represent a collection
of key or value pairs you can retrieve
any value based on the Scala map key
these keys are unique however values
need not be unique also called hash
tables maps are of two types the
immutable and the mutable Scalla uses
the immutable maps by default if you
need to use the mutable Maps you'll need
to import scalloped collection mutable
map class explicitly to use both types
of maps in the same map you can continue
referring to the immutable map is map
but you can refer to the mutable set as
mutable map the example given on the
screen shows how to declare immutable
Maps
pattern-matching is supported by Scala
for processing messages this feature
allows you to match on any sort of data
using first match policy this built-in
mechanism is the second most widely used
feature in Scala after closures and
function values the example given on the
screen depicts how to match against an
integer value the case statements block
declares a function mapping integers to
strings the keyword match gives an easy
way to apply a function to an object
a method containing
listed parameters are applicable to
arguments like normal methods this is
because the implicit label has no impact
it's the method misses arguments for its
implicit parameters those are provided
automatically the actual arguments can
be of two types first these are all
identifiers zwei that are accessible at
the method call point without any prefix
and denoting an implicit definition or
parameter and second these are all
members of companion modules belonging
to the implicit type of parameters
labeled implicit
in the example given on the screen a
method sum has been defined that is
computing the sum of a list of elements
through unit and add operations note
that the implicit value cannot exist at
the top they need to be members of a
template
the stream class executes lazy lists in
which elements are evaluated only when
required an example is given on the
screen
note that in Scala a stream is a lists
with its tail as a lazy value a value
remains computed and is reused in other
words the values are cached
this demo will show the steps to use
different types of data structures in
Scala
in this demo you'll learn how to perform
different operations on the list data
structure by using head tail and is
empty methods
you
here is a class for immutable linked
lists representing ordered collections
of elements of type this class comes
with to implementing case classes Scala
nil and Scala double colons that
implement the abstract members is empty
head and tail this class is optimal for
last in first out or LIFO stack like
access patterns if you need another
access pattern for example random access
or fi fo consider using a collection
more suited to this than a list
you
let's summarize the topics covered in
this lesson
skala is a modern and multi-paradigm
programming language supporting the same
data types as java des the basic
literals used in scala are integer
floating type character string boolean
and symbol Scalla provides various
operators like arithmetic relational and
logical bitwise and objective quality
type inference is a mechanism that
allows to emit certain imitations and
return types of methods a function is a
first-class value
print of objects Scala has traits that
allow to recreate interfaces and
implementation model of Java collections
are containers of things you can
retrieve any value based on the scale
map key which is unique but not its
values pattern matching allows to
process messages
a method containing implicit parameters
are applicable like normal methods the
stream class executes lazy lists in
which elements are evaluated only when
required
with this we come to the end of lesson 2
introduction to programming in Scala of
the Apache spark and Scala course the
next lesson is using RDD for creating
applications in spark