Introduction To Apache Storm Certification Training | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

hello and welcome to the introductory
lesson of the Apache storm developer
course offered by simply learn in this
lesson you will be introduced to this
course and the various aspects
supporting Apache storm let us begin by
exploring the objectives of the course
by the end of this course you will be
able to describe the need for big data
describe the fundamental concepts of
storm describe the architecture and data
model of storm install and configure
storm interface with storm describe
tried nth extension to storm and use
storm for real-time data processing in
the next screen we will focus on the
course overview this training course
provides big data fundamentals gives an
overview of real-time big data storm use
cases and components storm architecture
and data model storm installation
configuration interfaces to storm and
trident on top of storm let us now go
through the target audience for this
course the target audience for this
course includes professionals aspiring
for a career in big data analytics
professionals research professionals IT
developers testers project managers etc
it also includes those looking for a
change in their career this course also
has high value for aspiring students we
will go through the prerequisites for
this course in the next screen the
prerequisites for this course include
knowledge of any computing system and a
good knowledge of Java some knowledge of
Linux or UNIX based systems is desired
following lessons would be covered in
this course the table lists the lesson
numbers along with the name of the
lesson covered let us start this
exciting journey by introducing you to
Big Data this concludes introductory
lesson in the next lesson we will
introduce introduction to Big Data hello
and welcome to the first lesson of the
Apache storm course offered by simply
learn this lesson will provide you an
introduction to big data further it will
introduce you to the real-time Big Data
concept let us explore the objectives of
this lesson on the next screen by the
end of this lesson you will be able to
describe the concept of big data and its
three V's talk about the different sizes
of
Datta describe some use cases for big
data explain the concept of Apache
Hadoop and real time big data processing
and describe some tools for real-time
processing of big data on the next
screen we will discuss the concept of
big data
Digital data has exploded over the last
two to three years Facebook Twitter
YouTube and sensory networks have been a
few of the major contributors to the
huge growth of data data is growing at a
very rapid pace
data volumes are in terms of millions of
gigabytes the technology has evolved to
handle this amount of data the
technology to store and process large
volumes of data the technology to
analyze and make decisions based on the
huge volumes of data have all evolved
the data as well as technology to
utilize this data represents big data
next we will explore the three V's of
big data big data is normally
characterized by three V's volume
velocity and variety not necessarily in
that order
these are the main V's of Big Data there
are other V's also considered but not
popular such as veracity visualization
value and many more veracity here refers
to the truthfulness of data next we will
explore each of these three V's volume
refers to data volume data volume is the
size of digital data internet and social
media effect has resulted in the
explosion of digital data data has grown
from gigabytes to terabytes
to petabytes to exabytes total data on
the internet was 8 exabytes as of 2008
and by 2011 it is exploded to 150
exabytes it is growing at such a fast
pace that it reached 670 exabytes in
2013 that is like 30 percent growth per
year in another 10 years it is supposed
to exceed 2:7 zettabytes
how can one store and handle this much
data there are so many new terms to
describe the size of data now let us
understand the various terms used for
different data sizes this table shows
the various sizes used for big data we
all know kilobyte megabyte and gigabyte
a terabyte consists of 1000 gigabytes
and 1 petabyte is about 1,000 terabytes
new terms such as exabyte zettabyte and
yottabyte have been added to address big
data sizes when we say big data we
normally mean sizes in terms of
terabytes or more
now that you know the different big data
sizes let us look at the second V of Big
Data velocity of data refers to the
speed of data ingestion or data growth
there are millions of web pages being
added every day data gets created from
different sources such as desktops
laptops mobiles tablets and sensors
manufacturing facilities have thousands
of sensors that generate sensor data
every few seconds people use one or the
other devices to create data on twenty
four by seven basis we might think that
the data external to an organization
like on the Internet is growing but the
data created internal to the
organization is also growing at a faster
rate this is due to the increase in
global customer base of organizations
and also due to increased transactions
and interactions with the customers
there are many contributors to this data
growth web and social media online
billing systems ERP implementations
network and machine data growth and
revenues of organizations also means
that their data is also growing at a
rapid pace moving on we will look at the
third V of big data
data variety refers to the different
types of data that is being created one
of the major reasons for that is
multimedia and social media effect so
these days data is not just plain text
it includes images audio video XML and
HTML there is structured data such as
databases and XML and also unstructured
data such as program logs blogs like
WordPress and user comments on Twitter
and Facebook
there are many industries such as
Transport science and finance which are
adding a variety of data every day
let us reiterate the three V's of big
data volume velocity and variety now
that we have explored the three V's of
big data let us look at the evolution of
data over the years digital data has
gone through a cycle over the last
twenty to thirty years starting with
unstructured data we started with
document editors creating plain text
documents then files data handling and
spreadsheets increase the usage of
digital computers introduction of
relational databases revolutionize the
structured data many organizations have
created a large amount of structured
data using relational databases this
expanded to data warehouses and storage
area networks to handle large volume of
structured data then came metadata
concept that describes the data and semi
structured data such as HTML with the
advent of social media like Facebook and
Twitter and structured data has explored
in the last few years
thus data has come through a full circle
next let us look at some of the features
of big data big data has many features
it is extremely fragmented due to the
variety of data it does not provide
decisions you need to figure out how to
use it to make decisions big data is not
only unstructured data the structured
data component of Big Data extends and
complements your unstructured data big
data is not a substitute for your
structured data since most of the
information on the Internet is available
to everyone it can be used for good
causes and at the same time can be
misused by antisocial elements
to disturb peace generally Big Data has
a wide horizon so you may have hundreds
of fields in each line of your data it
is also very dynamic as gigabytes and
terabytes of data is created every day
and can change every day like the
weather big data can be both internal
generated within the organization or
external like social media or YouTube
moving on let us explore some industry
examples of big data
here are some industry examples of big
data in the retail industry big data is
used extensively for affinity detection
and market basket analysis they use it
to answer questions like when a customer
visits a store and buys a product X
which other product Y he or she is most
likely to buy they want to place the
product Y next to Product X so that the
customer has a more pleasant shopping
experience credit-card companies want to
detect any fraudulent purchases at the
earliest so that they can alert the
customer as soon as possible
banks want to scrutinize not only the
private data but also the public data
available about a customer so that they
can minimize the risk while giving loans
in medical diagnostics doctors can
diagnose a customer's illness based on
the symptoms instead of depending on
intuition digital marketers need to
process a lot of customer data to find
effective marketing channels based on
last twenty to thirty years of stock
market data algorithmic trading makes it
possible to maximize profits on one's
portfolio now we will look at some more
industry examples of Big Data almost
every industry has some use of big data
insurance companies can use it to
minimize insurance risks for example a
person's driving data can be captured
automatically by cars and forwarded to
insurance companies so that the premium
can be increased for risky drivers
manufacturing units and oil rigs have
thousands of sensors that generate
gigabytes of data every day this data is
analyzed to reduce risks and costly
equipment failures advertisers use
demographic data to capture their target
audience in a better way terabytes and
petabytes of data is analyzed by
genetics to come up with new models
power grids analyze large amount of
historic data and weather forecasting
data
to forecast power consumption as the
data is available to public law
enforcement officials have to be one
step ahead of the anti-social elements
to detect misuse of data and prevent
crimes so like we mentioned earlier
almost every industry uses big data in
one or the other way as we are now
familiar with the concept of big data
and its uses let us look at how big data
analysis is different from traditional
analytics with big data you use all your
data instead of sample data for analysis
in traditional analysis analysts take a
representative sample of data from the
available data and do their analysis to
provide their conclusions with Big Data
technology all the available data is
used for analysis you may find
associations in data predict future
outcomes and provide prescriptive
analysis prescriptive means you can say
this will happen instead of this may
happen the image on the screen depicts
how with the help of traditional
analytics you copy sample data to a
small database and run analysis on that
however with big data analytics you use
all the available data without sampling
now let us do a quick comparison between
Big Data technology and traditional
technology using big data for analysis
also means you make data-driven
decisions instead of decisions based on
intuitions with data you can support
decisions which would otherwise be left
to chance
analysis using big data can help
organizations increase their safety
standards reduce maintenance costs and
prevent failures here is how traditional
technology is different from Big Data
technology traditional technology has a
limit on scalability
whereas Big Data technology is highly
and massively scalable traditional
technology uses highly parallel
processors on a single machine whereas
Big Data technology uses distributed
processing with multiple machines in
traditional technology processors may be
distributed but data is stored at one
place
whereas with Big Data technology data is
distributed to multiple machines
traditional technology depends on
high-end expensive hardware of the order
of more than $40,000 per terabyte
whereas Big Data technology leverages
commodity Hardware which may cost less
$5,000 per tibi traditional technology
uses storage technologies like si N or
storage area network to store data
whereas Big Data technology uses
distributed data with data redundancy
the cost factor is very important for
CTO and CEOs to lean towards Big Data
technology moving on we will discuss the
concept of Apache Hadoop which is a
popular Big Data technology Apache
Hadoop is one of the most popular
frameworks for big data processing
Hadoop has two core components HDFS and
MapReduce Hadoop uses HDFS to distribute
the data into multiple machines and
MapReduce to distribute the process to
multiple machines Hadoop distributes the
processing to where the data is and uses
the principle of moving processing to
data instead of moving data to
processing firstly the data is divided
into multiple parts like data one data
to data three and so on which gets
distributed to multiple machines then
the processing is done using the CPUs of
each machine on the data of that machine
the diagram on the screen indicates how
HDFS distributes the data to multiple
machines and how MapReduce distributes
the processing to multiple CPUs on those
machines next let us look at HDFS
component of Hadoop HDFS is the storage
component of Hadoop it is an acronym for
Hadoop distributed file system it stores
each file as blocks with a default block
size of 64 megabytes this is quite large
as compared to 1k or 4k on Windows HDFS
is a write once read full many times
file system also called worm blocks are
replicated across nodes in the cluster
where the default replication is 3 let
us illustrate this with an example
suppose you store a three hundred and
twenty megabytes file into HDFS it gets
divided into five blocks each of size 64
megabytes as 64 asterisk 5 is equal to
320 if there are 5 nodes in the cluster
then each block is replicated to make 3
copies each to give a total of 50
blocks these blocks are then distributed
to five nodes so that no two replicas of
the same block are on the same node the
diagram on the screen shows how a 320 MB
file is divided into multiple blocks and
stored on five data nodes let us move on
to understand how MapReduce functions
MapReduce is the processing framework of
Hadoop it provides highly fault tolerant
distributed processing of the data
distributed by HDFS MapReduce consists
of two types of tasks mappers are tasks
that are run in parallel on different
nodes of the cluster and process the
data blocks Maps are actually key value
pairs after the completion of the map
tasks results are gathered and
aggregated by the reduced tasks of
MapReduce to reduce is to summarize and
consolidate reducers give the final
output of MapReduce each mapper runs on
the data block on that node data
locality is preferred this follows the
paradigm of taking process to the data
having learned about how HDFS and
MapReduce function let us explore the
concept of real time big data real time
big data refers to handling massive
amount of business data as soon as the
data is created to get valuable insights
and prescribe immediate actions here
real time means as soon as an event
happens using real time big data tools
you will be able to read and write data
in real time filter and aggregate in
real time and visualize data in real
time you can also process millions of
records per second next we will cite
some examples of real time big data
processing zookeeper is a distributed
coordination service of Apache it is
highly scalable and fault tolerant it
contains an open source library of
recipes it facilitates building
relations between distributed processes
and applications it provides useful
recipes to handle common issues in
distributed process coordination we have
come to the end of this lesson now let's
do a small quiz to test your knowledge
let us summarize the topics covered in
this lesson Big Data is characterized by
three V's
volume velocity and variety almost every
industry can use Big Data technology as
compared to traditional technology Big
Data technology uses commodity hardware
instead of expensive hardware Apache
Hadoop is a popular product to process
big data and has two core components
HDFS and MapReduce real time processing
of big data is required for some
industries kafka storm Cassandra
SPARC and HBase are some of the tools
used for real-time processing of big
data zookeeper is used for distributed
process coordination
you
this concludes the lesson introduction
to Big Data in the next lesson we will
introduce Apache storm hello and welcome
to the second lesson of the Apache storm
course offered by simply learn this
lesson will provide you an introduction
to storm it's data model architecture
and components let us start with
exploring the objectives of this lesson
by the end of this lesson you will be
able to describe the concept of storm
and explain streaming describe the
features and use cases for storm discuss
the storm data model describe storm
architecture and its components and
explain the different types of
topologies now let's get started with
understanding the concept of storm storm
is a real-time stream processing system
it is an open source and a part of
Apache projects it is a tool that
processes big data and provides fast and
reliable processing it can ingest high
volume and high velocity data it is
highly parallelizable scalable and fault
tolerant moving on let us explore the
uses of storm storm provides the
computation system that can be used for
real-time analytics machine learning and
unbounded stream processing it can take
continuously produced messages and can
output to multiple systems as we have
been discussing about streams so next
let us now understand what a stream is
in computing a stream represents a
continuous sequence of bytes of data it
is produced by one program and consumed
by another it is consumed in the first
in first out or fi fo sequence that
means if 1 2 3 4 5 is produced by one
program another program consumes it in
the order 1 2 3 4 5 only it can be
bounded or unbounded bounded means that
the data is limited unbounded means that
there is no limit and the producer will
keep producing the data as long as it
runs and the consumer will keep
consuming the data a Linux pipe is an
example of a stream the Linux command is
cat space log file vertical bar WC hi
Fionn l in this command cat log file
command produces a stream that is
consumed by WC - L command to display
the number of lines in the file next we
will explain industry use cases for
storm many industries can use storm for
real-time big data processing such as
one credit card can use it for fraud
detection on swipe to investment banks
can use it for trade pattern analysis in
real time three retail stores can use it
for dynamic pricing for transportation
providers can use it for route
suggestion based on traffic data five
health care providers can use it for the
monitoring of ICU sensors six telecom
organizations can use it for processing
switch data next let us look at the
storm data model storm data model
consists of tuples and streams a tuple
is an ordered list of named values
similar to a database row each field in
the tupple has a data type that can be
dynamic the field can be of any data
type such as a string integer float
double boolean or byte array
user-defined data types are also allowed
in tuples for example for the stock
market data if the schema is in the
ticker your value and status format then
some tuples can be ABC 2011 20 good ABC
2012 30 good ABC 2012 32 bad XYZ 2011 25
good a stream in storm is an unbounded
sequence of tuples for example if the
above tuples are stored in a file stocks
txt format then the command cat stocks
dot txt produces a stream if the process
is continuously putting data into stocks
dot txt format then it becomes an
unbounded stream next let us look at
storm architecture storm has a
master/slave architecture there is a
master server called Nimbus running on a
single node called master
there are slave services called
supervisor that are running on each
worker node supervisors start one or
more worker processes called workers
that run in parallel to process the
input worker processors store output to
a file system or database storm uses
zookeeper for distributed process
coordination the diagram shows the storm
architecture with one master node and
five worker nodes the Nimbus process is
running on the master node there is one
supervisor process running on each
worker node there are multiple worker
processes running on each worker node
the workers get the input from file
system or database and store the output
also to a file system or database a
zookeeper cluster is used for
coordinating the master supervisor and
worker processes moving on let us
explore the Nimbus process a log
processing program takes each line from
the log file and filters the messages
based on the log type and outputs the
log type input a log file containing
error warning and informational messages
this is a growing file getting
continuous lines of log messages output
output type of message error or warning
or info let us continue with the sample
program this program contains a single
spout and a single boat the spout does
the following opens the file reads each
line and outputs the entire line as a
tuple the bolt does the following reads
each tuple from the spout and checks if
the tuple contains the string error or
warning or info outputs only error or
warning or info the diagram shows the
outline of spouts and bolt the spout is
named line spout it has a loop to read
each line of input and outputs the
entire line the omit function is used to
output the line as a stream of tuples
the bolt is named log type bolt it takes
the tuple as input if the line contains
the string error then it outputs the
string error if the line contains the
string warning then it outputs the
string warning
early if the line contains the string
info then it outputs the string info
next let us explore the storm components
storm provides two types of components
that process the input stream spouts
spouts process external data to produce
streams of tuples spouts produce tuples
and send them to bolts bolts bolts
process the tuples from input streams
and produce some output tuples input
streams to bolt may come from spouts or
from another bolt the diagram shows a
storm cluster consisting of one spout
and two bolts the spout gets the data
from an external data source and
produces a stream of tuples the first
bolt takes the output tuples from spouts
and processes them to produce another
set of tuples the second bolt takes the
output tuples from bolt 1 and stores
them into an output stream now we will
understand the functioning of storm
spouts spouts is a component of storm
that ingests the data and creates the
stream of tuples for processing by the
bolts a spouts can create a stream of
tuples from the input and it
automatically serializes the output data
it can get data from other queueing
systems like Kafka Twitter RabbitMQ etc
spout implementations are available for
popular message producers such as katha
and Twitter a single spouts can produce
multiple streams of output each stream
output can be consumed by one or more
bolts the diagram shows a Twitter spout
that gets Twitter posts from a Twitter
feed and converts them into a stream of
tuples it also shows a Casca spout that
gets messages from kefka server and
produces a tuple of messages next let us
look at how storm bolt functions storm
bolt processes the tuples from spouts
and outputs to external systems or other
bolts the processing logic in a bolt can
include filters joins and aggregation
filter data examples include processing
only records with status is equal to
good processing only records with volume
is greater than 100 etc aggregation
examples include calculating the sum of
sale amount calculating the max
dock value etc a bolt can process any
number of input streams input data is
deserialized
and the output data is serialized
streams are treated as tuples bolts run
in parallel and can distribute across
machines in the storm cluster the
diagram shows four bolts running in
parallel bolt one produces the output
that is received by both bolt three and
bolt four bolt two produces the output
that is received by both bolt three and
bolt four bolt three stores the output
to a Cassandra database whereas bolt
four stores the output to Hadoop storage
next we will explore the functioning of
storm topology a group of spouts and
bolts running in a storm cluster form
the storm topology spouts and bolts run
in parallel there can be multiple spouts
and bolts topology determines how the
output of a spout is connected to the
input of bolts and how the output of a
bolt is connected to the input of other
bolts the diagram shows a storm topology
with one spout and five volts the output
of spout one is processed by three bolts
bolt one bolt two and bolt three bolt
four gets the output from bolt one and
bolt to bolt five gets the input from
bolt three this represents the storm
topology the input to spout one is
coming from an external data source the
output from bolt four goes to output one
and the output from bolt five goes to
output 2 let us illustrate storm with an
example problem the stock market data
which is continuously sent by an
external system should be processed so
that data with good status are inserted
into a database whereas data that are
with bad status are written to an error
log storm solution this will have one
spout and two bolts in the topology
spouts will get the data from the
external system and convert into stream
of tuples these tuples will be processed
by two bolts
those with status good will be processed
by bolt 1 those with status bad will be
processed by bolt two bolt one will save
the tuples to Cassandra database bolt
two will save
tuples to an error log file the diagram
shows the storm topology for the above
solution there is one spout that gets
the input from external data source
there is bolt one that processes the
tuples from the spout and stores the
tuples with good status to Cassandra
there is bolt two that processes the
tuples from the spout and stores the
tuples with bad status to an error log
now let us look at the concept of
serialization deserialization
serialization is the process to convert
data structures or objects into a
platform-independent stream of bytes it
is used to store data on disk or memory
and to transmit data over network the
purpose of serialization is to make data
readable by other programs for example
the object name colon character 10 ID
colon integer may have data bracket John
1 0 1 bracket this can be serialized as
John backslash 0 0 x 65 this represents
that the string John is followed by null
character and then hexadecimal
representation of 1 0 1 to read the data
programs have to reverse the
serialization process this is called d
serialization serialization
deserialization is also known as sir d
abbreviation of serialization
deserialization now we will go through
the steps involved in submitting a job
to storm a job is submitted to Nimbus
process to submit a job you need to one
create spouts and bolt functions to
build topology using spouts and bolts 3
configure parallelism for submit
topology to Nimbus the diagram shows a
Java program fragment for submitting a
job to storm it first gets a storm
topology builder object next it sets the
spouts and bolts for the topology with
the set spouts and set bolt methods it
also sets the connection from the output
of spouts to bolt using the shuffle
method shuffle grouping is a type of
grouping of input that we will discuss
in a subsequent lesson after setting the
spout and bolt the program sets the
number of workers for the job to two
which represents the number of workers
that will run in parallel finally using
the submit topology method the topology
is submitted to Nimbus process
now finally we will look at the types of
topologies download the vmware player
from VMware site for your system and
double click the executable to start the
installation
the installation wizard gets activated
and displays the welcome screen for the
installation of VMware Player
click Next to continue the installation
process on the License Agreement screen
select the field I accept the terms in
the license agreement after reading the
full agreement click Next to continue
the installation process choose the
default location and click Next to
continue the installation process
you
on the shortcut screen select the
shortcuts you wish to place on your
system and then click Next to continue
the installation process on the
acceptance screen ready to perform the
requested operations click continue next
the installation will start and will
take a few minutes to finish
you
once it is done click Next to continue
once the setup wizard has successfully
completed click finish to complete the
installation
you
next we will learn how to install simply
learns vm for Hadoop download and unzip
the Hadoop sudo server dart are a are
provided by simply learn
you
next start the VM player click open a
virtual machine to open an existing VM
browse and select the unzipped file from
the earlier step the actual location may
be different on your system
you
if asked to take ownership select take
ownership
you
click Play virtual machine to start the
virtual machine
you
if asked if you have moved or copied the
VM select copied it
you
if you get the software update screen
select remind me later option it will
take some time to open
you
when you get the login prompt use user
ID and password as simply learn to login
you
use the keys ctrl alt to move between
the VM screen and window screen after
login type ipconfig at the command
prompt to get the network configuration
of the machine note the IP address as we
will need it later
download win SCP from this URL install
and configure it to connect to the IP
address mentioned in the previous step
download putty from the given URL
install putty and configure it to
connect to the IP address above run
putty by double clicking on putty Exe
enter information as shown in the screen
you
you enter is the one you noted down
earlier for your system
you
click Save to save the settings and then
select open to open a connection to the
VM when you get the login prompt use the
user ID and password as simply learn to
login
congratulations you have successfully
logged into your VM from putty we will
use this to set up Apache Cassandra in
later lessons you can open multiple
connections to your VM using putty let
us summarize the topics covered in this
lesson storm is used for processing
streams of data storm data model
consists of tuples and streams
storm consists of spouts and bolts
spouts create tuples that are processed
by bolts
spouts and bolts together form the storm
topology
storm follows a master/slave
architecture the master process is
called Nimbus and the slave processes
are called supervisors data processing
is done by the workers that are
monitored by the supervisors this
concludes the lesson introduction to
storm
you