Introduction To Apache Kafka Certification Training | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

hello and welcome to the introductory
lesson of the Apache Kafka developer
course offered by simply learn in this
lesson we will introduce you to the
course after completing this course you
will be able to describe the importance
of big data describe the fundamental
concepts of Kafka describe the
architecture of Kafka explain how to
install and configure Kafka explain how
to use Kafka for real-time messaging
this training course provides
information on big data fundamentals
real time Big Data Kafka origin and
features Kafka architecture and a data
model Kafka installation configuration
and monitoring and interfaces to Kafka
the target audience for this course
includes professionals aspiring for a
career in big data analytics
professionals research professionals IT
developers testers and project managers
it also includes students and those
looking for a change in career the
prerequisites for this course include
knowledge of any messaging system basic
knowledge of Java or any programming
language some knowledge of Linux or UNIX
based systems is desired there are 5
lessons in this course the table lists
the lesson numbers along with the name
of the lessons covered
this concludes course introduction the
next lesson is big-data overview
hello and welcome to lesson one of the
Apache Kafka developer course offered by
simply learned this lesson provides an
overview of Big Data after completing
this lesson you will be able to describe
a big data list the three V's of big
data lists the various data sizes used
for big data describe Apache Hadoop
explain the concepts of real time big
data processing list some tools that
handle real time at big data digital
data has exploded over the last two to
three years Facebook Twitter YouTube and
sensory networks have contributed to the
rapid growth of data data volumes are
measured in terms of millions of
gigabytes the technology has evolved to
store process and analyze large volumes
of data and to make decisions based on
it the technology to utilize large
volumes of data represents big data big
data is typically characterized by three
bees they are volume velocity and
variety the other characteristics
include porosity which refers to the
truthfulness of data visualization value
and so on volume refers to data which is
the size of digital data the impact of
the internet and social media effect has
resulted in the explosion of digital
data data has grown from gigabytes to
terabytes petabytes exabytes and
zettabytes in 2008 the total data on the
Internet was 8 exabytes it has exploded
to 150 exabytes in 2011 and reached 670
exabytes in 2013 which calculates a
growth rate of 20% a year it is expected
to exceed 7 Zeta bytes in the next 10
years the table shows the various sizes
used for big data the size along with
the power and description of each data
term is given the terms kilobyte
megabyte and gigabyte are familiar
terabyte is about one thousand gigabytes
and petabyte is about one thousand
terabytes the new terms added to address
big data sizes are exabyte zettabyte and
Yoda byte typically the term big data
refers to data sizes in terms of
terabytes or more velocity of data refer
to the speed of data ingestion or data
growth millions of webpages get added
every day data also gets created from
different sources such as desktops
laptops mobiles tablets and sensors
manufacturing facilities have thousands
of sensors that generate data every few
seconds people use different devices to
create data throughout the day due to
the increase in the global customer base
and transactions and interactions with
customers data created within an
organization is growing along with
external data there are many
contributors to data growth such as web
and social media online billing systems
ERP implementations Network and machines
growth in the revenues of organizations
indicate growth in data data variety
refers to the types of data such as text
images audio video XML and HTML there
are three types of data structured data
where the data is represented in a
tabular format for example my sequel
databases semi structured data where the
data does not have a formal data model
for example XML files unstructured data
where there is no predefined data model
data is defined at runtime
for example text files industries such
as Transport science and finance add a
variety of data every day digital data
has evolved over 30 years starting with
unstructured data initially data was
created as plain text documents next
files were created data and spreadsheets
increased the usage of digital computers
the introduction of relational databases
revolutionized structured data as many
organizations used it to create large
amounts of structured data next data
expanded to data warehouses and storage
area networks or sans to handle large
volumes of structured data then the
concept of metadata was introduced to
describe structured and semi-structured
data finally with the advent of social
media such as Facebook and Twitter
unstructured data has exploded in the
last few years the images depict data
evolution starting with unstructured
data
to a structured database and ending with
unstructured social media comments some
of the features of Big Data are as
follows big data is extremely fragmented
due to the variety of data it does not
provide decisions however it can be used
to make them it includes both
unstructured and structured data where
structured data extends and complements
unstructured data however big data is
not a substitute for structured data
furthermore since most of the
information on the Internet is available
to anyone it can be misused by
anti-social elements big data is also
wide therefore there could be hundreds
of fields in each line of data next big
data is dynamic as gigabytes and
terabytes of data are created every day
finally big data can be internal
generated within the organization and
external generated on social media here
are some industry examples of big data
in the retail sector big data is used
extensively for affinity detection and
for performing market analysis a retail
company wants to find out the product Y
which the customer is most likely to buy
after buying the product X the company
wants to place product Y next to product
X to ensure the customer has a pleasant
shopping experience
credit card companies can detect
fraudulent purchases quickly to alert
customers while giving loans banks
examine the private and public data of a
customer to minimize risks in medical
diagnostics doctors diagnose a patient's
illness based on symptoms instead of
intuition digital marketers need to
process huge customer data to find
effective marketing channels based on
the last 20 to 30 years of stock market
data algorithmic trading can maximize
profits on a portfolio insurance
companies use big data to minimize
insurance risks an individual's driving
data can be captured automatically and
sent to the insurance company to
calculate the premium for risky drivers
manufacturing units and oil rigs have
sensors that generate Digga bits of data
every day that are analyzed to reduce
the risk of equipment failures
advertisers use demographic data to
identify the target audience terabytes
and petabytes of data are analyzed in
the field of genetics to design new
models power grids analyze large amounts
of historical and weather forecasting
data to forecast power consumption as
data is available to the public law
enforcement officials must take
necessary measures to detect misuse of
data and prevent crimes in traditional
analytics method analysts take a
representative data sample to perform
analysis and to draw conclusions using
Big Data technology the entire data set
can be used instead of sample data big
data analysis helps to find associations
in data predict future outcomes and
perform prescriptive analysis the
outcome of prescriptive analysis will be
a definitive answer and not a probable
answer big data analysis helps to take
data-driven decisions instead of
intuition based decisions it also helps
organizations increase their safety
standards reduce maintenance costs and
prevent failures traditional technology
can be compared with Big Data technology
in the following ways
traditional technology has a limit on
scalability whereas Big Data technology
is highly scalable traditional
technology uses highly parallel
processors on a single machine whereas
Big Data technology uses distributed
processing with multiple machines in
traditional technology processors may be
distributed with data in a single
machine however in Big Data technology
the data is distributed to multiple
machines traditional technology depends
on high-end expensive hardware that
costs a more than forty thousand dollars
per terabyte on the other hand Big Data
technology leverages commodity Hardware
that costs less than five thousand
dollars per terabyte traditional
technology uses storage technologies
such as San where as Big Data technology
uses distributed data with data
redundancy the cost factor is important
for of chief technology officers or CTOs
and chief executive officers or CEOs to
lean towards Big Data technology in
computing a stream represents a
continuous sequence of bytes of data
it is produced by one program and
consumed by another it is consumed in
first-in first-out sequence for example
if one two three four five is produced
by one program another program consumes
it in the order one two three four five
only it can be bounded or unbounded
bounded means that the data is limited
unbounded means that there is no limit
the producer will keep producing data
till it runs and consumer will keep
consuming the data a linux pipe is an
example of a stream the linux command is
catalog file vertical bar WC - L in this
command catalog file produces a stream
that is consumed by WC - L to display
the number of lines in that file Apache
Hadoop is the most popular framework for
big data processing it has two core
components they are Hadoop distributed
file system or HDFS and MapReduce it
uses HDFS to distribute the data into
multiple machines and a MapReduce to
distribute the process to multiple
machines it uses the principle of moving
processing to data instead of data to
processing first HDFS divides the data
into multiple sets such as data 1 data 2
and data 3 next MapReduce distributes
the datasets to multiple machines such
as CPU 1 CPU 2 and CPU 3 finally
processing is completed by the CPUs of
each machine where the data is stored so
CPU 1 processes data one CPU 2 processes
data 2 and CPU 3 processes data 3 HDFS
is the storage component of Hadoop which
stores each pile as blocks with a
default block size of 64 megabytes this
is larger than the block size on Windows
which is 1k or 4k HDFS is a write only
read many times form of filesystem which
is also called worm blocks are
replicated across nodes in a cluster
HDFS provides three default replication
copies the image illustrates the concept
for example a 320 megabyte file is
stored in
to HDFS the file is divided into five
blocks each of size 64 megabytes as 64 x
5 is 320 if there are 5 nodes in the
cluster each block is replicated to make
three copies to result in a total of 15
blocks these blocks are further
distributed to the 5 nodes so that no
two replicas of the same block are on
the same node MapReduce is the
processing framework of Hadoop it
provides highly fault tolerant
distributed processing of the data by
HDFS it consists of two types of tasks
mappers are tasks that run in parallel
on different nodes of the cluster and
process the data blocks maps are
actually key value pairs after the
completion of map tasks results are
gathered and aggregated by the reduced
tasks of MapReduce reduce is used to
summarize and consolidate reducers give
the final output of MapReduce each
mapper runs on the data block on that
node data locality is preferred this
follows the paradigm of taking the
process to the data some of the tools to
handle big data in real time are as
follows
Apache Kafka Apache storm Apache spark
Apache Cassandra Apache HBase Kafka is a
high-performance real-time messaging
system it is an open source tool and is
a part of Apache projects it provides a
distributed and partitioned messaging
system that is highly fault tolerant it
can process millions of messages per
second and to send the messages to many
receivers storm is a real-time stream
processing system it is an open source
tool and is a part of Apache projects it
provides fast and reliable processing of
big data it can process unbounded
streams that sends data to storm
continuously it can interface with
messaging cues such as Kafka to get
input message data and store the
received data into a real-time big data
database such as Cassandra Apache spark
is considered to be the next-generation
MapReduce it is also an apache open
source project it is used to transform
distributed data
it provides data transforms beyond map
and reduce it processes data faster than
hadoop mapreduce when entire data fits
in memory
spark is found to be 100 times faster
than hadoop mapreduce whereas in other
cases it is found to be at least 10
times faster spark is suitable for a
batch in real time processing it
provides spark sequel for a sequel
interface to big data it provides
built-in libraries for machine learning
and a graph processing machine learning
consists of programs that can learn
based on the data without being
explicitly programmed a graph is a set
of nodes and edges connecting these
nodes graph processing consists of
algorithm to process the nodes and edges
of a graph Cassandra is an apache open
source no sequel database with the
following characteristics it is highly
fault tolerant with no s POF or a single
point of failure
it is highly available machines which
are also called nodes are logically
organized in the ring architecture
real-time read and light fast rights
with tunable consistency the level of
consistency can be controlled among
multiple nodes that contain data
provides a simple sequel interface and
interface is similar to sequel to insert
update and select the data it is a key
value database each row of data has a
primary key to identify the data it is
highly and horizontally scalable with
thousands of nodes in a cluster the
image shows the Cassandra logo where the
nodes are organized in the ring
architecture Apache HBase is another
open source and no sequel database it is
a distributed database with columnar
storage that is built on top of HDFS it
provides real-time read and write random
access to data it supports large
databases in the order of terabytes and
petabytes it is not relational and it
does not support sequel real-time big
data refers to handling a massive amount
of business data as soon as the data is
created to get valuable insights and
prescribe immediate actions here will
time refers to event that occurs using
real time big data tools you can read
and write data in real time filter and
aggregate in real time visualize data in
real time process millions of records
per second some use cases for real time
big data are as follows a telecom
provider wants to provide data plans to
customers based on location here the
location data is received continuously
and has to be processed in real time a
bank wants to indicate the ATM location
based on customer location here the
customer location data is received in
real time and recommendation has to be
made immediately a car manufacturer can
alert the car owner on any urgent
maintenance required on the car based on
the data provided by the car during
driving measurement data of various
sensors in the car has to be streamed to
the car manufacturer in real time
a news channel may monitor breaking news
items across the globe real-time news
data from hundreds of sources has to be
prioritized and selected for breaking
news
a security system may monitor movements
in a stadium during a game any
suspicious movements need to be reported
immediately a telecom network provider
wants to use the least congested network
for each call the decisions have to be
made in real-time
a credit card company wants to prevent
fraudulent transactions here probably
both real-time and offline processing
maybe involved a stock market
application recommends stocks to buy
every second based on the market
conditions volatile market conditions
have to be analyzed in a real time here
is a quick recap of what we have learned
in this lesson
big data is typically characterized by
three V's they are volume variety and
velocity the various data sizes used for
Big Data include kilobyte megabyte
gigabyte terabyte petabyte exabyte Zeta
byte and a Yoda byte Apache Hadoop is
the most popular framework for big data
processing it has two core components
they are Hadoop distributed file system
and MapReduce real time big data refers
to handling a massive amount of business
data as soon as the data is created to
get valuable insights and prescribe
immediate actions Kafka storm Cassandra
SPARC and HBase are some of the tools to
handle real time processing of big data
this concludes Big Data overview the
next lesson is introduction to zookeeper
hello and welcome to lesson 2 of the
Apache Kafka course offered my simply
learn this lesson provides an
introduction to zookeeper after
completing this lesson you will be able
to describe what zookeeper is and how it
functions and explains some of the
common problems of distributed systems
you will also be able to illustrate the
data model for zookeeper and compare how
the two types of Z nodes are different
from each other
furthermore this lesson will help you
discuss a few zookeeper recipes and the
way they handle some of the problems of
distributed systems zookeeper is a
coordination service of Apache that
helps manage the activities of
distributed applications it is highly
scalable it also has an open source
library of recipes for distributed
systems such as leader selection
exclusive locks book keeper and so on
these recipes facilitate building
relationships between distributed
processes and Apple
patience one of the main features of
zookeeper is that it helps handle
partial failures in distributed systems
before you learn about zookeeper let us
understand what distributed applications
are and what sort of problems arise
while using them distributed
applications are run on multiple
machines in parallel they function by
following the divide-and-conquer
principle this means that they divide
large jobs into smaller jobs which are
then run in parallel on multiple
machines they are horizontally scalable
if adding more machines reduces the
execution time for example if 10
machines do a job in 10 hours adding 10
more machines may have the execution
time by 5 hours they are vertically
scalable if increasing the memory CPU or
other resources of each machine reduces
the execution time for example
increasing the memory from 100 gigabytes
to 256 gigabytes may reduce the
execution time of a job from 10 hours to
5 hours the diagram shows 3 machines
connected by a network switch an
application can be distributed to run on
them in parallel these machines are also
referred to as nodes in a cluster one of
the major issues that crops up while
using distributed systems is partial
failure another common problem is a race
condition deadlocks and inconsistent
states are also some challenges of
distributed systems let us look at each
of these difficulties in detail partial
failure is a major challenge in
distributed applications suppose there
are two nodes node 1 and node 2 in the
distributed system node one sends a
message to no 2 through the network
however the network fails before node
one receives an acknowledgment from node
two
as a result node one does not know if
note two got the message or not people
know the actual status only after the
network gets connected again
this is known as a partial failure in
distributed applications though partial
failures cannot be prevented tools like
zookeeper provide a mechanism to handle
them efficiently a race condition takes
place in distributed applications when
multiple machines are waiting for one
resource to become free suppose there
are four different nodes in a
distributed system let us assume that
currently only node one is using the
resource so it has an exclusive lock on
the resource all the other nodes from
node 2 to node 4 are waiting for the
resource to become available when node 1
releases the resource node 2 to 4 raised
to acquire the resource only one of them
succeeds while the others go back to the
waiting State this process continues
till all the nodes get the resource this
is called a race condition deadlocks
occur when there is a cyclic dependency
on resources the diagram illustrates a
deadlock situation there are two
machines machine 1 and a machine 2 and
there are 2 resources resource a and
resource B machine 1 has locked resource
a and is waiting to lock resource B at
the same time machine
2 has locked resource B and is waiting
to lock resource a since none of the
locks can be acquired or released and
leads to a deadlock to resolve a
deadlock one of the processes has to be
killed and redo the processing detecting
deadlocks are generally cpu intensive
and expensive operations
inconsistencies take place when changes
are not propagated to all the machines
in the distributed system for example
let us consider a salary data which is
initially equal to 100 this data is
replicated to both the machines the data
is later updated to 200 this changes
first propagated to machine 1 the value
stored on it is now changed to 200
however due to some failure this update
is not propagated to machine 2 so the
value stored on it remains as 100 in
such a situation if the process a reads
from machine one while process B reads
from Machine 2 each process will get 2
separate values of the salary data this
leads to inconsistencies zookeeper helps
coordinate distributed applications it
provides a very simple interface it is
expressive which means that it provides
basic blocks that can be used to build
larger applications it is also highly
available and reliable this is because
it runs on multiple servers at the same
time so even if a few servers fail it
continues to function to have a fault
tolerance for in machine failures it is
recommended to have 2 n plus 1 machines
running a zookeeper service for example
if you want to have a fault tolerance of
3 machines then you should have sookie /
running on 7 machines another feature of
zookeeper is that it has loosely coupled
interactions machines using it do not
have to know each other
zookeeper is actually an extensive
library of recipes for distributed
coordination
the zookeeper data model consists of a
hierarchical tree of nodes called Z
nodes the tree of nodes is similar to
our directory structure in Linux each Z
node stores a small amount of data and
has an Associated access control list or
ACL ACL represents which users can read
write and or update the Z node a Z node
can only store a maximum limit of one
megabyte of data the diagram here shows
a tree structure of Z nodes there are
two z nodes slash Kafka and its last
HBase at the root level at the second
level there are two more Z nodes slash
Kafka slash node 0 and slash Kafka slash
node 1 zookeeper Z nodes can be of two
types
persistency nodes and ephemeral Z notes
note that when a Z node is created its
type is specified and it cannot be
changed later persistency nodes are
permanent and have to be deleted
explicitly by the client they stay even
after the session that created the Z
node is terminated at Fumero z nodes are
temporary these e nodes are deleted
automatically when the client session
creating them ends ephah me rosie nodes
are used to detect the termination of a
client alerts known as watch can be set
up to detect the deletion of the Z node
z nodes can be sequential to do this you
can set a sequence flag while creating a
Z node the value of this flag is an
increasing counter the counter is
maintained by the parent z node for
sequential z nodes the sequence number
is appended to the name of the z node
sequential z nodes are used to specify
ordering of the z nodes in the diagram
you can see that under slash kafka
parent three sequential z nodes node 0
node 1 and node 2 are created in a later
lesson you will learn how to install
zookeeper and Kafka on an Ubuntu Linux
system however if you need to work on an
operating system other than Linux you
can access the software provided by
VMware this software allows running one
operating system on another using a
virtual machine this is facilitated by
VMware Player for non-commercial use the
player can be downloaded and used free
of cost from the VMware website
simply learn has created a virtual
machine on vmware player on this machine
known as hadoop sudo server comes with
their pre installed Ubuntu 12.04 LTS
operating system and Hadoop setup it can
be opened with the VMware Player and can
be used for installing kafka Hadoop sudo
server can be downloaded from the given
link putty is a popular free tool for
connecting to Linux systems from Windows
through a remote terminal it overcomes
some of the limitations of the VM and
for example it allows moving the mouse
pointer with ease scrolling in the
window and copying and pasting text
putty can be downloaded from the given
link when SCP is a popular tool for
copying files between Windows and Linux
it stands for Windows secure copy it can
be used to copy the files from the local
windows to the Ubuntu VM running in the
VM player when SCP can be downloaded
from the given link this demo will show
the steps to install Ubuntu virtual
machine or VM and connect with putty
download the VMware Player from the
given link and double-click the VMware -
player - 7.10 - 2 4 9 6 8 - 4 dot exe
file to start the installation
the installation wizard appears with the
welcome screen of the VMware Player
setup click the next button to continue
the installation process
in the License Agreement screen select
the I accept the terms in the license
agreement radio button after reading the
full agreement click the next button to
continue the installation process
you
in the destination folder screen choose
the default location and click the next
button to continue the installation
process in the software update screen
uncheck the check for product updates on
startup checkbox click the next button
to continue
in the user experience improvement
program screen uncheck the help improve
VMware Player checkbox
click the next button to continue in the
shortcuts the screen click the next
button to continue the installation
process
in the ready to perform the requested
operation screen click the continue
button the installation process will
start this will take a few minutes to
complete once this is completed click
the next button to continue
you
in the setup wizard' complete screen
click the finish button to complete the
installation
start the vmware player
in the welcome screen click the open a
virtual machine option to open an
existing VM
download and unzip the Hadoop pseudo
server rar file provided by simply learn
you
the extraction process begins
open the vmware player again and click
the open a virtual machine option
Brow's and double-click the unzipped
hadoop sudo server file note that the
actual location may be different on your
system
opened the Hadoop pseudo distributed
server dot vmx file
click the take ownership button
you
click the play virtual machine play
button to start the virtual machine
click the I copied it button in the
removable devices pop-up window
select that do not show this hint again
checkbox and click the ok button
in the software updates window click the
remind me later button to continue
you
you
when you get the login prompt enter the
login ID as simply learned and password
as simply learn
you
hope that you can use the ctrl + alt
keys to switch between the VM screen and
the window screen after logging in enter
ifconfig in the command prompt to get
the network configuration of the system
you
write down the IP address of the system
as you will need it in the later part of
the demo
download putty using the given link
install and configure putty to connect
it to the IP address saved earlier
further download winscp from the given
link when SCP is a resourceful utility
to exchange files between the Windows
and Linux VM you can install and
configure winscp to connect to the IP
address saved earlier
run putty by double-clicking the putty -
0.64 - installer exe file in the
security warning window click the Run
button the Welcome wizard appears click
the next button to continue
in the select destination location
window click the next button to continue
in the select Start menu folder window
click the next button
in the select additional tasks window
click the next button to continue
finally in there ready to install window
click the install button in the setup
wizard uncheck the view readme.txt
checkbox and click the finish button
open putty
you
you
in the putty configuration window enter
the hostname with the IP address saved
in the earlier part of this demo simply
learn at the rate 192.168.1 89.1 to 8
you
you
select simply learn from the given
options and click the load button
insured the IP address is 192.168.1 89.1
to eight next click the Save button to
save the settings and the Open button to
open a connection to the VM
you
in the putty security alert window click
the yes button in the login prompt enter
the user ID as simply learned and the
password as simply learned
congratulations you have successfully
logged into the virtual machine from
putty this VM will be used to set up
Kafka also in the later lessons
note that multiple connections can be
opened on the VM using putty this
concludes the process of setting up and
installing the VM
to annastolz ooh keeper on the simply
learned vm first update the installation
libraries then use the app get install
or command to install zookeeper when
prompted to enter a password type simply
learned all in lowercase type Y if asked
for any confirmation
to start the zookeeper server you first
have to configure it to do that set up
the directory permissions for zookeeper
you can then start the zookeeper server
using the command given here
note that the zookeeper server listens
on the port 21 81 by default
zookeeper installation comes with a
command-line interface if you recall
2181 is the default port for zookeeper
the command-line interface gives the
following command prompt the
command-line interface can be used to
check the Z nodes and create new Z notes
in the commands given here the values
200 201 and 202 are the data we want to
associate with the Z nodes the table
shows the commands that can be used in
the zookeeper command-line interface the
create command that can be used to
create a z node the option - e is used
to create an FMEA rosie node however if
you do not specify this option a
persistent z node is created the option
hyphen s is used to specify a sequential
z node the help command is used to get
information about the available commands
path is the full path of the z node to
be created and starts with a forward
slash some data has to be associated
with the z node and is specified with
the data the LS command is used to list
the z node directory tree a path
starting with a forward slash is
specified you can also add a watch on
the path to be alerted about any changes
to the path get inset commands can be
used to fetch the data of the Z node or
to update the data delete can be used to
delete the Z node at the path finally
you can use quit to exit the command
line interface
like the command-line interface
zookeeper also provides ap is that can
be called from either Java or any other
language zookeeper has different api's
to create Z nodes and watches getting
and setting data as well as to delete Z
nodes in addition it also has ap is that
can be used to get all the children of a
Z node and the check of a Z node exists
there's also an API to synchronize all
the Z notes so that all of them have the
same data
when a process is sending data to
another process it has to handle partial
failures zookeeper provides a recipe to
tackle such a situation in the example
given here
process 2 is trying to send information
to process 1 the receiver creates an
ephemeral Z node with the same name as
the session ID or process name in this
case the Z node is created with the name
slash process 1 the sender keeps a watch
on the receiver z mode
if the data reaches the receiver process
successfully it informs the sender and
then deletes the Z note however if
process one fails the Z note is
automatically removed the sender gets an
alert about this change through the
watch
it then takes an appropriate action such
as rescinding the message in this
process we use watches and the ephemeral
character of Z nodes to handle partial
failures let us now look at another
zookeeper recipe leader election uses
ephemeral sequential nodes to create an
automatic node order let us see how the
three processes process 1 process 2 and
process 3 used a leader election
mechanism each process creates a
sequential fo Merrill's e node under
parent slash kafka with the prefix node
thus process one gets to Z node slash
Kafka slash node 0 process 2 gets to Z
node slash Kafka slash node 1 and
process 3 gets to Z node / Kafka slash
no - too busy node with the least
sequential value is chosen as the leader
process - and process 3 in turn become
followers so process 2 watches its
immediate proceeding sequence number
slash Kafka slash node 0 likewise
process three watches flash Kafka / node
1 to automatically reorder nodes the
sequential property of Z nodes is used
if process 1 completes successfully it
deletes the Z node / Kafka / node 0 this
alerts process 2 as it has a watch on
the leader Z node now process 2 becomes
the leader as it has the lowest sequence
value process 3 remains a follower as
the preceding Z node is not modified in
this case the watch feature serves to
change the leader when this process
helps you avoid a race condition note
that only process one and process two
are handling the leader change however
if process one dies before completion
then the Z node / Kafka / node 0 is
automatically deleted as it is an ephah
Merrill's ii node this alerts process -
as it has a watch on the leader z node
now process 2 becomes the leader as it
has the lowest sequence value process 3
remains a follower as the preceding z
node is not modified leader election
helps distributed processes to function
by automatically handling a node failure
here is a quick recap of what we have
learned in this lesson
zookeeper is a distributed coordination
service partial failure is a major issue
in distributed coordination zookeeper
uses a hierarchical tree of Z nodes
zookeeper has two types of Z nodes
persistent and Neph amaryl zookeeper
provides recipes for handling common
problems in distributed systems
sequential property is used to order Z
nodes a watch can be set up to get
alerts on changes to Z nodes this
concludes the lesson on introduction to
zookeeper
you