Kafka Tutorial | Apache Kafka Tutorial For Beginners | Kafka Architecture |What Is Kafka|Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
[Music]
after completing this lesson you will be
able to define Kafka describe some use
cases for Kafka describe the Kafka data
model describe Kafka architecture list
the types of messaging systems explain
the importance of brokers Kafka is a
high-performance real-time messaging
system it is an open-source tool and is
a part of Apache projects the
characteristics of Kafka are cockta is a
distributed and partitioned messaging
system that is highly fault tolerant and
scalable it has been tested to process
and send millions of messages per second
to several receivers Apache Kafka was
originally developed by LinkedIn to
handle their log files and later handed
over to the open source community in
early 2011 it became a main Apache
project in October 2012 a stable Apache
Kafka version 0.8.0
was released in February 2015 and a
stable Apache Kafka version 0.8.1 was
released in May 2015 which is the latest
version Kafka can be used for various
purposes in an organization such as
messaging service millions of messages
can be sent and received in real-time
real-time stream processing Kafka can be
used to process a continuous stream of
information in real-time and pass data
to stream processing systems such as
storm log aggregation Kafka can be used
to collect physical log files from
multiple systems and store it in
essential location such as HDFS commit
log service Kafka can be used as an
external commit log for distributed
systems event sourcing a time ordered
sequence of events can be maintained
through Kafka Kafka can be used to
aggregate user activity data such as
clicks navigation and searches
from different websites of an
organization such users activities can
be sent to a real-time monitoring system
and Hadoop system for offline processing
an example is illustrated in the image
the information from customer-facing
portals are sent in real time to the
Kafka cluster the Kafka cluster consists
of one or more servers that process the
messages in parallel the information is
sent to a real-time monitoring system to
monitor the user clicks navigation and
searches the information is also saved
in a hadoop system for offline
processing the Kafka data model consists
of messages and topics messages
represent information such as lines in a
log file a row of stock market data or
an error message from a system messages
are grouped into categories called
topics for example log message and stock
message the processes that publish
messages into a topic in Kafka are known
as producers the processes that receive
the messages from a topic in Kafka are
known as consumers the processes or
servers within Kafka that process the
messages are known as brokers a Kafka
cluster consists of a set of brokers
that process the messages the image
illustrates the Kafka data model it
shows a Kafka cluster that consists of
three brokers there are two producers
sending messages to the Kafka cluster
and to consumers receiving the messages
from the cluster producer one creates
messages for topic 1 whereas producer 2
sends messages for topic 2 these
messages are processed by the three
brokers in parallel and sent to the
consumers consumer one is interested in
topic 2 so it receives the messages for
topic 2 similarly consumer 2 is
interested in topic 1 so it receives the
messages for topic 1
the brokers in the Kafka cluster handle
the process of receiving storing and
forwarding the messages to the
interested consumers a topic is a
category of messages in Kafka the
producers publish the messages in two
topics and the consumers read the
messages from topics
a topic is divided into one or more
partitions a partition is also known as
a commit log each partition contains an
ordered set of messages each message is
identified by its offset in the
partition messages are added at one end
of the partition and consumed at the
other the image illustrates a topic
simple that is divided into two
partitions the writes are completed at
one end and the reads are completed at
the other it shows six messages in
partition zero and five messages in
partition one the offset of message one
in partition zero is zero as it is the
first message the offset of message six
in partition zero is five the messages
are written in the order one two three
four five and six whereas they are read
in the same order as one two three four
five and six the next message in
partition zero will be message seven
which will be written at offset six the
next message for partition one will be
message six which will be written at
offset five topics are divided into
partitions which are the unit of
parallelism and Kafka partitions allow
messages in a topic to be distributed to
multiple servers or brokers so that the
messages in a topic can be processed in
parallel a topic can have any number of
partitions
each partitions should fit in a single
Kafka server the number of partitions in
a topic decide the parallelism of the
topic the image illustrates two
partitions of a topic simple partition 0
consists of 6 messages whereas partition
1 consists of five messages partitions
can be distributed across the kafka
cluster each Kafka server or broker may
handle one or more partitions a
partition can be replicated across
several servers for fault tolerance one
server is marked as a leader for the
partition and the others are marked as
followers the leader controls the read
and write for the partition whereas the
followers replicate the data if a leader
fails one of the followers automatically
become the leader zookeeper is used for
the leader selection as explained in the
previous lesson
the image illustrates the partition of a
topic simple here the partition zero is
assigned to server one and partition one
is assigned to server two these servers
process the messages in parallel to
increase throughput the producer is the
creator of the message in Kafka
producers place the message to a
particular topic and decide what
partition to place the message into for
example a producer may place a message
into partition zero of topic simple
another producer may place a message
into partition one of topic simple
topics should already exist before a
message is placed by the producer
messages are added at one end of the
partition by kafka the image illustrates
a producer that creates three messages
and sends them to different topics and
partitions in Kafka message one is sent
to partition zero of topic test one
message two is sent to partition one of
topic test one and message three is sent
to partition zero of topic test two the
consumer is the receiver of the message
in Kafka each consumer belongs to a
consumer group a consumer group may have
one or more consumers the consumers
specify what topics they want to listen
to a message is sent to all the
consumers in a consumer group the
consumer groups are used to control the
messaging system the image illustrates
the three consumer groups they are
consumer group one consumer group two
and a consumer group three the consumer
Group one consists of three consumers
called consumer one consumer two and
consumer three the consumer group two
consists of two consumers called
consumer 4 and consumer 5 the consumer
group 3 consists of a single consumer
called consumer 6 Kafka architecture
consists of brokers that take messages
from the producers and add to a
partition of a topic brokers provide the
messages to the consumers from the
partitions the producers create the
messages and send them to a particular
topic and a partition of a Kafka cluster
a topic is divided into multiple
partitions the messages are added to the
partitions at one end and
soon in the same order each partition
acts as a message queue consumers are
divided into consumer groups each
message is delivered to one consumer in
each consumer group zookeeper is used
for coordination among the kafka brokers
the image illustrates the Kafka
architecture that consists of two
partitions called partition one and
partition two the two producers are
sending messages to the two brokers in
the cluster the brokers add the messages
to the partitions and the messages are
taken from the partitions in the same
order as insertion the messages are sent
to two consumer groups the image also
illustrates that the Kafka cluster or
brokers interact with zookeeper for
distributed coordination Kafka
architecture supports two types of
messaging systems known as published
subscribe and queue system the publish
subscribe system is also called pub/sub
in this system one system broadcasts the
messages and the consumers subscribed to
receive the messages each message is
received by all the subscribers so if
there are 100 messages published each
subscriber receives all the 100 messages
in the same order that they are produced
in the queue system each message has to
be consumed by only one consumer if
there are multiple consumers each
message is consumed by any one of the
available consumers in the same order
that they are received the image
illustrates the implementation of a cue
system consumer one consumer two and
consumer three belonged to the same
consumer group so out of the six
messages two messages are received by
consumer one two messages by consumer
two and two messages by consumer three
note that the messages are received in
the same order that they are produced so
consumer one receives message one
consumer two receives message two and
consumer three receives message three
after this consumer one receives message
for consumer two receives message five
and consumer three receives message six
the image illustrates the implementation
of a publish/subscribe system consumer
one consumer two and consumer three
belong to three separate consumer group
so all the six messages are sent to all
the three consumer groups called
consumer group one consumer group two
and consumer group three since there is
only one consumer end consumer Group one
it receives all the six messages in the
order one two three four five and six
similarly consumer two and consumer
three also receive all the six messages
in the same order brokers are the kafka
processes that process the messages in
Kafka each machine in the cluster can
run one broker the brokers coordinator
monk each other using zookeeper one
broker acts as a leader for a partition
and handles the delivery and persistence
whereas the others act as followers
brokers receive the message from the
producer and send it to the consumer
groups Kafka guarantees the following
guarantee one message is sent by a
producer to a topic and the partition
are appended in the same order this
ensures that the message is produced
earlier do not get ahead of the messages
produced later the time order is
maintained very strictly guaranteed to a
consumer instance gets the messages in
the same order as they are produced
which means that the messages are never
out of order if the messages are
produced in the order one two three four
five six they will be received in the
order one two three four five and six
this is important in messaging systems
as the dependency is on the time order
of messages guarantee three a topic with
replication factor in tolerates up to n
minus one server failures for example
when the replication factor is specified
as 3 there will be no loss of messages
even if two machines fail LinkedIn or
the website WWL intercom is the largest
network of professionals and is the
originator of Kafka Kafka is used by
LinkedIn to manage streams of
information some of the uses of Kafka at
LinkedIn are as follows
monitoring Kafka is used to collect
metrics from various systems and to
create monitoring dashboards messaging
Kafka is used as message queues for
content thieves and as publish/subscribe
system for searches analytics Kafka is
used to collect page views and clicks
from customer facing websites and to
store the information into a central
Hadoop based analytic system a building
block for distributed applications Kafka
is used as a building block for
distributed applications and for
building distributed databases and
distributed log systems Kafka uses the
primary backup method of replication in
primary backup method one machine or one
replica is called a leader and is chosen
as the primary the remaining machines or
replicas are chosen as the followers and
act as backups the leader propagates the
rights to the followers and waits until
the rites are completed on all the
replicas if a replica is down it is
skipped for the right however koepcke
will write a copy to the machine once it
is back if the leader fails one of the
followers will be chosen as the new
leader this mechanism can tolerate n
minus 1 failures if the replication
factor is in which can be specified at
the topic level persistence means a
message can be delivered even if the
machine that handles the message fails
Kafka uses the Linux file system for
persistence of messages persistence
ensures no messages are lost Kufner
relies on the file system page cache for
fast reads and writes all the data is
immediately written to a file in a file
system so that they can be recovered
even if the machine fails messages are
grouped as message sets for more
efficient writes message sets can be
compressed to reduce network bandwidth a
standardized binary message format is
used among producers brokers and
consumers to minimize data modification
here is a quick recap of what we have
learned in this lesson
Kafka is a high-performance real-time
messaging system Kafka can be used as an
external commit log for distributed
systems Kafka data model consists of
messages and topics
Kafka architecture consists of brokers
that take messages from the producers
and add to a partition of a topic Kafka
architecture supports two types of
messaging system called publish
subscribe and queue system brokers are
the Kafka processes that process the
messages in Kafka hey want to become an
expert in big data then subscribe to the
simply learn channel and click here to
watch more such videos to nerd up and
get certified in Big Data click here