Understanding Cassandra Administration | Cassandra Adminstration Tutorial-1 | Edureka Video Lecture | Cassandra: Learn and Master - Database Management

so today we are going to learn about the
administration of Cassandra clusters my
basically monitoring managing and doing
some performance tuning etc how do you
really want to manage a Cassandra
cluster okay its Pacifica self
administration so agenda basically would
be how do you actually plan ahead the
cluster diploma basically you have to do
some planning rates when you say
capacity planning etc if I have to pick
up a choice as to what is the number of
nodes that I should be using for
performing a cluster so what those
choices you have to make so for that one
how do you actually make the decisions
how do you actually go and do that
because our capacity plan but something
which you look at and then we look at
some of the basic configuration of
Cassandra how do you monitor this under
cluster how do you maintain a clock
Cassandra cluster how do you remove add
a new node or remove an existing or how
do you do load balancing there are many
many different things that can be done
and then on a security lines how do you
really give different permissions for
the ski mask to the users and then other
thing is how do you tune the performance
okay okay so let's go forward we touched
upon four so cluster deployment so when
you plot a deployment you surely need to
see you would need to understand what is
your capacity that you're looking at say
what is the amount of application data
that you want to support what kind of
hardware you want to use what so the
various different things like what is
the number of nodes that you need
actually for your cluster that's your
capacity planning the type of hardware
that you use the key space replicating
options that you will really have and
then the node configuration tools so we
will talk about each of them in detail
as to what they actually mean and the
purpose behind them okay coming to
hardware selection
what are the various parameters that you
would look for when it comes to hardware
selections you surely have to select the
CPU the speed of the CPU that you really
want to use what is the amount of memory
that you would actually use a ram
basically the DRAM that you would need
for a system to deploy a cluster then
what is the type of network that you
would used to connect the type of
connections between the nodes in the
cluster itself and then there what is a
disk space how much disk space you would
actually need for say storing say
hundred bytes of hundred kilobytes of
application data so how do you actually
plan your disk sizes so there are some
things which we will learn in detail
okay so as we all know right memory is
more memory naturally means larger cache
size which automatically means fewer
stables because you end up having bigger
mem tables which means that I can have a
lot more data inside the memory and I do
not have to write it or serialize it to
a disk based representation which is SS
tables in etcetera so hold mode recently
written data always so more memory
automatically means larger mem tables
fewer SS tables and larger cache sizes
which surely is a good advantage so it
depends on how do you want to deploy so
how much memory is that you have at your
disposal so if you look at it in general
it's a good refinement with the overall
performance so you have more memory
please go ahead and add more memory it's
as simple as that
so coming to
a very important part of the entire
setup how do you do it this television
there is something called capacity and
i/o throughput so the two different
important factors that you would look
for when it comes to disk selection ok
let's talk about disk in detail in u.s.
so coming to CPU we always talked about
a CPU a general thing right we already
talked about KarenT general mechanism is
a typical deployments of Cassandra
clusters this actually works with ATP
course because that sort of gives you a
best price performance ratio hitting
Cassandra clusters of the most commonly
used versions are the eight core systems
so one thing is we all know right since
it's very highly concurrent and
naturally we will benefit from a many CP
codes but if you look at increasing or
rather scaling vertically where you're
adding more and more CP cores in the
system the price automatically goes up
of your hardware systems right so if you
want to write get the best price
performance ratio the recommended
configuration for a Cassandra cluster is
eight CPU cores or rather the pump
commonly most commonly used version is
HCP ORS virtualized machines yeah there
are many mechanisms what you can
actually do using VMs and workers almost
all the cloud providers actually follow
that model a network coming stability it
has to be mainly reliable and absolutely
redundant network interfaces because you
don't want the communication between the
nodes go down which automatically
creates a problem and which we all know
we have seen how the amount of amount of
communication it happens between the
nodes inside the Cassandra cluster those
gossip sort of fronts every second and
so hence it has to be absolutely
reliable and redundant as far as the
network interfaces are concerned it
should handle traffic without absolutely
neither without any mortal necks
so now this capacity planning what is
the disk capacity planning so this
capacity learning when you look at it I
think the two important factors that
when you talk about is the user data
sets are basically the application data
and the usable disk capacity so
basically if if I pick up a disk of say
one TV or one terabyte how much of it is
actually can be used for application
data or basically how much is the usable
disk capacity and how much is basically
for them internal overheads or for
multiple copies the memory stuff getting
written into the the disk based systems
so you have mem tables which are already
there and they get written to say this
is tables and his systems have to be
combined and put it back for compaction
while doing the compaction to analysis
table so there's a lot of activity that
happens internal to the Cassandra notice
or is it the amount of disk capacity
that is usable or rather usable for
application purposes will be much less
than what is the actual disk capacity so
wheel its which understand how do you
calculate that part okay so usable disk
space per node in the number of nodes is
basically the useable disk space right
the per node number of nodes gives you
the total useable disk space in your
cluster so assuming that you have
something called raw capacity which
means like you have say you pick the 1tb
discs so into the number of discs say
for example you have five discs so your
total capacity rocket battery is about 5
TB now there is a basic formatting
overhead which is about 10% so ready
using the formatting overhead by 10% you
will basically have 90% of the formatted
disk space now formatter disk space
actually has to be multiplied by 50% the
compaction and the repair disk risk um
which has overhead of about 50 percent
which means that you are you go to
useable disc spaces exactly half of your
formatted space now assuming that you
have started with 1 TB as the raw
capacity so because of the 10% gone you
are talking about almost like 900 GB of
the disk
is in the format of disk space level now
out of 900 GB we are talking about only
50% being actually usable disk space for
the application which means that if you
have a 1 TB disk then application can be
you plan only 450 GB for the application
the rest of the disks is actually used
mostly for the overhead overhead as well
as all the compaction inter pares that
could happen inside the system and
etcetera so you should always plan for
45% of your disk space actual raw disk
space as your application so in case if
you have if you want to support say for
example 10 TB of application data what
you are talking about is basically
always like lawsuit 2020 nodes with 1 TB
disks right because just about 450 GB is
what you can use per node so we are
talking about close to say 20 or 25 for
kind of 4 nodes with 1 TB disks can
would be needed to actually support 10
GB of application data so we are so in
the releasing the 45% of the actual raw
capacity is what is usable for the
application and the rest of the things
should be left for system to work with
ok so coming to the node config options
what are the various different types of
node configuration that you can do
inside the Cassandra Giambi the way the
settings that you can do inside a node
configuration can be broken down into
four different parts but we call it as
gossip settings storage settings
partition the settings and snitch
settings we already looked at some of it
but she basically but we will probably
spend some more time just to understand
the things in more detail first historic
settings we already looked at it what is
commit log directory and the data file
IO trace so the default one is garlic
cassandra so that means to change in
case if you
want to really go and write it to a
different particular folders because it
actually has a default folder of
Alexandra but in case if you have any
because Varla Cassandra is owned by root
you want this change it you can change
the commit log directories the data
boiler trees and then the saved cashes
directories those three things for you
to actually control the storage of your
node Cassandra okay it is the storage
settings perfect now when it comes to
gossip settings gossip we all know read
it's basically talking about one is not
the listened address the IP address of
the particular node which the other
Cassandra node will use to connect and
communicate so one is naturally the
listen address the other one is a
cluster name it is actually the cluster
is name is basically signifies the
bigger entity into into which this node
is a part of so it's a it's a mechanism
of grouping a set of nodes into one
single cluster so the cluster name
actually all nodes which belong to one
single cluster they all have the same
class name similarly there is something
called initial token now this search is
actually gone away inside there they use
something called virtualized nodes that
we'll talk about it but so the initial
token basically says what is the
responsibility of this particular node
what is the hash that it is responsible
for which tells basically the partition
the particular data partition that this
particular node is responsible for is
actually indicated by the initial token
parameter inside the configuration the
storage board basically the internal
communication port it's something which
is defaulted 7,000 should be same for
every node in the cluster absolutely
same to everyone because everybody talk
to each other and so they need to have
one common mechanism one common port
which they will use to connect to each
other and talk coming to seeds seeds we
already understood what feeds our seats
basically is a list of all common SAP a
comma delimited list of No
bless no die piece which basically help
in the bootstrapping process so in case
if a particular new node is brought into
the cluster or Rob and it needs to
obtain the metadata so they'd like it
looks at this list of seeds and then
picks up one of the node from that seed
list and connects to them and obtains
all the metadata and this the list of
feed should be exactly the same in all
the nodes every node inside the cluster
have to have the same list of seeds okay
and it should always be including a node
from each data center absolutely because
in case if a particular say do you have
multi data center deployment and in case
a particular node goes you have added
your oddity or adding nodes in different
data centers there it would rather go to
a seed which is local to the data center
than actually go to a seed which is a
remote of the data center which is in a
remote data center so that's the reason
why you should have in the seed list one
node from each data center at least okay
this is cause of settings partition s
there are different two types of
partitions one is called the random the
other one is a byte order so what does a
partition do it basically determines how
the data is distributed across the nodes
the random partition R basically makes
sure that it's absolutely consistently
distributed across and the load is
distributed across the node which means
that it's absolutely balanced it takes
make sure that the load is properly
balanced inside the cluster so if I have
five nodes it makes sure that the data
is absolutely distributed across these
five notes in equal equal part equal
numbers since like say if I have say 50
records and I have ten nodes so it
basically makes sure that the
distribution is almost close to five per
node in the cluster random of
automatically makes sure that it is no
round robin fashion it makes sure that
it's actually
rightly balanced inside the cluster
whereas in the case of byte order
partition
so how does the runt okay bite order
partition is something which is a
special which is meant to be mainly used
when you are actual if you absolutely
know about your data and how you will
actually organize it and what are the
different types of queries that you
would run all the information if you
really know then you can use byte order
partition we will talk about the biota
partitioning immunity okay so how does a
random partition process will or work it
basically gets a rocky so obtain 0 slow
key it generates an md5 value md5 hash
of the Rocky and the resulting hash hash
value is restricted between 0 to 2 power
127 because the hash size the total
hashed size which X Center cluster uses
to store the information is about 128
bit higher so the total range is 0 to 2
power 127 so it will make sure that the
hash is that is generated is restricted
in that range e then it obtains a token
associated for each node 7 in case if I
have three nodes I basically create
three different tokens which means 0 to
say 1/3 and 1/3 to 2/3 and 2/3 - 1 so
you create basically three different
partitions of the data and obtain the
hash associated with your 1/3 value and
then assign it to the new one
similarly for the 2/3 value into the
node 2 and then the one basically will
be assigned to the third map assuming
you have three nodes in the cluster in
this case or it is basically showing it
0 to 2 power 42 is the first node and to
power 42 2 to 4 84 is the second for
second node and to power 84 to 2 for 127
is the third node so it basically gives
you a tool which you can use to
calculate the tokens if you really know
so in case you try if I know when I am
deploying a cluster that the number of
nodes are say 10 then Cassandra actually
provides a tool as part of the
distribution itself where you can say ok
my total number of nodes in my
deployment actually his of data centers
two data centers or one
data center and what is the number of
nodes in each data center if you can say
it automatically gives you the token
value calculated and then you can just
take the token value put it in the
configuration and start the node okay so
that's a very random partition to
processbook
okay so random partition basically
heists advantages naturally even
distribution because the data is rightly
balanced inside the cluster but the
disadvantage is wise it's inefficient
range queries because say if I have data
which is written into say 50 keys that
you want to really write inside the
cluster so you take say first 50 keys
and write 1 by 1 then the way it gets
distributed is the moment you write to
your first key it basically puts it into
the first node the second key goes to
the second node the third key goes to
the third node etc so it tries to
distribute written an even even manner
because of which if we have to run say a
query and from say 1 5 it basically has
to fetch the data from 5 different nodes
and not really from one single node so
that's the reason efficiency wise that
if you are doing range queries it's
comparatively inefficient but where it
looks at the overall load inside the
cluster it's actually evenly distributed
in across all the nodes in the cluster
so that play random partitioner is
probably the default and the best one as
far as the overall distribution is
concerned but if you are really doing a
lot of range queries or rather your
application needs a lot of range queries
then probably random distribution is not
the right way for you or application
coming to byte order partitioning the
might order partitioning the concept is
simple where your tokens are actually
calculated manually this is like say I
have I have to store data of say 0 to 1
million keys so and I have set in nodes
so you want to basically say I will
actually break it down into say 100,000
so each node will have about 100,000
keys assuming that I have 10 nodes now
the difference is since I have made it
in such a way that it's zero to 100,000
is in node 1 so all data which gets
written within that all updates
associated with those keys
we'll go to node 1 and then the rest of
the appropriate leads map inside the
cluster so because of which in case if a
situation comes in like you write a lot
of data and a lot of operations are
actually happening on a set of people
which is probably are whose IDs are say
from let's consider instead of 3 300,000
to 400,000 is what it happens is they
are the set of people who are actually
have the maximum number of updates so in
that time what happens is that becomes a
hotspot a particular node is getting too
many updates because you have done
you're my daughter partition where you
have actually controlled you have
actually calculated the tokens manually
and you have done a manual assignment on
the nodes say for example that it here
it says 1 to 999 so and then you're
basically broken down your stuff etc
into each of them having about 100 keys
ok each of them over 100 keys
now what happens basically is to assume
that you have a huge number of data
that's coming out of your 10 million
rows you right into the consent of
cluster so assume that data is between
300 to 400 data range and only 10% data
is between 500 to 505 not one to 600 and
then 10% is the rest of the data the
observation basically will be that all
data will get returned into node 4 which
means almost like 8 million rows will go
into those 4 because almost all the
updates are anything that's happening is
happening on the data which is from 301
to 300 which automatically creates a
hotspot that node becomes absolutely
slow even though you have too much of
capacity available in the entire cluster
you cannot really use because the node 4
is responsible for these keys and
because of which hotspot gets created
the cluster becomes very very slow the
node becomes very slow it cannot take it
takes more time to run the queries and
respond back where even though you have
a huge amount of capacity inside the
cluster which is unused
so this is a problem with respect to
byte order partitioning and again what
happens when a node is added in the in
case of the following random
partitioning or the bye-bye daughter
partitioning okay so this is exactly
what happens so in the case of Dragon
partitioner basically a new new token
range is quickly assigned because now it
knows that it knows that there is a new
node into the cluster so that will occur
automatically make for partition s4
partitions and quickly assigns the new
partition the fourth partition to the
new node there is absolutely minimum
amount of data moved from one of the
doors to the new node which is now rest
it is responsible for basically now so
the new partition er is actually having
some responsibility so all the data
associated with that responsibility will
get transferred to it but when it comes
to the byte order partition s first
thing is you have to add it manually
manually and you have to calculate the
token size manually so you have to
calculate the actual toker information
and then place it in whichever place
that you want whether you want to place
it between 3 & 4 or 4 & 5 you have to
define calculate the range and manually
add it to the cluster this is exactly
what happens in the case of byte order
partitioning so how does the replication
work on pretty similar lines so you
hopped in the hash and the hash is
written to particularly node first
choose the node in this case the green
node so the green node is you're
responsible node and then it will use
depending upon your replication factor
affecting two or three and then
appropriately using the strategy so the
strategy is a simple then it will find
the replicas as the email takes node in
the cluster
traversing in the clockwise direction so
in this case the red node and then the
red node actually will be the place
where you will store the replica okay
pretty much again the similar stuff
which we talked about simple strategy
naturally is the clockwise nodes there
is absolutely no consideration for rapid
data center location in the case of
simple strategy but in the case of
network topology strategy it
automatically takes care of that one
second okay so the simple strategy to
put up all these strategy this for the
sake of because we talked about
administration where it all these things
actually be part of its let's recent
being just mentioning here this
assignment is absolutely based on the
partition strategy so it will always try
to find a different track for placing
the replicas so in cases for no other no
racks are available it automatically
falls back to the simple strategy and
uses the clockwise mechanism that's the
reason why now the new mechanisms in
saying that the default one is at hook
to Bali strategy because it falls back
the simple network strategy in case if
for you don't have multiple racks of
multiple data center deployments whereas
in the earlier case it used to simple
strategy because you was the default one
now its network topology strategy is the
default one so in case if it doesn't
find any racks or new multiple data
centers it automatically falls back to
the clockwise mechanism of simple
strategy and as a result it's become so
it is the default strategy that used in
cider is under cluster deployments okay
coming to stretches so what is a sneaky
snitch basically is it determines that
will delay relative forced proximity as
I said it's something which helps in the
better routing inside the between the
nodes in the cluster so its gathers
information about you know network
topology so that it can route the
requests in a more efficient manner
between the nodes that's a main purpose
of the synergy there are different edges
that are there and the default one
basically is a simple switch which is
used in the case of general single node
single data center deployments and
simple switch actually follows a pretty
similar kinds more than love it knows
that it's actually you're following
simple strategy so it notes how to how
to actually locate the nodes because
you're talking about clockwise direction
also taking the same data center within
the same rack most often than not so
it's simple switch will do the job for
you but when you have say multi data
centers and multiple tracks and etc
there are different types of switches
called a rack and data center Yvette's
match etc that will be used and also
which is suitably compatible with any
replication strategies that use it so
hardly matters which is replicable is no
strategy that you use if they are all
compatible with everything what are the
different types of switches if you look
at it these are the various types of
switches that are available one is a
simple switch which is a default one and
always used in the wicked comfortably
uses same concept of walking clockwise
through the nodes for routing
information but when it comes to rack
inferring switches it automatically
tries to place the copies on different
tracks in your data center very
important because it tries to make sure
that the rack inferring switch will make
sure that the talk copies are placed in
different tracks so that you have rack
failure can be avoided so in case of
sorry
you
in case of a rack failure you still
don't have don't lose it and the copies
are still available for the tools of the
request inside the cluster so Rock
inferring switch actions Mitch actually
uses the location of the racks and make
sure that the copies are placed in
different tracks it uses the IP to infer
the network topology and the second unit
of the IP address is used to identify
the data center we look at this actually
there is actually a file called
Cassandra topology door properties which
is what we called as a property files
which allows you more control when using
a rack of your strategy so it says
something called Cassandra topology door
properties which gives you information
about the location of your replicas but
other the nodes in the cluster and their
which data centers they actually belong
to so all this information is actually
provided inside a file called Cassandra
topology two properties and there is
another level of the city called easy to
snitch which is very specific to Amazon
prompts a mass M is on elastic clouds
the advantage basically is in the case
of Amazon the location of your VMO
whatever is available to you it keeps
changing because you don't really know
the the physical part of it that's the
reason easy to snitch actually uses
Amazon's AWS base API to determine the
actual location of your node in the
cluster and then uses that IP is to
actually route it efficiently inside the
Amazon plastered that's the reason easy
to snitch is used when you are setting
up your cluster inside the Amazon ec2
actually we use it inaudible our
deployment actually runs completely our
old miner than my old company the entire
deployment runs you should run inside
Amazon we have bought some 2,000 nodes
or something that we're inside the in
Amazon cluster and all of them market
use easy to snitch because which really
helps in the
locating as well as efficient routing
inside the easy to clout okay cool so
casino deposited properties look
something like this where it says the IP
the IP address of the node which data
center it belongs to and which racket
belongs to so this is the format in
which it is written inside Cassandra
top-10 properties
the first one is IP address second one
is a data center third one is the drug
details every node in the cluster should
have this file and this file should be
exactly the same in all loads is the
Western ok
the first one is IP address second one
is abc1 or TC to easy data center the
third one is the rack corner rack to
details so coming to monitoring the
center so as we said Cassandra Cassandra
is a java application right it's
actually something which runs incidents
on inside a JVM so Cassandra provides
you java management extensions which
sort of helps you in there monitoring
the Cassandra system pretty well so it
provides you and I'm sure anybody who's
worked with Java who know what JMX means
java management extensions actually is
the mechanism it's basically a sort of
stubs or something like books which are
provided inside your application which
you can use from the external world as
an outsider I can connect to these hooks
and obtain useful information about the
application runtime so that's the
mechanism which JMX provides
out-of-the-box it's a very commonly used
for variety of java application controls
application control operations in in any
of the java applications so for example
low available memory detection thread
deadlock detection six cetera so you
want to understand about the class
loader tracings log level controls that
money and even provides you some
management capabilities where you could
afford some operations on your java
application using your hooks so the
hooks are actually provided in such a
way that you can run an operation on
your application directly from there you
can do that I'll show you how do you
actually work with that when it comes to
Cassandra simplest way is to another
option is say there are other lot of
tools available in the commercial world
one one thing which is available in data
taxes it provides you something called
an op center so you use op Center and
you can actually monitor the clock
Cassandra cluster comfortably from there
you can actually even do a lot of
management activities we
you could add remove notes from the
cluster and etc so in addition to that
there are some open source tools like MX
poor J and etc which could be used to
monitor or your Center clusters I will
show you how do you actually work with
the something called J console which
when you are coming to management
extensions so when it comes to
monitoring what are the key attributes
that you need to track power column
families there's something called read
count read latency a write count write
latency and pending tasks so when you
when you see that the queue of pending
tasks is increasing which automatically
means that the node is able not able to
pick up your tasks and it's able to
process it which automatically say set
you know good is going to get slower and
slower and slower so your read and write
latency will tell you exactly what is
the time it's taking to processor read
request or a write request okay so those
are the other set of attributes that you
really want to monitor on a per column
family basis now maintenance so what is
maintenance wind is this basically when
you want to actually say update
information about the different
statistics which are there inside the
node you want to obtain the ring
information of how the ring is
performing what are the total number of
nodes which are alive you want to bring
down or bring up a node you want to
create snapshot you want to create do
some load balancing and some basic
maintenance has to like you want to say
just get your thread pool statistics you
want to get your information on what's
happening inside the node so in all
those cases you want to have some
traffic tool so this end up provides you
our tool called node tool so the node
tool actually helps you to achieve for
all these things it looks at your
cluster understands and modify its
activity it gets limited statistics
about the clusters you can see the range
each node maintains
you can do moving data from one node to
another you can decommission a node you
can repair a node that's having trouble
so no to
something which is available by default
in the open so distribution itself ok so
just look at some of the things now okay
so there is something called info so if
you say node tool give the IP address to
which you particularly connecting and
then say info it gives you information
about your drink some of these things
you are changed in the new versions very
called state as the next cetera but will
remove will run some of it and then
we'll see what it provides what is the
kind of information that it provides an
Exeter okay so you pass the address of
the door to which you want to connect to
and then obtain the information about
the ring the info is the command
similarly you have commands for
different different things like if you
say ring it gives you the ring
information where you're talking about
basically the different partitions which
are there and what is the status of each
of those partition or the nodes itself
say the case of right now it's like if
you say drink it basically gives you
here it's showing up with the physical
address physical IP address which is
actually is the partition that it is a
physical node shapes and it's saying
what is the status of each of those
partition it's up or down in the case of
virtual nodes it actually shows many
many partitions like this inside the
cluster I will see some of it anyways
yes we will see in practical
then it
two different types of charts what we
call it as CF stats and TATP stat CF
starts basically means column family
statistics and AP stats is basically the
thread pool statistics so you can
actually see all these information using
note 2 and understand how the system is
performing what is the information that
you have available so how is it
performing today and what is the what
are the different types of information
that it provides as part of these
statistics we can look at all these
things similarly it can give you when
you say TP search it gives you
information something like this so we
said this is because of sada right so
there are many thread pools which are
there inside the ok Center cluster so
one is like file utils messaging service
pool stream stage response traces there
are many many different types of the Red
Bulls that are there and it shows what
is the number of active threads inside
the pool what is the number of pending
requests and how many have been already
completed so far by the thread pool so
it provides you information on these
lines when you run the thread pool
statistics so on the basic wintry stuff
you could do repair flood cleanup
etcetera so there are another set of
commands which are provided by the node
tool prepared a city is when you want to
repair a particular node because it's
has some issue or it's not really having
a proper synced up inside the cluster
and you want to basically repair and
make sure that it is back to normal and
everything is working fine with it so
you can use repair which actually makes
causes Cassandra to execute a major
compaction make sure that everything is
all the unwanted stuff is thrown out and
then it makes you heard that it is
actually back to normal and working
together as a normal node area that
normal node in the cluster
so you provide a key space as an
argument when where the compaction is
required and then you run it on a
specific host connecting were specific
host ok so similarly you have flush
flush actually means that you actually
flows your mem tables to the disk so
this is another very common process
that's used by the administrators is
whenever you want to really do some
maintenance activity in the cluster you
don't want any in-flight data so
basically something which is sitting in
the memory so you don't want any
in-flight data so you could actually do
is you make sure that your run flush on
the system so that everything is flush
to the disk from memory and then
afterwards you do all your maintenance
activities removing adding an old and
etcetera all the stuff you do it as
opposed to that one only similarly there
is something called cleanup that you can
do a lot of the maintenance activities
generally use where you want to change
the replication factor of the
replication strategy so OH
the purpose is say today I have a
repetition factor office at 3 but I over
a period I felt that my replication
factories from 3 I have to change it to
2 then you can run you have to run a
cleanup but ideally you would rather not
want to do it because but if it's an
absolute leap then you can do it using
clean up you don't change the
replication factor later because it's in
a great simple right so if I have say 3
replicas which I have been maintaining
all along and then suddenly if I change
it to 2 which means that my third
replicas to be discarded of percent are
clustered so that's the reason it's
called a cleanup ask where it ends up
cleaning up the third replicas and
making sure there are only two copies of
all the data that's stored inside the
classroom this might take a good amount
of time to make sure that some of these
things are done but it surely provide
you an option to do it that's what it
means here ok so
it flush and cleanup could be are under
certain teach commands that are used for
mainly one is flush for flushing in them
tables to this table to the displaced
your presentation and cleanup is used
when you want to change your replication
factor replication strategy so you have
so that you sort of clean it up and make
sure that it's actually taking care
across the cluster in all the nodes okay
then this ramp shots another very
important data backup mechanism so in
case if you want to say that's it what's
the purpose basically you make a copy of
some or other keef pastes in node and
save it to some different place so that
you have a separate data place file
which you can use in case if the node
place say for example if the node goes
down and then you can actually restore
it to one of your previous snapshots and
then just allow it to sync up and across
the replicas and make bring it back into
the absolute sync across the cluster one
our previous snapshot and then that one
our previous snapshot basically will be
out of sync with the set of data rights
of that data it will get synced up in a
shorter form so as you've met assume
that you do not have a snapshot so if
you do have a snapshot basically it has
to sync up all or the entire data right
it has to sync up the entire data from
the other replicas in the cluster so
that takes longer
so because of which snapshots are very
very useful and the way you take
snapshots is re using just the snapshot
as a keyboard or you want sorry I mean
for the entire node or you can take it
to a single key space using - the key
space name followed by the snapshot
keyboard on the notary and when you want
to restore you basically have done the
node you remove the oldest a stable
income o'clock copy the data from
snapshot directory to the regular
directory and then to restart the node
and then you automatically restores it
to that particular snapshot and then it
will start running continuously from
that also snapshots as I said this is
very very important and it should be
used in
okay okay so then coming through
decommissioning a note when you want a
reality commissioner these are the steps
that you need to follow so the mechanism
that actually found it it follows is
there you could actually do using node
tool node tool and say you can say
decommission a particular node as
command then what happens is it happens
in this sequence the operations are
happening inside the model in the
sequence first it shot down shots down
the gossiper because third it doesn't
receive any of the data from the other
nodes then it brings down the messenger
the messaging service is shut down and
then it brings down the state or manager
so that all you have thread pools and
everything that are there for the
different stages these are all brought
down then it marks the status of the
node as decommissioned and then it it
also as far as the rest of the cluster
is concerned this particular node does
not exist anymore so the node is gone
and then you have only two nodes in the
cluster in this case in Otay BC so the
big is gone in the a and C are the only
two nodes where the partitions are
actually readjusted to that then da de
da is occupied in these two nodes
instead of the third node which was
supposed to be that what's the process
of making a database snapshot across
different nodes so there is no
specialized process as such so you have
to take your snapshots on a per node
basis it's not really across the across
different nodes there is no way to do it
across different nodes the node tool is
basically for your particular node so
you have to take snapshots of each node
separately there is no simp
of doing it across all the nodes in the
transgender so it could be something
like okay you probably could write a
program or some sort of a process which
actually goes picks up all the nodes and
runs on each of them as a remote process
or something and takes a snapshot of
everything that can be done but it's
basically an automated mechanism that
you would actually create but but there
is no direct way of doing that in the
crystal so the node tool actually works
on a per node basis so you have to do it
on every node performance journey let's
talk about it in detail okay okay so
first let's go back to running some
practicals and then we will come back to
performance queuing okay so we looked
chicken soul and then let's run some no
- so it's in the bin folder okay after
saying low close because this is my
running on the local system if I just
say status it gives me the status of the
cluster here what is it saying is it's
basically saying I have three nodes in
my cluster and the partition ownership
of each of them what is the total load
what is the number of tokens so the
tokens if you see it basically 26 is
what it says which means that it's
actually talking about virtual nodes
we'll talk about what virtual nodes are
give me a second okay so then it's
saying what are the racks that reduces
because it's simple deployment which is
only single there is no specialized
racks which default this rack one and
when you say un-un mints its appended
normal so it's the status which is
showing here so if you see it's means
it's up and it is a normal so this is
basically the way to monitor your status
of the nodes in the cluster so what I
will do basically is now I'll bring down
one of them okay I'll bring down one of
the nodes just to see how it behaves
pixel if I run this again so it says
down just down and normal so it is up on
or down a normal in case if it's in a
bad state like it's actually leaving the
cluster like if your decommissioning or
your basically adding a new cluster then
you'll have other states likely ring
joining and then if you want to move
when you are moving data from one to
another one from once a node to another
node it will give you the other status
is called moving joining and leaving
there pretty much something which will
be you seen in the the practical
scenarios right now I just brought down
the instance so it says normal so if I
restart the node it will basically say
it's up and not on again so right now
it's down still because it's not started
so it's still down short started now
it's up and normal okay all of you
getting it right
okay so similarly instead of the status
if you say a ring right so ring shows so
many things see did you see that it
basically shows huge number of you four
things here but if you look at it what
is it showing end of the day if you see
exactly what it is queueing one second
okay okay so it says it if you look at
it it's a zero the same IP address 1
2011 zero zero three then one twenty
seven zero zero one and then one 2700
two etc etc so what's it actually end of
the shoot day showing is it's basically
showing a virtual partitioner so each
node has about 256 virtual nodes and
it's showing all the virtual nodes in
the ring and how does it work okay if
you just go back a little bit just to
understand and show you what happens
okay so this is the way it works
basically okay ah come on second I think
I have a presentation once again where
am I
okay so if you look at it here the
concept is basically is as long as right
now when we create a ring week week ring
we create a ring with the node as a set
of partition s right so I have say ABCDE
I have like say a whatever does not
matter node 1 node 2 node 3 node 4 node
4 I have node 6 that sir the default
mechanism which is at the top
now the actual virtual nodes what
happens is each node is by default
broken down into 256 virtual nodes
that's a default number but you can
reduce increase whatever you want
whatever is suitable for you know you
are need you can do it it all depends on
how much amount of partitions that you
really want to do okay so so what
happens is each node is broken down into
smaller smaller pieces so what is the
cluster the cluster looks like something
which is similar on sure that the
below-the-line
so the cluster basically has no small
small partitions and it's formed with
the small partitions but right now there
is no still no advantage because each
node the partitions are still together
right so because of which you are still
representing the set of partitions as by
using the physical node itself so what
happens next is it actually does a
shuffle shuffle at the partition level
so when it does say sheriff let the
partition level so what is it doing it
basically shuffling it partition develop
now each node is basically broken down
and then it is actually shuffling the
virtual partitions in a way there is
that dependency on the physical node for
a specific part it is gonna be no
because all all your data is actually
dealt with at a much smaller granular
level which is basically the 1 by 256 of
the actual partition in a way because
you are making each partition into 200
take small small pieces now this is the
advantage basically when you make it out
into this set of
virtual partitions or virtual nodes
whenever the node goes down the amount
of time it actually takes to bring it
back ease is it's much faster because
you are talking you are dealing with
smaller partitions and smaller
partitions are not really all of them
are not really the two master copies
here in a way right so they can actually
do shuffling back and forth very easily
and somebody who's not a master copy
who's just basically a replica of
another one they could actually get
synced up with the appropriate replica
in the other node and etc it's like
you're dealing with data at a much
smaller chunk which means all your copy
recovery everything that you are really
working out or rather manipulating does
not have any physical dependency anymore
and it's easy to work with that because
the chunk is size is reduced quite a bit
so that's the advantage of virtual nodes
and by default it's virtual nodes right
now so each node basically it shows
users 200 and physics as the default
virtual nodes but if you really want to
don't want to use virtual nodes and you
want to go back to the the physical node
concept you can always go and change in
the configuration from num tokens the
configuration parameter is called um
tokens where it actually say is 256 as a
default value so instead of that you
take it off and put it as initial tokens
and it will go back to the node wise
clustering mechanism which was the older
mechanisms but since virtual nodes is a
much better feature and it actually
helps you to shuffle the data much
better inside the cluster and it helps
you to deal with data in smaller sizes
the smaller chunks which surely takes
away the physical dependency as well as
it really helps in recovering because
recovering or any of the other
operations because now you are operating
on smaller chunks then that is the
bigger chunk which is the node level
chunks make sense this is the virtual
node concepts so here what it's showing
is basically the virtual nodes okay so
now let's render
her ring we did we did status we will do
what takes evasive stats so column
family statistics it shows all the
column families in this matrix so this
is these are the things that you chose
so for example I have a key space called
roads it is what does it read count what
does it read latency what is it right
count right latency and the pending
tasks so it shows information about all
these things on a per key space basis
well there's a compacted partition
maximum by its minimum by its local
right latency local read latency mem
table data size so it basically throws
information about all column families
statistics in the entire system on a
parking space basis so if you get at cql
hotels it's showing the estes table
compression ratios see here if you have
some this space used space used a stable
compression ratios number of keys which
are there this is just basically a
example that I have used for running
some CQ research you look at it border
the bream what is a bloom filter space
used compacted partition maximum minimum
bytes mean wide set cetera so all these
information is something which is
provided on a per key space basis will
you be able to access each of the
virtual notes directly what do you mean
that if you order that you want to do
actually that's the question Sneha so
what what is the purpose what what do
you mean by able to access each of the
virtual notes directly see they are
not really a physical entity they are
not tied to a physical entity in a way
but you surely have the token
information right when we ran the
command it showed the token information
right this is the part this is the
partition token hash right so this shows
what is the responsibility or for other
where that virtual node starts and where
it ends on a specific node but when you
say access directly what is that you
want to do what is the constant what is
a your right looking at actually so
you're thinking will it actually enter
tape yeah basically yes right it
maintains would be easier because now
you're dealing it you don't have all the
chunks on the same node in the same host
right because it chunks us smaller
chunks are shuffled across the cluster
so now the cluster is basically awfully
say if I have four nodes you're talking
about almost like thousand 24 small
small partitions and you are not really
talking about four four big partitions
so naturally smaller partitions are easy
to manage because when you are copying
it across the nodes or when you are
actually recovering the repairing all
your stuff that you are doing you are
doing it are much smaller sizes that's
the reason I said the maintenance will
be much easier make sense so said CF
starts be sure we saw what it shows then
TP stats this talks about all the thread
pools inside the system so TFTP star
shows exactly similar to what I showed
you so it's on a pool name active
pending completed blocked all time
blocked so if you see any of these stuff
actually blocked northern black you
clearly know that something is stuck in
system and you need to clear it so the
thread pool statistics shows something
like this and then it is truth about all
the message type something like what is
the number of requests response messages
which have gone to system what is the
read repair range slices requests that
have gone into the system
the mutations basically the number of
queries that you have combined together
and if there is anything which is
dropped out of them something which is
already made but it's dropped so it
shows all these information as part of
their TP stats and they're all the
different thread booths painted hand of
everything is a stage as I said
everything is in the entropy staged
migration stage so each of them is a
stage and each each of them has its own
thread pull and the thread pool is
actually showing the information sake
gossip actually has a maximum here
because gossip naturally does so many
things because it just talks to the
nodes on regular basis then they have
not done lot of work on this cluster the
gossip is still a big number because
it's whenever you bring up that loss
methods actually uses it runs a lot of
gossip messages back and forth which are
again handy with a concept red buoy okay
so that is TP stats okay let's do one
thing let's run this to see what are the
various other things that are available
okay so you look at it just for the sake
of understanding and actually they've
improved the note really quite a bit
from what it was earlier because now it
provides you too many options which can
actually do which you can do so all the
will will come and say something called
see a histogram see of stats clean up we
talked about clean up compact you want
to run prompt action directly on a
specific key space and column families
you can run it suppose if you want to 44
set okay compaction stats yeah this is
another interesting stuff that you look
at then you sense it described cluster
those are the thing which it actually
talks about it prints the names nature
smart meter it lets you look at some of
those things okay and then yeah it has a
stable strain to put got save info if
you look at it okay let me assure on
some of it okay
gossip info so this talks about the
gossip info with respect to this
particular node with other modes so if
you see this this is basically what it
maintains on a per node basis what is
the status of each node is it normal or
normal normal which data center which
vac what is a severity host ID RPC
address release version of Cassandra
that it's running so our information is
actually maintained inside it's gossip
information my paranoid basis
similarly you said Compaq in stats right
okay so compassion start shows about me
it shows basically pending taxon that's
an active compaction the remaining times
in case if you see any of these traffic
means that your compaction is running
right now and there is something which
is pending for you to run the basically
if it's about to run the compassionate
is running the compaction and most often
than not this is one case where you can
actually check if the particular node is
even its particular node is very slow
and it's performing really at a much
lower level than what is its actual
capacity so some of these things could
be give you information about what is
actually happening at the time so in
case if chose pending tasks and some
active compaction remaining time as some
values inner automatically means that
it's actually running compaction at this
time because of which you might see some
performance issues in the system okay so
then what else we said described cluster
let's look at the correct class
blastocysts
the name of the cluster is stress
cluster it uses dynamic endpoints nature
then partition wise it's no more three
and schema versions these are the three
nodes in the cluster and that's the
version of the schema that they all of
them are running okay
and then what else what are the other
interesting stuff that we can look at a
ring we looked at the pair and all is
not going to work right now because it's
you cannot do any of the stuff here
because I have to bring it down and I
have to create some bad situation and
then only you can use repair and etc
which is not a straightforward thing
then you have charts okay snapshot by
just to snapshot it goes and creates
snapshots for all all all keys bassists
so it is created a snapshot all key
spaces so if you could go back bar lip
sandra data so which one has some data I
don't know okay and if you guys look
totals please have some data is it this
laptops it writes into this muffler
oldest reach of them see this is what it
is the created one right okay see this
here in this one are the snapshot
directly that it's created is basically
the it's actually names it in line with
that one okay so you look at for this
particular number go to this particular
type three and this has a snapshot which
is the latest is something that you have
created so it is created its own up
short for each of the devices so all
these databases are actually the ones
which are for Italy created a snapshot
and put them into this particular folder
okay so similarly would have done for
everything because I didn't give any
specialized thermo partitioner is the
default random partitioner which the
Cassandra uses that the latest version
of Cassandra 2.0 onwards they use what
we call it as super or partition which
is nothing but a random partner but
murmur is basically an algorithm which
is used by even distribution it works
better than the old random partition
which they which Cassandra is to use and
right now the moment partitioner easter
is the default randomized partition err
that's used inside the center okay so
here this is the snapshot okay so go
back to Sandra okay not - no - okay
Oh snapshots we did and what else do we
have anything interesting that we can
look at oh okay so this is another
interesting one but we don't do it right
now something called scrub which
actually rebuilds as a stables from for
one or more column families here you can
use it if required when you see that
something is not really in place you
could use it for scrubbing refresh
remove a remove node is one thing which
use to current node more so it basically
forces the completion of everything and
then remove the provider ID so in the
set the existing node pool the new one
which is actually the which comes with
2.4 unknown words provides your Lord of
options which earlier version never knew
is to provide
so describing describe ring shows at
open ranges inform given key space etc
etc and there is another interesting
stuff so you see there's something
called enable auto compaction etc for
the key space column families you could
do this also here enabled backup any
will gossip or disable gossip you can do
all this stuff if you want to really a
basically provides you very minutely
well control that you can do on a
partner base Isis tables compaction
throughput and then etcetera holster
them set get threshold cash key store
these are probably the ones which are
interesting but you can try each of
these things offline and then you can
look at some of the information that
that is one which is related to that and
then describe what we set this ring hi
what happened we did describe cluster
and described ring once right this works
so what does describe ring okay it
didn't work am i typing that comma
right first scribe ring and keyspace
okay describe ring and say key space
hotels okay so this gives you an
absolute token range of the key space
pattern order so on one 27001 what is
the token available star token in token
the afternoon point which is happening
which has information about this key
space etc in the entire cluster how is
this particular key space is distributed
inside the cluster this is like the
virtual note which you talked about
right so you have a star token token
very similar to this you will have
weight per per virtual note there is
something called a start and it's
something called end which is the
responsibility of that virtual note okay
so yes
yeah and decommissioning and all I'm not
going to do right now because those are
more of I want to pull out the node and
etc from the cluster free okay
something called info right or info
gives you this kind of information so it
gives you the node information basically
token the ID is a gossip active is
active or not they do transport layer
generation number which is basically an
ID for this particular node data center
one track one key cache rokosz etcetera
etcetera information about this
particular node and it's an ID this is
ID hash ID of the particular node ok so
that is the node tool part of it ah what
else what else do we have
I think the first last and final thing
is the performance tuning so we'll
probably do this end close for today so
on the performance tuning wise right if
you look at it what are the made is very
important things that can affect
performance
so one naturally if you have a bigger
memory which means you have bigger mem
tables and bigger caching no caches and
key caches will be big which means the
overall read performance increases quite
a bit so a bigger Ram automatically
gives you a better performance that's
one part of it and then when it comes to
writes it's always recommended that you
use a separate device so basically if
you have multiple disks that you are
using in the system keep one create
comment locks on one of the disk and the
other rights on the other disks of a
lyricist Able's and etcetera will be in
a different disk so that gives you
basically a better IO throughput on a
regular basis because there's a lot of
write bigger happening backing on each
node so automatically that works better
for you if you have to
print discs that use for your comment
logs and your general writes which is
your stables and mm tables and extra so
sorry mm tables dresses tables etc so
you have you basically try try and
create two different discs one for
coming clocks and one for this one and
then in addition to it the general
performs right when it comes to Java
it's a java application so you have
other things basically your concurrent
threads which you are running in the
system we saw that we mission when we
run the J console it showed that the
concurrent number of threads that are
available of the total size of the
readwrite threads is about 32 which is
most often than not more than enough
room for for most of the general cases
but if you want to really increase it in
assume that you have more codes then you
can raise that to 64 and all which might
give you a better performance compared
to what it is but 32 is more or less
enough because of the sate else's system
that it works that it is more or less
enough for very high loads in general
but if you have more cores and more CPU
power at your disposal you can always
increase her to 64 or not okay in
addition to that the general JVM tuning
and everything will work so basically
you want to you want to basically change
your garbage collection algorithms you
want to increase your heap space you
want increase your row well some of
those parameters could be tuned very
similar to any other java application
because it's a it's a JVM based java
application so you can use all your
general Java performance tuning also as
part of the cassandra performance tuning
itself okay caching memory
separate device for commit logs etcetera
and then the general performance tuning
of Java so that's setting quantities
with respect to performance tuning
Cassandra are there any log files yes
there are lab Foyle's log files
what do you mean log files are you
saying with respect to the general
running of Cassandra that's already here
right so when I actually run it I'm
after running it
for Chrome which means that all my logs
are actually been on the console so when
I run it here they are all sitting here
right there is the logs these are the
logs here but if you don't want to run
it you go right to another log file
that's not a problem but console helps
me a lot better so why I use console
moves often than not anything else okay
this is commit logs and data and save
cash there is no other special law which
it creates to the extent that I know so
there is something called Garlock Sandra
so this is a system log gear this is all
the information associate with the
system this is another log file which is
that which will show information about
the particular thing that you have every
day everything that's actually running
on the system it gets logged into the
system but log so basically how do you
actually give information or rather
punishing to a particular user inside
the cluster so I can actually say
somebody has I've been permission
somebody has just a read/write
permissions summary I can create a
multiple levels of permission per key
space if required because the
authenticators and the system which are
available they can be modified
appropriately so let's show it to you
for now but will will probably run it on
the next class
okay so you look at it sorry what up
okay so we are looking for of contrite
okay so we Cassandra or tml so if you
look at authenticators here so there is
something called the default
authentication which is basically a
lowball authentication Orlando a lower
earth indicator is the default one which
will be used here although all
authorizer and allow our Authenticator
have the default ones which are used but
you can change them to password
Authenticator and Cassandra authorizes
which are indirectly which will give you
a control on different types of
permissions that you want to create on a
per key services and you can actually
create key space-wise per machine just
very similar to the grant that you do in
your AR to be in this one right where
you want to Chris a grant right on
something grand read on something all
those stuff can be done on pretty
similar lines using the on Authenticator
sin authorizers so change the
Authenticator I mean I could do it but
it's to shut down restart excetera so I
will show you in the next class so a
password Authenticator I can change and
then once I shut down and restart
automatically the new node will work on
the password authentication mechanism
which means that if I do not give a
password it will not work me it's not
allowed me to login so all those things
will actually can be done at our
authorization and authentication levels
okay thank you bye
you