OpenNMS 101 - Module 12: Thresholds Video Lecture | A Complete Guide: OpenNMS - IT & Software

hi folks and welcome back to another
episode of open em f's 101 the video
series where we hope to bring you up to
speed on the open NMS network monitoring
platform
my name is tarick a log and i'm happy to
introduce module 12 the final module so
13 modules lucky number
lucky module number 12 is on threshold
emember from the last episode and the
101 series is designed to build on each
lesson so I hope that you followed
through up in this point we created this
class group file to do data collection
on this long object right here that's
1.3 that 6.1 at 4.1 blah blah blah and
we did that to collect on a output of a
script we had done we had run this
script called class collect where we
basically did an output of let me get
rid of that without the mail message or
just echoed a random number so it's
going to echo a number between 0 and
32767
okay so um I want to jump ahead of
what's going on I'm going to create a
threshold here because we do data
collection every 5 minutes so it's going
to take upwards of 5 minutes for this to
fire and I don't want to have to wait
around for 5 minutes so let's go and
check so in order to do thresholds
refresh this so here is our screen I'm
going to go to configure open in ms will
go here to configure thresholds I want
to edit the medicine MP threshold and
group I'm going to discuss this in a
minute and I'm going to add a basic
threshold for our class random data
source and I'm going to set the value to
be 50,000 15,000 and I'm going to rearm
it at 10,000 now I'm going to go ahead
and say no example class random example
so I'll save that as our our threshold
now in order to again get rid of a
chance I am going to edit this 10th
class Colette and instead of echoing
random I want to echo
20,000 because I know 20,000 is greater
than 15,000 so this should fire okay so
let's talk about fresh holding now there
are four basic kinds of thresholds in
open and a mess there's a low threshold
this is things like free memory you know
you weren't a certain amount of free
memory for your devices to be healthy
and so if the amount of free memory
drops below a particular threshold you
want to fire a threshold a bit more
common is the high threshold let me know
in this space is exceeded let me know
when temperature is too high etc now
most network monitoring tools do low and
high thresholds if they do threshold at
all those are the two main ones you do
low and high we also add the idea of
absolute and relative sometimes you want
to know if a particular threshold
changes by given amount for example
before eBay added all these cool little
tools that you could get on your mobile
app to let you know when people bid on
your auction
I used to use open an MS to monitor my
auctions and I would have open an MS
alert me when a particular value change
spot when a particular auction changed
by a penny so I set the absolute
threshold to be 0.01 or one cent so if
the price changed by 0.01 or more it's
written threshold alert a relative would
be a particular percentage you know let
me know if a particular value happens to
double say dis space utilization if
you're at 30% utilization and suddenly
you're at 60 percent utilization you
might want to know about that because
unless you're expecting it it might be
unusual and you need to investigate so
four types thresholds low high absolute
relative now thresholds come in two
types there's basic threshold and a
basic depressional you check the
collected value against a particular
value single value and if that
collective value is higher or lower etc
if it meets one of the four kinds of
thresholds it'll trigger the threshold
again expression based thresholds are
triggered against either a single value
that has had
um math applied to it or against
multiple values for example in the net
S&P agent they have a disk utilized
threshold you just utilized odds Eddie
hey so you can find out it's eighty-nine
percent utilized and sixty two percent
utilized it's just they're very very
nice Oh systems such as say a Windows
system Windows supports the host
resources nib which tells you the total
size of the volume and how much is being
used in raw numbers it doesn't give you
a percentage so you either have to set a
threshold at you know one gig to Gig
three gig etc or using expression based
threshold you can take the amount of
disk space used divided by the total
amount of disk space multiply that by
100 and you get a percentage and so
we're going to discuss how those does
happen also there's some fresh hope and
concepts the idea of a trigger so let's
take it CPU utilization for example I
might want to know if my CPU is at 100
percent utilization because that could
be bad but let's say I have a system
that occasionally hits 100 percent
utilization normally it's around 50 or
40 percent but every so often a base
spike to 100 I may only want to get
alerted I mean we want to get out of bed
because my pager went off or my phone
went off because of this this condition
if it's there for a certain amount of
time so you can set the trigger the
default triggers one so basically hey if
it violates the threshold once boom we
create an alert what you can do is you
can increase that though you could say
let me know if it's two or three times
so in this case it would have to be
consecutive values that violated the
threshold so this way Pacific
utilization if you set it to free it
would basically be roughly 15 minutes
before you would get alerted now we on
this is a some people call it's horrific
sits the idea that okay we violated in
threshold well don't notify me again
until it's dropped below a particular
value so the idea is hey the special has
been triggered but if it's for
right around the threshold you may want
to you know don't trigger me every time
they cross the threshold I have a little
picture to demonstrate that and then
finally remember UV is unique event
identifiers these are the events that
get sent whenever something happens
within opening MS and so we have a
default high threshold VI and we have
the threshold VI but suppose you want to
create your custom a custom yi I oh it
opened in a mess we have a drink machine
that I monitored and I monitor how many
drinks you know I have an absolute
threshold that says let me know if a
drink was sold I'm just weird that way I
want to know if someone buy the coke
someone goes out buys a coke I get an
email and it tickles me to death I'm
like haha some about code so as an
absolute threshold let me know if the
number of drinks in the Machine has
changed by one or more and then I also
have an expression based threshold which
says take the number of cans I've sold
subtract it from the number of cans I
put in the Machine and let me know if
that value is 3 or less so that's a low
threshold expression based threshold
because if we have less than three cans
of a particular soda I need to go add
more cans now getting back to this idea
of a rearm okay
so here we have a graph it's a very very
technical graph we have this the this
value and it goes up and then down a
little bit then up then down a little
bit up and down a little bit and let's
say this red line represents the
threshold so right here it crosses the
threshold so it's cross the special and
it goes up quite a bit and then it drops
down below the threshold and crosses
special again then it drops down below
the threshold some systems um would it
would trigger every time you cross the
threshold so I get a couple alerts here
well if you set up a real on value which
is represented by this green line then
this won't trigger again until the value
drops below the screen line so if this
value were to come back up it would
trigger again but this this is the first
trigger that is the second one little
fire and this this won't happen if we
set up a rearm but because basically
across this line as many times it as it
once but until it's actually rearmed it
will we set a threshold now I've brought
along four enough so let's go over here
and
look at my status alarms and I bet a
ballon Hey look I have an alarm so click
on this guy so you can see we had this
default UEI
which is high threshold exceeded and it
tells us that the class random data
source which is described this class
random example hit a value of 20000 or
the threshold was 15,000 and a trigger 1
this won't happen until it is it won't
happen again
until it is we aren't and so this is our
alarm you can do all kinds of alarm nice
tough with this now I want to reset this
and again I'm not going to wait on
chance I want to edit this and I'm going
to change the value to 2000 which means
the next time it fires next time it's
collected it should collect 2000 which
is considerably less than 10000 and so
it should we on our alarms go back to
our logs here ok now while that's going
on let's talk more about what we did
with thresholds okay in the so you
should be familiar by now with this idea
of packages the polar and the data
collector both had this idea of packages
so for example with the polar you could
collect you could have three different
packages you can say have a gold package
which pulls a lot of services every two
minutes then you might have a silver
package which just does say an ICMP
check and checks HTTP running or every
five minutes on a group of servers then
you might have the bronze package which
just pings now these are these are
things I want to know that they're out
there but I don't really care that much
about them so I'll put them in my bronze
package
well ping them every 10 minutes and as
long as I can ping them I'm happy so
that's an idea of packages and collect D
it's the same idea
so then collect B it has this idea where
I could pull every five minutes and do
data collection every five minutes and
that gives me particular graph but maybe
I have something I needed to collect
faster so I might want to collect every
minute of every 30 seconds while I could
create a new package that does that now
we used to have a daemon we used to have
an actual separate process called fresh
D now what would happen would be collect
they would write to an RD file and all
fresh to you would do is
read the rd file there were a couple of
problems with that the first problem was
synchronization so let's say a threshold
gets violated but Thresh the checks they
are defiled just before gets updated so
tech CRD file there's no threshold
violation it gets written so now I have
to wait five minutes I have to wait some
amount of time for threshold to to to be
checked and another thing was our d tool
uses time value averaging so if let's
say I had a value at one and for half of
the time and then I suddenly had a value
of ten then the value that actually gets
stored in that little bucket in that
little time slice of the Rd database
would be something more like close to
five it'll be the average the time value
average of that and not the actual value
collected well this was stupid so we got
rid of it we moved everything into uhm
we moved everything into plug D so
collect the checks thresholds when we do
data collection now the argument was
made why don't we just put the threshold
in the collect D configuration and I
argued against that because I may want
to collect dis space utilization on
everything but I may want to set
thresholds differently so certain
systems may be there I'm used to them
growing very very slowly so I might just
want to know how you let me know when it
gets to 98% utilized because I've got a
couple of terabytes on this thing 98
still gives me several hundred gig so I
can check it at my leisure and figure
out where to free up space but let's say
I have a mission-critical system that
tends to happen to balloon up quite a
bit I may want to trigger when it's at
80 percent or 75 percent et cetera and
so what I can do is then create new
packages and so I can put my filters on
I'd say for these machines I want to
check the thresholds every you know at
once set the threshold at 75% let me
build a machine go in 1798 so here's a
package called net SNMP and we created
this package net S&P to limit checking
of values that were only related to
medicine P agents
now that S&P grew out of the
University of california-davis so the
object ID for that is 2021 and then s P
so basically this filter says look for
any service any service on an IP address
of zero zero zero zero when the node
system object ID is like 1 to 306 that
one's at 4.1 at 2021 which is UC Davis
or eight and seventy two which is
medicine P if so I want to check this
threshold in remote or home net SNMP and
that's just a group that we put in the
various thresholds we want to check and
this is very very similar to collective
if you remember collect B we had these
collected schemas so we had two default
schema and in the default schema we put
our data collection groups which was our
like class group file that we did same
kind of thing here we had these we
trigger so the Thresh D configuration
points to threshold in groups which are
in the thresholds dot XML file now
here's part of the menaced and P group
but I'm not going to look at this XML I
mean this XML is fine it's valid you can
look at it if you want it's an opt open
in a mess at City thresholds on XML but
what we're going to do is actually look
at the GUI because we have a GUI and you
don't have to do any of this on the
command line so I'm going to jump back I
just want to look at that we hit it
right at the time so you notice that now
we have we set the value to be 2000 so
2000 actually sent a rearm so if I
actually click on this particular vent
you'll see it's a rearmed event and so
we got our rearmed event and now you can
see the rearm event cleared the
threshold event so kind of cool so
getting back to our slides
so GUI so again here's our class group
file this is the data collection file so
it's in the data collection directory
where we collected on a data source
called class random alias is just
another name for data source alias is
how we store it so we're storing this as
a file called class random dot our ID so
getting back to the GUI the way I do
this is I went to figure of an MS I
found my thresholds here on my threshold
in groups
there is no way in the GUI to create a
new group it's the only thing you can't
really do through the GUI for
thresholding instead what you would do
is edit the thresholds on XML file and
create a null group you just create the
group XML tag hit this request a reload
and it would show let's just go ahead
and do that so I'm just just for an
example I'll type in minutes at C I'm
going to edit thresholds and I'm going
to go with over the bottom so let's go
and grab this group 1 2 3 so let me grab
that put that here I'm going to get rid
of all the stuff in the middle I'm going
to call this class group I'll save it
I'll go back to the GUI
I'll get wreaths row reload the
threshold packages and then we have our
class group so I just created a new
group now if we edit it it would put
everything in the thresholds file but
I'm going to instead I went and edited
this med SNP and so here's my GUI
and I went into the class where I have
an example I added a basic threshold
I put class writing them again I chose
this data source to match the alias
right here so this class random alias
right there and I said it equal to
fifteen thousand ten thousand and three
our while we're here I want to talk
about a few other examples let me go
back to the group so here we have two
high thresholds we have one on the data
source NS this percent and one on NS
this percent node notice this data
source type this basically just tells
open in the best way to look for the
particular values so if I go here let me
clear this if I go to optimus share or
the S&P one that's where our node one
day to store it and i do a list and you
can see here is all the node level data
here's some interface data and here's
that disk index that is the generic
resource that stores those values and so
that tells me where to look for a file
called NS this percent
we also have one for inodes as well and
it's this percent nodes and so this
tells you how the items are available
and then we have a relative change the
relevance changes 33% so we put a 1 in
front of it so it basically says if this
the value we just collected is more than
one point three three three times
greater than the previous value trigger
this alert this lets me know that my dis
percentage will not by 1/3 again I've
never seen that actually so it's a
pretty conservative at least in our
network it's pretty conservative
threshold now I wanted to show you
something about on this inodes one I'm
going to edit it when we gather things
out of a particular when we when we we
gather this information we're checking
thresholds you may want to ignore
certain file system let's say I have a
read-only file system well it's always
going to be at 100% utilization so I
don't necessarily want to trigger on it
so you can use resource filters and
these can be quite complex you can have
multiple filters and you can end them
together or you could have a group of
filters where if any of the conditions
are met you want to filter based on that
so if you look here we have a filter
operator or because when we have a one
filter and we say look at the NS disk
path name this is a value we collect and
basically this is a regular expression
which says ignore any disk do not do
thresholds on any paths which are /proc
slash sis or slash dev slash PBS and so
so Riddle ignored those particular
systems so this does give you quite a
bit of leeway in in monitoring which
thresholds you want to trigger so
there's things you need to ignore you
can use a filter and you can ignore
those various things now I want to talk
also about expression based thresholds
so if I go over here for this if I do an
SMP walk so this this value here 1 3
this is 20 21 this is a UC Davis
obviously this has to be load average so
this is the one-minute load average the
second instance is the 5-minute load
average in the third minute is
third instance is the 15-minute load
average and notice this is four four and
eleven those are actually in hundreds so
if I do an up time you'll see that the
load average is 0.03 so in this case
would be zero point zero four zero point
zero four and a point 11 now you can
just know that and you could say hey if
I wanted to trigger I could set up a
basic threshold on that value and I
could say hey if I want to trigger at a
value a load average of 10 I put in a
thousand so it's raw
basic expressions or raw values but you
know operators might need will confuse
you might be a little confused over that
so you can create an expression based
special in this case we took the load
average the five-minute load average and
divided by 100 that gives us the raw
values so now instead of being 11 it
would be point 1 1 and so we could
trigger now on 10 and rearm at 7 point 5
so now this load average will trigger a
flood average everything's above ten and
or we know our money drops below seven
point five now the main reason for using
expression based thresholds is in this
idea of what if I don't have I need a
percentage but I don't have a percentage
what I have is a total maleta usage in
this case swap so we collect with men
S&P a data source called memory
available swap and a memory total swap
so if you divide total by avail the
total into avail we get a percentage and
then we multiply 100 action make it a
percentage instead of being decimal and
so we could say hey let me know in this
case a low threshold that says let me
know if the value drops below 10% so if
I have less than 10% swap left triggers
the alert and it doesn't we arm until
I've got 15% so notice in a case of a
high threshold you rearmost will be less
than the threshold value and that will
be reversed for a low threshold value so
everyone got that it's pretty
straightforward so what we did was
because we use Ned SNP extends for our
example here we said it made sense to
edit the net SMP group and then we the
nano-sim fee group has a list of basic
thresholds high and relative frames
thresholds based upon this percent and I
doz percent we have some expression
based thresholds load average and swap
and we added our class random threshold
so we had to know the data source
everything else you can kind of pick
from a drop-down we're working on that
there is an open there's an open ticket
on an open feature enhancement to allow
you to pick that from a list we said it
to be 15,000 and when we did the rearm
at 10 and what we got we had the
threshold alert again we kind of cheated
we wouldn't edited the script to enable
us to do this and and we went ahead and
showed how the Lee arm works now I got
one last thing I want to talk about in
this lesson and that is I want to edit
this threshold and I want to do some
custom UV is custome eyes can be just
make it a little easier to a little
easier to track a particular event now
hide ourselves no thresholds were fine
in fact if you remember from our
notifications module you can actually
filter notifications based upon
parameters one of those parameters would
be the data source name so if I had a
high threshold and I wanted to make it
specific to the class random data source
I could I could created a notification
where I set the parameter equal to class
random what's easier for me conceptually
is to just create specific events and in
this Christ I just created a Yui I for a
high random threshold and a UI for the
REO and the random threshold now notice
the Yui is if you remember they are just
strings I try to organize them so since
this was an exercise for class I did UVI
to open a forward slash org slash class
then I created a attack here I could
have just put you add up food when you
add up bar and as long as I knew what
those were I could create those go ahead
and save that and so and now you can see
we've got these trigger bbi's here I'm
not a class random example now now if
you click on those particular if you
click on that particular event you can
then do your own notifications so I
could create a notification based upon
the high random event another
notification based upon the we own
random event now where is that stored
well it turns out when I save that in
opt opt opening a mess at the events
which is where all our events are stored
let me make sure you guys can see that
we have this thing called programmatic
events and so the programmatic events
this file get created if it doesn't
already exist and open in ms will create
default events for you
so I you can see here's my random event
and here is my rearm event
notice it builds in the alarm data
already now it's just guessing you know
this is pretty much guessing but this is
the file you would edit if I wanted to
make this a particular if I wanted to
make this a major event etc I could and
this is where I would edit it and then
if you remember from the events section
you just reload the events but in this
case it does give you a much it does
this all automatically you want to type
anything in this is the one thing I
really love about thresholds is the fact
that um it's pretty much instantaneous
so whenever you configure threshold
you're good to go
now I want to stress one thing because
of that opening a mess does not store
state when it comes to thresholds so
when if you trigger a threshold and you
do nothing if you know that pressure
will never trigger again until it's
rearmed but if you restart open an MS
it'll trigger the threshold again you
will get a second threshold event and so
what you can do is what I do
if I'm using thresholds at a customer
site or in any kind of production
environment I will go to configure open
an MS and I will turn off notifications
so I'll click notifications I'll turn it
off the icon will get the little Ghost
Busters no notice sign then I restart
wait a couple minutes wait for
everything to settle out and then I turn
back on this means that my users won't
get multiple notification events
cause a little bit of that a bit of
confusion there so that's one of the
nice things that's one of the things I
do to work around the fact that we don't
we don't have a state now we do have a
feature request to add that it's just
not here yet okay so there's our
threshold words we talked about custom
the eyes and then programmatic events
okay that's it that is pretty much it we
started out with we started out with our
overview of opening a mess we started
messing around with the GUI actually
since nineteen came out while we are I
was in the middle of this set of lessons
the cuy changed a little bit we added
some extra things but we talked about
the GUI and then we talked about events
and when we turn it into new
notifications and we talked about alarms
and then we did trap simply talk about
SNMP and then we talked about polling
and service monitoring and then to data
collection and we did thresholds notice
that you can do these thresholds on SMP
data sources you can also do them on any
other data source that you have
including latency so if you wanted to
know if I particularly if I go to a
particular node and I go to my resource
graphs I haven't been running this
virtual machine that often so you expect
there won't be much data but as you can
see here this is my ICMP response time
of course it's very very fast because
hey this is the open Animus machine but
I could set my threshold on here too and
it would actually trigger whenever one
of these was actually whenever the
latency got slow so it allows you to set
thresholds if um let's say your
thresholds five seconds you want to make
sure your website respond 75 seconds
well you might we're going to press an
alert if it happens to respond it for
and so you know hey it's responding it
for maybe set a trigger of two or three
that said hey it's been at four for
awhile I need to go in and check the
load and figure out why this system slow
before it hits
um so anyway um so I hope you folks have
found this to be useful patch stuff on
the back if you've gone through all
these you've basically covered the first
four days of our open in the next class
we would welcome welcome you to the
class if you'd like to go through this
with an instructor-led environment i
love teaching this class it's a lot of
fun the this isn't the end I mean we
have just barely scratched the surface
but what I hope to do is come up with an
open MS 102 and a 201 and the 102 will
basically be standalone lessons on
various aspects of opening a mess so you
won't need to have you need this
background you need the 101 background
but once you've passed that you'll be
able to pick any of the one or two
episodes that interest you and you'll be
able to do those at once I also have a
201 series that I want to do I might ask
some of the other people to help me on
that those will be more the high level
more of the high level features so we
might covered rules or things like that
again opening this is a vast platform
and there's tons and tons of things you
can do with open an MS and so it's what
it takes a while to get all of this
document but we do appreciate your
support Oh some people have written me
directly with questions I apologize on
for taking a while to get back we do
have mailing lists and discussion groups
you can go one of the best ways to go is
to go to chat dot open NMS com
this is a chat it's an instance a matter
most it's a lot like slack and we have
some images the larger here so we have
some if you're into development if you
have compass questions if you have
general questions the discussion groups
really good if you have questions about
minion which is our new distributed
version you can we also have Town Square
and Pokemon go because some of us play
Pokemon go and Town Square is more
funny stuff some people will go and ask
questions in the in the town square and
you'll be politely directed to to go to
the opening that's discussion and so as
you can see here it looks like we have
over two hundred three members so if you
ask a good question chances are you will
get you will get an answer pretty
quickly anyway I really enjoyed making
means the next ones will be different
we're moving offices we're getting new a
new office so tomorrow we're moving so I
really had motivation to get this one
done again thanks feel free to leave us
comments hope you enjoy these and really
really enjoy opening a mess as much as
we've enjoyed making it