OpenNMS 101 - Module 8: Service Assurance Video Lecture | A Complete Guide: OpenNMS - IT & Software

hi again and welcome back to open in the
mess 101 the video series where we
attempt to bring up to speed on the open
and immense network management platform
my name is tars Balog and in this module
module eight we're going to talk about
service assurance or service monitoring
those of you who have managed to suffer
through all of these videos so far might
remember back in module zero I talked
about the history of open an MS and one
of the things that we did there was
create service monitoring basically the
ability to run synthetic transactions as
a user would to monitor the status of a
service being provided over the network
and we have a lot of service monitors
I'm going to jump over here for a second
to the to a web browser I've gone to
Doc's WMS org which is our documentation
site under open and a mess I'm going to
click on 1901 horizon 1901 is the
release that we're using for this
exercise and I'm going to click on HTML
for the admin guide now down here in
section 4 is service assurance and is
very detailed explanation of what I'm
about to go through we have a large
number of monitors built in to open an
MS not all of them are included yet
there's some of the higher end monitors
that aren't included but at the low end
we have this idea of like an ICMP
monitor is the service providing a pink
into providing the ICMP echo service and
then we have the TCP monitor down here
the TCP monitor will connect to a TCP
port and listen to what comes back to
make sure that something is available
there we have another other we can
monitor radius we can monitor things
about VMware etc on the higher end we
have monitors such as the page sequence
monitor which actually allows you to go
and walk a series of web pages such as
it's provided by a lot of web
applications and you can make sure that
the web application itself is working we
also have a mail transport monitor which
isn't documented which will actually
send an email and make sure it arrived
we also have an integration with the
tools selenium which is a web
application testing tool so you can go
into selenium you can create
a test that creates a selenium XML file
that you could get important open in ms
and use that to test whether or not your
web apps running so lots of really cool
stuff so when it comes to actually
teaching service monitoring I'm always
kind of struggle with something to come
up with and so dawned on me that and
we've been talking in the last couple of
modules about SNMP and SMT especially
the net SP agent has a really cool a
directive called extend and extend
allows us to run a command locally on
the machine
so unlike say an SSH monitor where you'd
have to put password list keys and allow
the monitoring service to connect via
SSH which requires a fork and exec and a
lot of overhead we can actually use SMP
to connect to the Metis the P agent and
have it run a command and then get the
result of that command back and we're
going to use that to create a service
called class monitor so we're going to
test it to this man line make sure it
works we're going to add the service to
the node and then want to modify the
polar to actually pull the service now
talking about extend this extend is
really cool so we can run any any script
basically anything that you can run from
the command line you can actually run
from the extend command
now if the extend command is
resource-intensive
let's say you're doing something that's
going to take 30 seconds to a minute or
something to return what you may want to
do is to cron that and then just use the
extend command to return the output of
that crime but in the case of this
exercise we're gonna create a simple
script and we're going to just run it
using an SMT get so what we're going to
do now I know not all of your
programmers so I'm hoping this is not
too scary but here on my my shell I'm
going to go to the temp directory and
I'm going to create a file called class
monitor
top shelf and in it I'm going to say
that my shell is Bosch and I'm going to
run ecco okay I know that's a lot it's a
lot to process but no all we've just
done is created a bash shell script that
will echo the characters okay and I'm
going to save that and I want to make it
in executable so I'll check your mod
change mod + X to make an executable
class monitor does SH and now when I run
it I get my ok back very very simple
script but even knowing about it it's
very very simple but you may have just a
simple check you need to run on the
system that looks at a file or some
other value in the system and it may be
just enough to return that notice that
in the script I actually had to specify
the shell now if I take that out and
[Music]
actually run it it works that's because
I'm in a bash shell when I run the echo
ok when Ned estin P runs it it is
actually not running a shell so um but
this back can't type of talk and spend
time so what will happen is we have to
specify the language in here that's been
batch in order to make this command work
so I'll test it again and as you can see
it works so this is our very very simple
command now how do we put that into nen
S&P well we edit the SPD dot-com file
and we just add the extend command and I
just put at the bottom of the file so
what we are going to do is we'll edit at
CSP SMP decomp got on the bottom which
is where I put all my stuff and I put
extend class monitor temp
monitor Sh now you put the fully
qualified path when you put your extend
command in and you also want to make
sure when I did this originally I used
to stick it like var but when SELinux
kicks in sometimes it's very unhappy and
so I know that temp seems to be okay but
if you stick it somewhere else you may
have to modify your selinux permissions
to allow the command to run so this is
going to create an entry in a table
called extend called class monitor and
it's going to run this command and show
the output of that command now I'm going
to quit out here and now I need to
restart so system control restart SPD
that will restart the command now if I
do an S compete walk as you ve to see
that show you remember why are you so
nosy is the community string we're using
blow close and I'm just having to know
that this table is dot 1.3 that 6.1 4.18
pro 72.1 not 3.2 and so here is the
output of that file ah
see there's the there's the SELinux
issue I was telling you about notice
that it doesn't have permission to run
this now I'm just going to cheat I want
to just set in force equal off
doot-doot-doot set in force 0 so what I
really should do is make it so that
selinux can actually fix this but I
don't know how to do that to look it up
it's let's be on the part of the class
so I'm just going to turn selinux off
with timing but in a production system
you would want to enable whatever script
you're running you would want to allow
Ned S&P to run that script and notice
when I do it I actually don't get that
permission error I actually get the
output so let's kind of break down these
um
in trees now first of all this is the
top of the table at the very top of the
table there is an NS extend number of
entries scalar this is what that dot 0
means it's a scalar that says there's
only one entry in this table you can
have more voyages you get tons of
entries and every time you have an
extend entry it will update this number
so we're going to stay limited to 1 for
right now it tells you what commands
were running temp class monitor than Sh
there's no arguments you can actually
pass arguments with it now this is
something I want to see how many of you
remember this cache time by default a
net asipi will cache any command for 5
seconds this prevents someone from
poundings into doing lots and lots of
gets and killing your system by spawning
lots and lots and lots of these scripts
to run so basically it will run only
once every 5 seconds so if I if I do a
query and immediately do another query
it's only going to use the cached value
now you can change that with a set but
that can be very very annoying to have
to do so just go back one of our guys
has actually submitted a patch to the
Minister P folks to allow you to set
this value on the command line I don't
know if it's been accepted yet it may
take a while for that to bubble through
all the operating systems but in the
future if you had a query that might
take a minute you could change your
cache time to be 60 seconds or 120
seconds etcetera so that it would only
run once within a certain amount of time
setting is kind of cool we wreck
securing it it runs on Reed which means
it runs whenever we do the through the
actual lead and the things we are
interested in is the actual output which
is use final logs here so it shows us
the output of the first line which says
ok then the output full now what does
this mean let me edit my script and I'm
going to put echo foo echo bar and echo
Baz on here and now when I run the walk
you'll see the outputs changed a little
bit so you can see here the first line
is still ok but then it actually shows
you the full output so it's ok
foo bar Baz shows you all the different
lines and it tells you that there are
four total lines there was no error in
running the command and the first line
is okay the second round of food the
second line is bar this is bad this
could be really really useful suppose
you run a command and the command gives
you a bunch of output I don't know if
I've ever config is on here now I don't
think you install I used to run iw
config yeah it's it's in a wireless
tools but I used to run iw config which
would send output about the quality of
the wireless connection and then I use
the script to break all the important
stuff out into a single line and then
when I ran the script I could find out
this might be signal strength this might
be noise etc so you could actually go
and grab all this information parse it
out one line at a time and then use SPU
vo but a mess to go and collect this
information now I'm going to re-edit my
script get rid of those so now we're
back so we've got the one line so what
we need to do now is we're going to need
to actually pick something here to look
at so in this case we could take this
value we can pick this value I prefer to
pick this value this is definitely the
first line of the output with a dot one
and so what we're going to do is we're
going to actually monitor this object ID
now if net SP knows about MIBs a lot of
actually compile them so when you do a
walk you get all of this you get it kind
of interprets it as much as you can you
can add an option called - o4 option
capital o lowercase n for numeric and
now all the values come out in numbers
and this is what we're actually this
really long number here is what we're
going to use to detect that this service
exists and to monitor it now what's kind
of cool about this is if you look right
here this 99 this little section here
this is the instance
the instance of this particular entry in
the table and these numbers actually
mean stuff it's the word class monitor
in ASCII so 99 is an a is a see sorry 99
is lowercase C in ASCII 108 is an L 97
which is 2 before C is a 2 SS SS and we
have an M o n remember M comes before n
comes before
o and then so we've got mod and I
monitor so it spells out class monitor
this is really cool because you in you
might haven't say you want to run I
don't know 10 extend scripts and 3 are
on this server 7 on this server only
ones on this server well you can use
this object Ettie instance to actually
determine if a particular device
supports this object ID which i think is
cool i love medicine page huge nen SP
fanboys so ok so I'm going to copy this
I'm going to need it now getting back to
the slides
so explain the output we should explain
the numeric output again 99 108 97 115
115 that spells out class monitor now
here's our exercise so what I want to do
is I want to create an SMT based monitor
that will go out and check the output of
that command and make sure it says ok
and if it says anything but ok
it'll market services down so the first
thing I have to do is get your eyes and
open an MS the model is always node
which is a device and interface and a
service and so I have to somehow
associate the class monitor service with
those nodes that support it and in order
to do that I'm going to use a detector
so I'm going to modify my default
foreign source which if you remember
from the previous lessons the foreign
source shows us the detectors that are
going to use
I want to add a detector for this object
ID so now remember detectors and um
detectors and monitors are different
things for example if I connect to port
80 and I'm trying to see if this device
supports HTTP I'll connect to port 80
and I'll issue an HTTP GET if what comes
back is a perfectly formatted 500 error
the detector will add that to the node
because hey it's obviously a webserver
because it sent me a perfectly formatted
error message the difference is that in
the monitor a 500 is not acceptable so
when the monitor does a get and get to
500 it will mark these CP service is
being down now I'm running 1901 which is
the latest version of open ms horizon
and we've added a new new screen to the
front page it's a status map if I
actually had added locations to my nodes
in here I only have the one node but if
I added locations to those in here
they'd actually show up on this map and
we've got new totally new map navigation
controls republished project Atlas so
we'll cover that in a different lesson
but it's worth definitely worth checking
out getting back to this lesson I'm
going to go to my admin page I want to
say configure open is a mess I'm going
to manage my provisioning requisitions
and I'm going to go up here to the
default foreign source not you can
create the foreign sort of foreign
sources per requisition but in this case
I'm going to use the default for the
source so I'm going to leave it up to
you whether to use the default or not
and what I'm going to do is I'm going to
add a detector so the name of the
detector is dreadfully important this
will become the service name and it's
case sensitive I would not use spaces
you can use dashes you can use
underscore characters as a friend of
mine used to called underbars here
instead of underbars I'm going to call
this class - monitor and this is what is
going to become the name of the service
is going to get added to the services
table of the database so you want to
make sure you get this
part right because it's very very
important so capital C class - capital M
monitor that's what you want to call
this now for the class I'm going to use
the SNMP detector as again as you saw
there we have a ton of built-in
detectors um and so you can you can
choose you can use one of those and
modify it mad parameters and usually use
that to discover pretty much anything on
the network now if I left this alone it
would be just like discovering the SP
service which means it would connect to
port 161 and it would look for I think
it's system uptime - the system uptime
or system object editing it's one of the
basic system attributes and looks for if
that exists it adds it to the system but
I want this to be different I want to
look for another object so I'm going to
add a parameter and I want to add the
OID parameter and I'm going to put in
that long string then I put in 1 2 3
that 6 - 1 2 4 2 1 8 o 72 is in the S&P
guys don't want that to be that -
happens to be the extend table this
happens to be the object and then this
99 bubble blah blah blah spells out
class dot 1 is the instance so my my
index is not 1 and I'm going to say that
now what should happen is anything that
supports that object ID should get
discovered and for me that's enough you
know that is enough for for me to do it
now let's suppose I had that objectivity
and wanted to check on a particular
value you can add that as well
so you can actually get an object ID and
then check value for it we used to have
a service called router that we never
did anything basically we just looked at
the mid for the IP forwarding object ID
if IP forwarding is set to 1 this means
that the device even if it's a Windows
box even if it's a Linux box is acting
like a router meaning it has multiple
interfaces and it is forwarding traffic
between those interfaces and we need to
show that as a service but we never
monitored it I mean we don't monitor is
the thing routing and so confuse people
so we took it out but you can add that
into the detector by putting the object
ID for IP forwarding and setting the
value to be 1 and then if it's
the other value it will not be added so
I'm going to go ahead and synchronize
the sky yes now what it should be doing
is doing an import and it should show us
the class monitoring service so notice
there's class monitor notice that it was
not assigned to any of the other
interfaces and the reason is this is a
little tricky if you remember from the
SP exercise the default community string
for this is public so the fall community
string for SNMP is public and it by
default limits you to this system tree
well this object that is way outside the
system tree so we set up why are you so
nosy of our community string and we
associated that with 127.0.0.1 well why
are you so nosy does have access to the
extend table so that's why it's
discovered on this note however 172 21
23 and this wrong of ipv6 address they
are only using public I've not gone in
destiny configuration and said oh when
querying these interfaces use the wide
use of nodes of community string so it
doesn't have that object any so it
wasn't discovered on those interfaces
does that make sense this is kind of
important we've got detectors the
textures we want to go out there and
they're going to dead text things and
then they're going to add them to the
system but notice this says not
monitored so we haven't configured it to
be monitored now when we start talking
about data collection not monitors cool
I mean we may want to monitor stuff that
we do and we may want to do data
collection stuff we don't want to
monitor so it's not necessarily bad but
not monitored means there is no active
service monitor for this service and
we're going to change that okay go back
to slides in catch up so just to review
we went to our default foreign source we
added in a service called capital C
class - capital and monitor it was
abused the SNP detector and searched for
the object ID now one two three to fix
that one up
bloods on 114 that one right there then
we save the changes we rescan low close
we can synchronize or we can rescan
those we get events the Lexus a the
class monitor service has been
discovered so now we have a class
monitor service now the next thing we
have to do is get it monitored and we're
going to do this using the polar
configuration now let's talk about the
polar configuration file so get back to
my shell okay I want to go to opt open
msse and I want to look at the polar
configuration this is one I learned tab
completion if you're going to work with
open MMS you're going to be editing
stuff I don't know this is polar -
config.xml it's pio tab e tab so polar
configurations Pio tab which will
complete out to pole we have another
file called pull - outage that will stop
a hit E and then tab again and it will
complete it all out so there's a polar
configuration box now a couple of things
to talk about on this file first of all
number of threads there are 30 threads
running on this system right now this
means that we can do 30 service checks
well we do 30 service checks at a time
now how do you know how many service
checks you need to run well you know you
can kind of base it on the number of
services so let's say I have let's say F
30 I have 30 threads and it takes five
seconds to do a poll which means I can
do 12 a minute so I take 12 so I'm doing
360 per minute and then I've got five
minutes so I can do 1,800 service checks
a minute a poll cycle because by default
it's five minutes now chances are it's
going to take considerably less in than
like two seconds so that would be 30 per
minute 30 per minute 30 threads from
three threads five minutes 4500 service
checks so in a lot of cases you don't
have to up this but you may so you may
need to go just take a look kind of look
at your your your threads and then try
to figure out if you're going to need
more of them 30 is usually good for a
starter but I've upped this as high as
300 it does use memory so you want to
kind of be careful when you're doing
this um this other option path outage
enabled path knowledge enabled right
here to set the fault if you set it to
true as we talked about in notifications
we had a thing called path outage and
it's specifically for the node down
notification so when a node down event
gets generated if path outage is enabled
and you've defined your paths before
sending the note down notice it'll try
to ping the up string parent if it can
is ping the upstream parent it sends the
note down notification if it can't it
doesn't happen and that's one way of
preventing a flood of notices it's off
by default which you can turn it on here
and then finally have a feature I hope
you never ever ever use
it's called service unresponsive abled I
almost hesitate to talk about this but
in the interest of no magic I am going
to tell you what this does don't use it
it back tip evil don't use it back in
the day - today when open MS was
starting our customers would sometimes
complain they would say open an MS is
reporting say like Shane servers down
it's trying it says mail is down and it
isn't and we'd walk through exactly what
the monitor does which is it was
connecting to port 25
it was issuing a hello and it was
waiting to get something back and it was
taking on the order of 10 to 15 seconds
which means you really need to fix your
exchange server your exchange server is
not meeting your service criteria which
was that an exchange messages should
come back within five minutes anyway
they complained enough that we added
this feature called service responses
enabled if you turn it on and I'm
begging you not
if you turn it on for TCP based services
if the TCP connection can be made but
the expected response does not come back
instead of marking a services down it'll
mark it as unresponsive the difference
between service unresponsive messages
and service down messages or no loss
service messages is that the
unresponsive messages did not count
against availability and it's just this
is wrong if you if you set your criteria
to be hey my mail server is supposed to
respond in five seconds and it doesn't
respond in five seconds that's the
problem with your service not a problem
with over a mess so don't turn this on
thank you now kind of in the addition of
this path outage we had this idea of
what's called node outage and what a
node outage does is basically say hey if
I get a node loss service message so I
connect I test a service and the service
is down before I send the node loss
service message I try to ping the
interface assuming the interface is
paintable
I'll try to ping the interface if I can
ping the interface I'll issue the node
loss service message because our student
services down if I can't ping the
interface there's a chance that the
interface is down or so when I can't do
that I'm going to generate an unknown
loss an interface down message and if
I'm going to generate an interface down
message but before I do that I check to
see if I contain any other interfaces on
the system and if I can't ping any
interfaces on the system I'm going to
issue a node down event and it will
suppress the interface down and then
their loss service events this could be
useful if I happen to have a large
device with maybe say ten interfaces and
three services on each of those
interfaces that's 30 services and if we
ended up getting a if that node went
down I did not want to get 30 node loss
service messages I just want to get one
no down message so again that that's
usually the desired behavior for
everybody however we've had customers in
the past who integrated with open and a
mess and all they wanted to get a
separate node loss service mess
for every service and they do the
correlational variant and so you can
turn this off so you can set the status
of the law you can also change the
critical service I mean if you are
monitoring I don't know a bunch of web
servers you may want to do this I've
never known anyone in production
actually change this because the idea is
you mostly am peeing is pretty much a
done basic check so what I would do is
leave that as ICP but again in the
interest of no magic I wanted to show
you what all those things meant now
after that we have our packages now I'm
going to go down skip this for a second
the package we're really concerned with
is example one this is the package that
ships with open in the mess now for each
package packages array packages allow
you to group together like devices and
determine how they're going to be
monitored so let's suppose you were
responsible for monitoring your
infrastructure and you had certain
really important machines so for that
you may create a gold package so you
have the gold package where you monitor
a whole bunch of services every minute
and then let's say you have another you
know standard production machines
they're not critical infrastructure
machines so you might want to monitor
them every five minutes so they'll get
in a silver package and then finally you
might have a bronze package which says
hey I just want to ping this guy every
15 minutes make sure he's up so in that
cases you can do different packages you
can create a gold and a silver to bronze
package and then use these filters and
ranges in order to sort the devices that
belong to this package so the default
filter for example one is IP add or not
equal to 0 at 0 0 at 0 this is pretty
much our null filter it's yeah gamma
filter every package has to have a
filter and so we use this one this says
if you have a valid IP address it passes
the filter it's going to be included in
the package now furthermore you can
refine based upon IP address IP for
address ranges an ipv6 address ranges
these are just kind of here as an
example you can take them out and it
still work
you have to have a filter but you can
add and remove these Rangers now um
filters are cool when we were talking
about SMT and as I did that joke about
the system location being you know North
Carolina USA North America earth Milky
Way galaxy universe I really stress that
people use that because if I wanted to
monitor let's say I had I don't know a
whole bunch of servers in Malta and my
connection to Malta was kind of crappy
so I could expect a lot longer response
times from my multiple servers versus
everybody else well I could create a
filter here that said in a system
location not like Malta and then I could
create another package which said system
location like Malta and then all my
multiple stuff would be in the multi
package online on multiple stuff would
be in the other package so I hope that
opens up some ideas to you about how you
can use different packages now I have
another example in an in a second this
section here this Rd step when we pull
the service we actually collect the
latency metrics how long it took us to
pull on this configures how that gets
stored and I don't want to cover it
until we talk about data collection I
want to skip this from the timing I
apologize but basically this just stores
the data that we get for the latency for
the poles and then here we list our
services there's good old ICMP the
interval again time and open NMS with
the one exception is in milliseconds so
this is 300,000 milliseconds or 300
seconds which is five minutes so every 5
minutes we're going to do an ICMP poll
we're going to wait three seconds for
them for the response back we're going
to try two times
notice that these are these are complete
try so we will send it out we'll wait
three seconds send it again wait three
seconds for a total of three attempts so
retries is two and then there's the
original attempt so that makes it three
and then again these parameters
determine where the latency data will be
stored will be stored in a file called
ICMP will cover this when we talk about
data collection now notice that all of
these services have these key value
pairs rather than hard code things into
our monitor configuration we use these
because
um take a look at this DNS service it
actually does a lookup and so it has
things like lookup and federal response
codes that don't exist in ICMP so this
allows us to actually go in and monitor
various add various parameters to
different monitors something with DNS if
you happen to have a bunch of if you
install up an MS point entertainment DNS
server and it's failing for some reason
what you can do is change this look up
by default we use local host this seems
to work very well against Linux hosted
DNS servers it tends to cause problems
against windows-based DNS servers so you
may want to put you know www.example.com
or something useful here
notice that we have other services we're
going to last a search which actually
has a URL that it looks at SMTP has a
different port FTP can have a user ID
and a password etc and these are all the
different services we monitor out of the
box now notice you're not going to
monitor any of these services unless
they're discovered so you've got to have
that discovery piece the detector the
detector adds the service to the
interface and then the polar will
actually to run it now here's an example
of all they change it I'll actually know
so we had so we have this really cool
monitor called strafe paint and it's
used to monitor all let me back up a
second okay I want to cover to cover
this this one other thing here one of
the coolest aspects of open NMS is the
downtime model which is this thing right
here so by default network management
platforms pull every five seconds it's
just what they do it's a great timing
it's usually you know it's not too
frequent it's not too infrequent five
minutes and say five seconds for five
minutes five minutes is a really really
good time the problem is for these
systems if they detect an outage they
usually don't pull again for another
five minutes
so you're going to end up no matter how
short the outages you're going to
end up with a 5-minute hit to your
availability now when I did when I
started working in network management
the buzzword was five nines
can you define eyes that can you find
that can you define that for 99.999% up
top and the problem of 99.999 percent
uptime is that equates to roughly five
minutes of downtime a year so five
minutes of unscheduled downtime a year
and when I would talk to customers I
would say okay you know what kind of
service level do you want to get into
they are finite
I'm going to see five nines I'm like
okay over what period of time a decade
of score of years of century and
something reasonable and they don't know
no a month well if you do the
calculations for a 30-day month on five
nines allows for roughly 27 seconds of
downtime and oh honey I got a buzz on my
phone
when you log in log in what so let me
check my mail oh yeah the web servers
down ding that whole process took about
25 seconds so the chance of me getting a
notification that something's wrong and
being able to fix it in 27 seconds is
pretty slow so pretty low so what I
would probably do so in these cases I
would try and talk them down I'd say hey
what about 4/9 cuz if you're doing fine
lines you might as well just say 100%
because 27 seconds of unscheduled
downtime is just I mean it might be
doable you might have something come up
and come down
you know that's automated but for
someone to actually look at the problems
not going to happen so I said why don't
we do four nines well four nines on a
30-day month is about four and a half
minutes of downtime and now I can get a
page and SSH into a web server and
restart it and then start looking
through the logs to make sure it was you
know find out why it died and I'm going
to be able to do it twice and still get
everything up but the problem is with
most monitoring solutions that
five-minute polling cycle means that
you're looking at five minutes about it
you're blowing for nine to begin with so
it open and amass we did something kind
of cool it's called the downtown model
and what happens is when an outage
occurs
momentarily increase the polling rate so
the way you read this line every package
has seven downtown model so the way you
read this line is says okay beginning at
x 0 love the outage and going to time
300,000 milliseconds or 300 seconds for
five minutes so from zero to five
minutes set the polling time to thirty
seconds and this is a complete poll with
wheat rhymes so what's going to happen
is that I detect an outage 30 seconds
later and we pull again and 30 seconds
later I'm going to pull again 30 seconds
and we're poor again I want to do that
for five minutes but it's be careful
that when you do your retries in your
timeouts if you're using the default
downtime model you don't want that whole
process to take longer than 30 seconds
so basically we're going to try we're
going to try we're gonna try we're going
to try we're going to do this for five
minutes and after five minutes I'm like
okay I've blown for nine so there's no
reason to beat a dead horse so why don't
I back off so what this slide says is
starting at um starting at five minutes
and going to forty three million two
hundred thousand milliseconds which is
forty three thousand two in a second
which is twelve hours so basically from
five minutes to twelve hours poll every
five minutes so back to normal so it's
been down so I can have a there are 30
second outage a minute outage a minute
and a half outage of four and a half
minute out of five minute outage or ten
minute knowledge or 15 minutes consider
because after that it really doesn't
matter now
again so what we do is you have another
line here this is beginning at twelve
hours and going to ten times twelve
hours or five days we're going to back
off even further and we're going to set
that to 10 minutes and then after five
days we're going to just delete now this
says delete but what we should do is
unmanage them service so what open NMS
should do is earn manage the service
after five days which means you can turn
it back on or basically say so now for
five days no one cares um we're going to
we're just going to unmanage it and I
don't actually like that so I want to
change that behavior but I'm going to
talk about that in a second but the cool
part here is you know I know the thing I
like about open NMS users is they're
self-selecting you found this video
and you've been talking for I don't know
I don't know how many minutes 30 30
minutes I've been prattling on you
listen to me for 30 minutes and so
you're obviously cool you're probably
very good-looking and you're definitely
smart so one of you is obviously sitting
there thinking but hey if I'm pulling
every five minutes I could have a
perfectly timed four minute and 59
second outage and you'd never catch
that's true that's why you probably need
to apply it that's important to you you
have to apply the Nyquist criteria which
is for the shortest outage you can
tolerate you have to pull it twice that
frequency so like if I wanted to make
sure there were no five-minute outages
on a pole every two and a half minutes
now where this actually works in
practice is notifications one thing I
hate because I'm on call I'm gonna call
a one week out of three and one thing I
hate is when I phone goes off and I'm
like oh you know phone goes off the
middle of the night I apologize for my
wife because she's a light sleeper she
wakes up give this phone I'm like crap
one of my customers has a problem and I
have to go to the other room get on the
computer and fix the problem what I
almost hate worse than that is as I'm
wiping the sleep from eyes and getting
ready to go into the other room my phone
rings again and says let me just okay
sorry we're up now everything's okay
what this downtime model does is during
that first part of the outage it tests
it so if it was if it was transient if
it was due to a network issue that's now
fixed or a service was restarting and
it's now running it'll actually find
that out in 30 seconds a minute etc and
then if you add a notification delay you
don't see the notification that's why I
always put my notifications in two
minutes so if it's for non-critical
services if I get a notification I'm
like I can wait two minutes so if it
fixes itself within two minutes which
will be five complete poles the one that
caused the outage and the four and the
two minutes that came after that then I
know that the service is ringing down so
we've actually tested it a number of
times before I get notified which I
think it's cool anyway we'll come back
to this in a second getting back to
packages we have a cool monitor called
strafe paint and we put it in its own
package
what straight thing does is right now on
the ICMP monitor does it deal very well
with packet loss if you think about it
if I actually send out a packet you know
a ping and it fails and then three
seconds later I retry and it fails and
three seconds later I retry and it works
the serve the ICMP service is considered
up now I had 66% packet loss but the
service is up so in order to measure
packet loss we turn on this thing called
strafe paint and actually use this at
home because I'm on a DSL circuit that
occasionally gets packet loss and it's
incredibly difficult to prove that to my
vendor because I'll call that Lee
although I continue the circuits up I'm
like yeah but only a few of those pings
are getting through and so what we do
with the strafe paying service is we
send out 20 pings every five minutes 50
milliseconds between pings and we try to
see how many of those come back and in
my case I said it the the fail European
Council to 20 which means all 20 have to
fail but if I wanted to if I wanted to
measure packet loss if I wanted 50%
packet loss I could put 10 I wanted you
know 25% packet loss I'd put 5 and you
can show how many of my 20 pings have to
fail before I mark the straight thing
services down now the trick with
straight thing is you don't want to use
the downtime model because you don't
want to retry if there's failures that
make sense so in the straight ping
downtime model we set this to be pull
every five minutes for 12 days and then
delete so if it's been down for more
than 12 days we played but we just we
keep the polling rate to be the same as
in here so 300,000 milliseconds make
sense and notice I limit the include
range so this ships out of the box to
just a very very small subset again you
don't necessarily want to be pinging the
crap out of your network unless you want
to so I tend to reserve this for certain
network interfaces just to make sure
that I'm not having issues on the
network
tactical so we ship to these these
deport we have the jmx packages at the
top which we'll talk about when we talk
about JMX which is kind of outside of
this class we have the example one
package which is our main package for
everything and we've got this straight
four package now I strongly recommend
that if you make customizations to this
file you create your own package which
is what I've done for kicking back to
our class monitor monitor so what I did
was I created a package which I called
package for class very very unique and
creative name their package for class if
you're in example.com
I honestly think you should make an
example comm package and put all of your
stuff there the main reason is first of
all all of your stuff is in one group
and the second thing is when you do
upgrades to open an MS having the polar
having all your stuff at the bottom of
the polar package will make merging a
lot easier I would never I wouldn't even
go and monitor Phi any of the services
at the top I would just turn them off
and recreate them in my own package with
my own criteria which brings up another
question what happens if you have a
particular service that's in more than
one package so like the idea of the gold
silver and bronze package is that only
the gold nodes will be in the gold
package only the silver nodes would be
in the silver package and only the
bronze knows you'd be in the bronze
package but once you have to have one
interface that happened to fall in all
three of them and the first one is
monitoring ICMP every minute the silver
one's monitoring every five minutes and
the bronze one's monitoring every 15
minutes well the answer is the last
scheduled one wins so in that case if
the bronze was the last package to find
it would pull a CMP every 15 minutes so
you could override everything up top by
putting the same services down here but
just just to make sure that's done
always set the status here equal to off
so here's our package so I just copied
this we're using the same kind of open
IP address IP at R naught equal to zero
at zero zero zero notice that um we have
this include range I just left it open
these aren't required I just tend to put
them in here for both historical reasons
and to remind me that this package is
open up to everything
again here's our little Rd section which
we're not going to talk about but then
we're going to talk about the class
monitor here so I've got my service name
capital C - and capital M class monitor
this has to exactly match what we
discovered in the detector so that's why
when we created the class monitor thing
in the detector I wanted this to be very
very careful about the name so class -
monitor capital C class capital -
capital M monitor now the interval I set
this to be 30 seconds because I don't
want to sitting around in this class
waiting five minutes for something to
happen so in this case I said in 30
seconds your mileage may vary you have
to determine for you what works best
five minutes I'm a strong proponent of
five-minute polls but in some cases 30
seconds even 15 seconds might be might
be called for since I made this service
up I set use a find equal to true we
don't really use anything with this at
the moment one of the ideas we thought
about is during upgrades to kind of
treat these differently than
user-defined equals false and then
status equals on you can create complex
service definitions and then just set
the status equal off and you can turn
them on when you need them and then
since we're going to use the S&P monitor
we have certain parameters that only
deal with SMT the first one is object ID
remember that long objectivity well
there we go there's that long object
that they remember this 99 blah blah
that's class monitor this is line 1
that's our ok line and we're going to
actually look at operator value equals
okay equal is equals this can be equals
this can be less than greater than less
than and equal to greater than or equal
to or if you put a tilde here in that
little tilde character you can make that
regular expression but in this case
we're going to say it with with equals
and the operand is okay so basically
we're going to say get this object ID
and make sure equals okay if it equals
okay then the service is considered up
if it equals anything other than okay
the service will be marked as down in
thanks and then every package has to
have a downtime well
there's a downtown now one last thing we
have to do I know this is a lot so
remember we create our package we added
our service to the package notice that
in this service definition there is no
indication of what type of monitor class
monitor is this gets defined at the
bottom so it's a very bottom of the file
we list all the different monitors we
ship with that's all the ones that we
ship with and this is the one I added so
I said class monitor uses the SNMP
monitor class the reason we did this
like in the detectors they're included
um you include those with each detector
you include the the class or the type of
detector using monitors on the other
hand we expect our customers to have
lots and lots and lots of packages so we
don't necessarily want them typing this
huge string over and over again so
that's why we put at the bottom so
anywhere in this file where I reference
class - monitor is going to know to use
the SNP monitor monitor code one place
this came in handy was here where we
have SSH where the SSH monitor here in
the bad old days the SSH monitor was
actually a TCP monitor all we did was
connect to port support 22 right connect
to port 22 and we would issue we would
see if the characters ssh came back and
if it did we would say hey this is ssh
the problem is that actually caused logs
because when you connect to an SSH
service it actually expects you to send
version information back when it
responds to you it says it expects you
to respond back and if you don't it's
causing logs and you know our users tend
to have a little bit of OCD I'm no
exception and we had people complaining
open and a message causing all these
logs to appealingly my system and so I
decided I'm going to create an SSH
monitor and it is it kind of like an
expect script so basically I said ok
we're going to issue that we're going to
make the complete supey connection we're
going to see if OpenSSH comes back if
ssh comes back then want to send
diversion them and that works fine log
went away everybody's happy except so I
made some mistake and my monitor did
not close the connection l so five
minutes later it would open another SSH
connection and five minutes later it
would open another SSH connection and I
think the default number of SSH
connections at least on Linux is
something like forty so basically this
was back in the 1.1 dot three days or
something like that
I basically dash'd all my users networks
because they were using the SSH monitor
and after three and a half hours or so
they couldn't necessitating devices
including the opening ms box this is
what they needed to stop opening a mess
to clear up all the all the errors so I
scrambled to the mailing list and I said
okay change this to be TCP monitor
restart open a message Republicans way
and then later on been read fix the SSH
monitor and made it as produces today
and it actually works and doesn't do
logs etc but that made one case you were
able to change just a few characters in
this file restart and you ended up
getting the monitor to change so that's
one of the main reasons that we put all
the motors in the bottom so again what
you will need to do is when you create
this as you'll need to add a package for
your services and then make sure any new
services that you add are defined down
here ok now so you don't have to do that
I actually put them up in our up I need
to change that URL so that's no go to
work um this is going to be um if you go
to the training slide so if you go the
indexof training slash config I'm going
to refresh this and you see there's an
exercise 8 grab that URL so copy that
link location and you can do the double
you get and get it back obviously copied
this from other slide where I had the
files up over there so you double you
get - n which will clobber it otherwise
you'll end up with a polar configuration
that one but copy that file or you can
merge the changes it remember if you
click on this don't use the browser
version because it puts all these little
dashes in instead you can look at the
source and then if you look at the
source you can see down at the bottom
here is where I put my package for class
so you can actually copy that paste that
into your polar configuration
and remember to grab the monitor line
down here at the bottom okay now you can
read at this point time you can restart
open ms but I want to try something so
let me go back to my webpage you can see
a little refresh notice the service is
not monitored we are trying to make it
so that you can restart each individual
demon in open NMS without having to
restart the whole thing that was that
demon config this reload even config
that we did for event D well the demon
for the power is polar D so I can send
that in it how do I know it's polar D if
I do an ops open NMS then open an MS and
then use the - D status command it'll
show me the status was open in a message
I'll show you all the running theme and
then there's the polar D demon so I just
restarted polar D and in the past it
didn't work but with 1901 you'll notice
that now my class monitor is set at 100%
so now the not monitored is gone away
and I'm actually monitoring my service
and I did not have to restart over
notice that I think very cool so so
especially in systems where you you want
to limit your downtown now you can just
reload the polar configuration so now um
what I want to do is test it so let me
go back if you remember we put our file
in temp so I'm going to edit my class
monitor shell script and one change it
to not ok and save it I'll go back here
now if you remember we set our polling
rate to be 30 seconds and so the cool
thing is within 30 seconds assuming all
this went right within 30 seconds this
class monitor should be marked as down
do to do this is the part where we do
you few more seconds
haha this may be where I need to restart
yeah it doesn't look like this is
working so obviously I ran that we load
daemon config file but it didn't didn't
work so I went to the page it'll
actually show you the parameters and
stuff that we configured so we know all
that's right but it's not marking things
down so obviously there's still a
problem with that reload potentate I
apologize
well at least we know now so I'm going
to do is opt open NMS bin open them and
stop stop open MS now when I start open
an MS it'll obviously notice that the
service is down this is just to show you
that that's not a that I don't need to
do that for the particular outage what
I'm going to do is I'm going to edit the
class monitor shell script again I'm
going to put it back but yeah I
apologize for that so that's what I get
for making stuff up in the middle of a
video so let me change this back to OK
and we'll start er up while this is
starting I want to jump back to the
slides and kind of review what we've
done so we created a polar package and
again I strongly recommend you create
your own polar packages for your stuff
and put them at the bottom of the file
above the monitoring list because it'll
make it easier to merge when we do
update again I created a filter that
basically said everything IP address not
equal to 0 0 0 0
service name set the class monitor
intervals 30 seconds here's that long
object ID that I basically cut and
pasted from our walk we said it equals
to ok which means it has to equal okay
exactly if I wanted it to just contain
the letters O and the letters K I could
put a tilde in here and that would make
it a regular expression and when we have
our downtime interval and remember we
have to add our monitor line every new
service we add we have to add at least
one we have to add this one monitor line
to tell open in a mess what monitor to
use in this case we're using this the P
monitor when we talk about the downtime
model again what the downtime model
allows you to do is temporarily increase
polling when there's an outage the main
reason for this is to determine as
quickly as possible when an outage is
resolved
so it doesn't impact your sleep so what
you can do is on notifications you can
add an initial delay and then if there's
a problem you know that it's been polled
it's been down for a certain amount of
time and that several polls have been
attempted and you can actually do that
set that here so that's what we do so
what we do is from 0 to 5 minutes into
the outage we poll every 30 seconds from
5 minutes to 12 hours we pull it reach 5
minutes
from 12 hours to 10 times 12 hours or 5
days we pull over 10 minutes and then we
manage the service if it's greater than
5 days now I don't like that because I
travel a lot and sometimes there'll be a
machine in the office that I'm
responsible for and if I'm gone for a
week and it goes down maybe it won't get
fixed so when I come back after remember
hey that went down I have to go and
forcibly manage the service back through
the GUI to make it the monitoring start
so to fix this what most people would do
is delete this downtime begin line which
is part of the answer let me jump back
to the polar config just to look just to
kind of do this on the screen so a lot
of people say oh well I don't want to
delete the service so I'll delete that
line and that's half the answer the
problem is if you delete that line what
will happen is let's read what we told
the computer we told the computer right
here we said ok from the beginning of
the outage for 5 minutes in the outage
pull every 30 seconds from 5 minutes to
12 hours pull every 5 minutes
from 12 hours to 5 days pull every 10
minutes and then stop since there's no
lines we have this end here polling will
stop and that service will never ever
ever ever come back up so if you want to
get rid of the automatic delete delete
to end doesn't need to be there so if
you delete the end what will happen is
now it'll pull 30 seconds for the first
5 minutes every 5 minutes for the next
12 hours 10 minutes forever
and that's what I do for all my services
because I don't want them to be forcibly
I managed and so I change the downtown
model for all of my systems thanks get
out of here okay let's go back we should
be able to refresh this log back in
since I think we restarted so notice the
class monitor system is still up and I'm
going to make it go down I changed it to
not okay now within 30 seconds we should
see this go down I apologize so that Leo
didn't the polar configuration file is
the one we were having the most trouble
with so you saw it actually reloaded the
monitor but obviously didn't schedule it
and so it actually wasn't being polled
now you see boom within 30 seconds I get
class monitor outage identified with
reason code and this is another thing I
live on open MS it tells you why the
poll failed the observed value of not
okay does not meet the criteria that it
has to equal okay if SMP were down I'd
get another outage I get a could not
connect outage and so it does give you
some idea what the problem is so now let
me go back remember right now my
downtime models kicked in so I'm still
doing 30 second polls and now within 30
seconds we should see it come back up
and just like before all of these no
loss service messages in those regain
service messages are actually creating
alarms so we're getting alarms we're
getting notifications all that good
stuff is still working on our brand new
service here there's on there we go whoo
you don't worry there so class monitor
outage has been in storage so if I go to
my alarms Hall alarms
you'll see there so there's a service
here if I hit refresh the bottom one
should clear because of the and then we
have a 30 second the cosmic clear model
is kicking in man my time is this area
I can't install and then the same thing
is notice we have no notices but if I
actually go and look at notifications
all acknowledge notices you'll see the
class monitor service did create a
notice and it was Auto knowledged when
it came back up and this is the beauty
of it if you put an initial delay in
you're not getting monitored you're not
getting notified so I think that's cool
so this works well with all of the
different monitors we have and we've got
a ton of them if you're say you're
coming to open in a mess from a tool
like Nadia's and you're used to check
scripts well you can use this extend
functionality in that SSP to schedule
your check scripts so you can just run
your check scripts on your box and then
have open an MS create VN SNMP if you
actually use n RPE to run your scripts
well there is an in our PE monitor so
you can do the same thing with open and
a messages and check strips the benefit
will be because everything is done in a
java virtual machine it scales we're not
actually doing the fork exact if you
absolutely positively have to run
something that requires a script we have
a general-purpose monitor which will
execute scripts I don't talk about it
very much because it's not it's not
something I like mainly because it it's
so expensive as far as resources go you
can't monitor ten thousand devices with
that but you can monitor ten thousand
devices with S&P or any of our other
monitors so getting back to our slides
so that's the downtime model here we
restarted the open an MS again the send
event command did not work worked
halfway I will open a bug to make sure
that we're still working on that but it
worked halfway we tested it
notice that we changed okay at change
it's not okay it told me what I did and
then finally I want to talk about a
thing called scheduled outages so when
we were talking about like uptime 99.99%
uptime and all that good stuff
I mentioned or in the scheduled downtime
well what about scheduled downtime well
you can do that in open ms we have this
ability if you go up on your
configure open in a mess on the admin
page and you go up under note I think
it's to be DDD where's under service
monitoring obviously configured
scheduled outages this allows you to
create outages so I can do something
like um you know outage or class so I'll
add a new outage and you can do nodes or
interfaces so I'm going to do local host
because that's our node and I'll add it
so I've added our node and I could say
there's I could be daily weekly monthly
or specific let's say daily and let's
make it from I don't know 1500 say or
1500 249 us to 1600 well 15 999 so we'll
do maybe then we'll add that outage and
then what is the effect you're gonna
have an effect just notification you get
a buhdeuce polling etc so I want to
check our package for class polling late
now I want to stress this is a poll
outage what this means is we do not pull
during this time so let me save this and
I'll talk about it so I'm saving this
class
you see it's red now which says it's
actually in in progress because if you
look at the time it's setup in progress
now if I go and I change one one I
change the class monitor to be not ok
like I did before
notice that I will not get any
indication and the reason being that we
are not polling this particular node
during the outage window since we're not
pulling it we'll never know that it's
went down and therefore it's kind of
like see no evil we just won't notice it
went down so it's critical to understand
how poll outages work how scheduled
outages work let's say you have a
standing maintenance window from I don't
know
from 2:00 a.m. Sunday morning this
radiance anymore so you put in a
standard outage every Sunday because you
can do a daily outage you can do a day
of the week outage you can do a day of
the month outage and you can do a
specific outage so let's say we said
every Sunday from 2 a.m. to 3 a.m.
there's no we have a scheduled outage
because that's the only maintenance well
I don't know about you but if I'm up at
you know 131 145 in the morning on
Sunday I might be eager to get it over
with you know hey I want to go back to
bed so let's say I start my maintenance
window at 159 instead of 2:00 a.m. so
one big night I started doing stuff if
open MMS happens to pole during that
time then what will happen is um will
market is down the moment we enter the
outage window we won't do anything so it
will stay down for the entire hour and
then at the end of the hour it will come
back up and so I'm going to refresh here
as you can see there's no outage it's
obviously been more than 30 seconds and
this is the cause denoted it's currently
in an outage women so it won't know so
if I go back here and turn this thing on
make the service come back up you'll
never see the outage we'll never get a
notification we'll never get an alarm
because the pole outage is going on and
now in about 10 minutes or so this
outage will clear this little warning
will go away from the node and then if I
take their work thanks thanks
so that's kind of it that's kind of it
for for service monitoring I mean
there's tons of stuff we're going to do
after this we're going to start
examining one of our more complex
monitors called the page sequence
monitor we're actually going to talk
about you know how do we create services
for things we can't pull we have a thing
called the passive status keeper for
passive services that we can't pull lots
of cool stuff to talk about a little bit
NMS anyway I hope you folks found this
useful
sorry for the mess up on the reload
demon config please check this out play
along and I hope you find these things
useful feel free to talk to us on the
open em Escom or in the comment section
of YouTube and until next time hope you
having fun with a bit of a mess