OpenNMS 101 - Module 4-2: Notification Commands Video Lecture | A Complete Guide: OpenNMS - IT & Software

hello welcome to another episode of open
MS 101 and this lesson were to continue
on the work we did in module four - one
where we did an introduction to
notifications we're going to build on
that what I hope is a much shorter
module where we're going to talk
specifically about notification commands
if you remember from the last lesson we
created a notification path a
destination path where we send an email
to an on call role we then sent it a
minute later we escalated to a group and
then we escalated to an individual user
and in all three cases we send an email
that was the command we used well
maybe we want to do something else I can
remember I was in Singapore many many
years ago and the customer actually
wanted the sound of a klaxon you know
the dudududududududududu
basically their third shift operators
weren't exactly the most dedicated
employees and sometimes they would kind
of doze off in the middle of the night
when something critical would happen so
what we did is we known as a speaker way
up near the ceiling in the NOC and this
feeling was maybe 14 15 feet tall and
ran a cable from there into the back of
the opening this server it did actually
have a an audio card in it and we were
able to send basically play a WAV file
that had that klaxon sound we're going
to show you how to do something like
that from within open and a mess so
here's our exercise the command we're
going to create is based on syslog we're
going to actually send a syslog message
I know that doesn't sound super sexy but
it's actually quite useful we had one
customer who had a kind of a manager
managers piece that wanted to get
information from open in a mess they had
an ability to do that through syslog so
if we could get opening a mess event in
the syslog it would allow them to send
that upstream so that's what we did so
we're going to add the safar command to
the system then we're going to edit our
path for class destination path and add
that in addition to the Java email we'll
add that as a command now the
notification system has an ability to
Matt
up events with down events in the sense
of notices so basically as an event
comes in it can cleared notices based
upon a down event and so we figured
we've use our two events we have our
happiness event and our unhappiness
event which is what we used last time in
this case what we're going to do is
we're going to actually send in the
unhappiness event cause the notice to
trigger and then we'll use the happiness
event to auto acknowledge it before we
escalate and so to show you that that
works so moving on I want to jump
actually to my to my shell here and I'm
going to show you the notification
commands file so here's a notification
commands file and I want you guys to
look right down here I'll kind of make
this a little clearer here is the Java
email command that we were using for our
destination path if you wanted to change
the name of this to email here's where
you would do it just change this name
from Java email to email now the first
attribute here binary equals false this
means it is a Java class we're going to
actually try to execute the Java class
if you are a Java programmer you can
build your own classes and this is
highly recommended simply because of the
scalability every time we run a binary
command we actually have to fork a fork
exec of shell run the command see what
comes back close the shell etc and
there's some overhead matt in this case
if it's a Java command it's a lot more
efficient
now the contact type here you see this
here is email this actually comes from
the user so um when we send a if you
remember when we created our user we had
fields like email page or email numeric
numeric pin various things like that
XMPP account this is where we set this
up so we're going to send an email and
we know this because this is this
just something hard-coded in the Java
mail notification strategy which is what
we're using for here and then we're
going to send - subject - email address
and the text message which is the long
message in the in the notice that we set
up and so this is a binary command it's
pretty simple now let's look at well I'd
see that's the Java command so let's
look at it by native command this is the
command we're going to add so you can
see we're calling a syslog this is how
it will show up in the GUI and then
we're going to execute the command user
bin the logger there's a little comment
here now we're not going to talk about
this in class I kind of stress the the
person who wrote this code English was
not their first language and honestly
think they confused
the term switch and substitution so
anyway you see a substitution here it
means that open an MS is going to just
send that to the command verbatim so
this command right here is going to
basically be I'll just add it up top so
the user bin logger space - Peaks that's
our first substitution the next
substitution is local zero warning so
we'll put that in there and the next
thing is - T which I believe it's like a
facility code and we're going to use
open in the mess and then you have this
switch this is actually going to
substitute the numeric message which is
the short message from our notice so
this will be whatever the whatever the
numeric message is that's going to go in
here so I honestly think that you know
we refer to this like - P and - T we
refer to them as switches and I think
these should be switches and the term
switch should be a substitution there is
an open bug on this
it's a enhancement request I'm hoping to
switch these it hasn't been addressed
yet I'm not sure it will it's not a
super high priority so
oh you're lucky for listening to this
lesson because you know that hey
substitution basically mean switch and
switch means we're substituting
something okay now the last thing to
take out of this is this streams equals
true this basically if it's streamed it
will allow for carriage returns and send
a control D at the end of the message
some things like the mail command on the
command line need that so let me just
copy this let's just kind of demonstrate
what's going to happen here so I'm going
to quit out of here without making any
changes and then I sent this message
whatever the remaining messages and if I
go to var log and I tail messages there
we go there is our syslog so came from
open in a mess there's whatever the
generic message is it was actually log
to a local zero warning so you can
change that if you want so that's what
we're going to try to do now the changes
I made to notification commands you can
add those yourself but I also added I go
to if I go to the training wwm s org
slash capital P training and click on
config under exercise 4 we now have
these two this command here so if you
want actually that's the one one to the
notification commands so at the very
very bottom you'll see here is we here
is the syslog command that we added so
you can just copy this down if you would
like or cut and paste remember if you're
cutting and pasting be sure to look at
source because that way you'll get the
more unadulterated file versus the the
one with the dashes that won't parse so
anyway the keys parts here remember
binary equals true which means we're
going to try to execute a command this
is a fully qualified path to the command
and then the various and sundry things
we're going to send to it
the next thing getting back to the
slides so there's a Java notification
command this is for the Java mail
notification strategy here's the binary
one that we just did now one of the
wonderful things about open MS that I
like is the ability to clear notices I
mentioned and I'll bring this up when we
we actually edit the path but I
mentioned that all notification pass all
the destination paths have an initial
delay and I use this a lot I usually set
it to be about two minutes and the idea
is if we get a down event such as a
no-loss service event i have up until
two minutes for it to clear if i get a
node regain service event for that exact
service it'll clear the notice I'm
getting I won't get a woke in the middle
of the night for something that was a
transient issue if it's been down for
more than two minutes yeah wake me up
I'm cool with that so this is how that
works in the note if D configuration
file right here I can it controls a
number of things and we'll take a glance
that in a second actually I'll pick rent
set it now because I think it shows up
better than on the on the slides so I'm
clear here again this files B note if B
configuration file so a few things to
note about this at the very top you'll
see this status equals on this is a
piece that controls that icon in the
upper corner remember the big red button
I was telling you about
they can turn notices on or off globally
again no magic this is how open MS knows
that those notices are on or off so
statuses are means notices are off
forget about all this other stuff I keep
removing it and it keeps getting put
back I don't know why you'd ever want to
change this but this does allow you to
this is kind of controls how notices are
selected from the database the lastly
the next thing out of here I want to
show you this match all equals true when
open MS started it was the first match
if you had multiple notices on something
the first match would win and some
people like that I think that it's
easier to match everything so I can say
just one event happened send it to three
different groups I might send it to the
group for that area if it was in
Singapore I could send it to the
Singapore group if it was a database
issue I could send it to the database
team and that could also send it to the
overall IT team again this is the
default that it'll match any notice that
the event matches and passes the filter
so you can change that if you want but I
suggest you leave it to true and finally
here's how the auto acknowledge works so
here's how notice this is the auto
acknowledge tag there's a slash Auto
knowledge tag at the bottom so this
placement says if the UE I hear no
Berghain service comes in automatically
acknowledge any notices where the u VI
is node lost service and the node ID for
the node regain service event matches
the mode loss service the interface ID
for the node regain service event
matches in your loss service and the
service ID matches for those lost
service of knavery game service does
that make sense you don't necessarily
want one event to clear a service that
might you know if I lose HTTP and DNS
and DNS comes back up we don't want it
to clear the notices based upon HTTP now
notice we do another one of these four
interface up or clearing the face down
notices but there's no service just note
ID in it 50 I noticed note up clearance
notices from there down and it's just
based on their lady turns out you have
to have at least one match you can't
leave that blank and so while we don't
have anything really associated with our
happiness and unhappiness event I added
the note ID so here we have class
happiness and here we have class
unhappiness so the class happiness event
will acknowledge the class unhappiness
event as long as the node ID in this
event matches that event thanks thanks
now you can also change the resolution
prefix now one of the things we've
struggled with I would really like to
have a system that was more pairwise to
basically said okay
when I send out a notice I want to I
want to have an event that is the up
event and I want have an event that is
the down event and these criteria should
be shared open ms doesn't have that
notice D is just based on events and you
know even if you put the word up for the
word down in the event opening that
doesn't care it's a string and so it
doesn't know that one event is not and
one event is a down so we struggled with
this in the original time when we
originally wrote this
we actually had notices for note up and
no down and so what what happened is
they know down message would get
generates a note goes down we send out a
note down notice and it starts an
escalation show it
emails the group pages the group emails
the manager the manager says hey we need
to fix this they fix it to acknowledge
the notice everything's fine we get a
note up so the note up would get sent
out walk the same path and so it would
send an email out to the guys who got it
they go oh hey yeah I know that stuff we
worked on it they'd forget about ten
minutes later they get a page dammit why
am I getting a page on this I know it's
up fine then the manager would pay he
goes yes I know it's up and then if it
escalated beyond that it would be waking
up the VP and the president saying I'm
up which was like not good you know
especially they didn't get it down so
this is kind of funny because I asked
the community I was like hey you know if
we did this we'd need some kind of way
to match ups with down's well it turns
out person in the community pointed out
there is a table called users notified
in the database that says for each
notice who was notified and how so we
did have that information and so we
wrote some code that basically said okay
if we have this automatic acknowledgment
if if we have a down that enough that
clears it down then we can send a
message to just those people who got the
down but we didn't have an easy way to
configure it or anything so we used a
telecom method so in Telecom they have
this this tendency they'll send an event
out say it's you know this device is in
yellow alarm well when it's cleared they
will send the same exact event but there
will be a cleared bit that's set the
base list here's the yellow alarm but
it's now cleared so what we did is we
basically said okay we're just going to
send the same
notice we sent out so if it said no dat
wwo pms org is down we will send the
same thing out except it will say
cleared no W other business totally get
down or resolved or whatever your
favorite word or in your particular
language should be you can change that
in this file does that make sense so
what we want you to take away from this
configuration is that if we have a
notice based upon class unhappiness and
a class happiness event shows up it will
automatically acknowledge all notices
based upon a penis as long as the node
IDs in both events match and it will
send out the same notice that got sent
out for the unhappiness event except the
subject line will have the word cleared
in front of it thanks thanks and again I
put that up in the configuration as well
so you can just copy it down instead of
having to type all that in or cutting
and pasting now getting back to our
exercise at this point in time so we're
add the syslog commands in those case
commands that's what we did we edited
notification commands that XML and added
that syslog commands the string at this
point in time that you should restart
open in a mess most things dealing with
notifications including notifications
bands are dynamic you edit those files
it takes effect immediately
but in order to make this new command
show up in the GUI we have to restart
open a mess I'm sorry about that
but that's just the way it is if your
hand editing all this stuff and you're
not using the GUI then you don't need to
restart but the purposes of this class
we're going to restart now we're going
to do is edit the destination path we
did for class which is called path for
class and when add the syslog to the
first target so let me go into my GUI so
here I'm in my web console so I'll go
configure opening MS configuring
expectations configured estimation paths
and we're going to edit the path for
class and we're going to edit our first
target on call I'm going to still send
for the on call person I'm going to hit
next
and where it says Java email if i zoom
down here you'll see now there's
since log and I'm going to control click
on syslog so that will highlight syslog
and Java mail where is it
Java emails highlighted and syslog I'll
hit next notice this is our initial
delay I'm not going to set one but
that's what I was talking about if you
want you can add an initial delay I like
two minutes and if that auto
acknowledgment kicks in you will not get
notified it'll automatically acknowledge
it so I think that's kind of neat so I'm
going to finish before navigate away
from there and that should do it now I'm
going to go back to my main page now I
cleared out all my notices so I don't
have any outstanding notices and if I
click on my on call schedule you'll see
that I am still on call currently on
call Taurus so I should get the message
so I'll go back here we want to go to my
from the window and I am going to
control our to go back to my history and
I'm going to look for my happiness event
or unhappiness event now this is the
event we sent I have to change it just
slightly I want to add a dash in one
that sets the node to one so when I send
the event it will be associated with our
local host node and I need to have this
match so that I can make the auto
acknowledgement work I'm not using auto
knowledge meant it doesn't matter but I
have to use I could use service I could
use IP address and in this case I'm just
going to use the node so I click that go
back to my console notice there's one
outstanding notice here give me a second
and will come to me if I click on the
outstanding notice you'll see it is a
sign to look host so that's because we
use the node one so I click on the ID
you'll see that it was sent to Java
email and it was sent to syslog at 5:11
so I'm going to go now before I wait a
minute I'm just going to go and send the
happiness event so I sent the happiness
event notice there's my mail so I did
get my email that the class was unhappy
now if i refresh this this notice you'll
see time replied and that it was auto
that's how Auto knows what works and so
now none of this escalation should
happen so if I hit refresh here again
there's that bug that'll we sort the
message but nothing will happen so I'm
going to wait and just to just to prove
that I'm right I'll wait a minute but
what I want to look at now is not only
so I can look at my mail and you can see
I've got my classes not happy email on
the 23rd at 5:11 and then I got my
cleared notice for classes not happy
that's that cleared thing coming into
play and now if I go to var log and tail
my messages you'll see I got the classes
I'm happy and it was followed by cleared
the classes I'm happy so my sis logs
working now again you can get really
creative with this you can send a klaxon
you can send you can do pretty much
anything that you can run on the open NS
command line now be careful if you're
sending notices to a hundred people you
might need a really beefy box in order
to to generate a hundred of whatever
you're doing at the same time but with
with a little bit of planning chances
are you could do it very very well I
mean one of the other things you could
do is you could you know just send the
script to one user and then have that
using notify those people this is where
our initial trouble ticketing
integration was we actually had a
notification class that would actually
post to a website so and it still exists
so if you ever want to believe it still
exists so if you wanted to you could
basically generate either an email or a
CP post that would go to a website
perhaps of your trouble ticketing system
fill out some forms and send that as
part of the notification so going back
to the slides so we add our system and
we've started opening a mess to enable
syslog to show up in the GUI we edited
our destination path for the class we
added syslog in addition to the Java
mail by hitting that control click
versus just a single click we generated
the unhappiness event with node equal
one that - N 1 we generated the
happiness event - clear if I go back
here I'm going to refresh and you can
see we're done no other messages we say
so that got a lot of knowledge so here's
our syslog cleared here's where we said
in the event and there we got dance
collection that's it that's a short one
I hope that's useful we will have some
more lessons posted pretty soon I hope
you're finding this enjoyable and thanks
for your time