Issues with email gateway

[Patrick Murtey]

We are having issues where the email gateway is sending a clear message but we are not receiving the original alarm notification, even if the auto operator in the NAS has both the critical and clear boxes checked. This occurs with multiple different  probes, but became really apparent with a dirscan probe alert recently.  Has anyone else encountered this behavior? 

 

TIA

Happy New Year

[RowanCollis]

Some probes will send clear alarms as part of their startup, could the probes have been restarted?

Do you need to send an email on the clear alarm? If you do then could you use the alarm count >1 in the AO profile ?

HTH

 

[David Michel]

Don't know of a 'oh yeah it's this' quick fix, and recommend sending a test alarm and follow it through the process to find the break point.

[Max Lim]

pmurt, what are the AO's matching on? Can you provide a screenshot or example of the AO?

From my experience, the content of a Critical alarm and a Clear alarm can be very different. I would only check both Critical and Clear if I'm 100% certain the content of both alarms are the same except severity. In my environment, I typically have a separate AO for the Clear alarm. This does mean double the number of AO's but my environment is quite complex and it's the only way to have notifications working the way we want.

 

Also, whenever I'm troubleshooting these kind of problems, I use the Alarm History in the Status tab of the NAS probe. From there I can see the full transaction of the alarm and can view the content of Critical and Clear alarm to make sure the AO's match as required.

[Patrick Murtey]

[RowanCollis]

I would recommend you change it from "on message arrival” to "on message overdue 1m” otherwise you will get an email on every interval of the alarm…. Unless that is what you want ?

I presume you want one email when triggered and one when cleared ?

[Garin Walsh]

Also note that you are filtering on "count < 2" and so, if this is like most UIM probes, you'll get an increment to that count every test interval. It won't be <2 for very long.

 

And Rowan is also right - Overdue age should be the default here. You will only want to select "on arrival" if you specifically know that you need to use that setting.

 

And MaxL is also right - get in the habit of sealing with clear separate from the other alarm levels - it'll save you headaches in the future - especially because some probes don't send informative messages when they clear something, they depend on the supp key matching for that. 

[Max Lim]

pmurt, The suggestion to use "On overdue age" is very valid and required, depending on your requirement. If you need some kind of delay before the notifications are sent, then that is the option to use. Keep in mind though, If you have a lot of AO's you can end up with more than one AO triggering for each alarm. You would need to make sure for each AO, it excludes the filters of all other AO's.

 

On the Advanced tab, there's an option to "Skip further profile checks on match". However, this option can only be selected when "On message arrival" is used. As Rowan mentioned, this typically causes you to get a notification for every interval count of the alarm. So to get around this, make sure "Message Counter" is "Equals to" value of "1".

 

Then have a separate AO for the Clear alarm with the "Message Counter" set to "Great than or Equal to 1".

 

So you end up with two AO's configured like this:

AO1:

Hostname = Servername

Severity = Critical

Message Counter = Equals to 1

NMS Domain = mgmri

Message String = /.*cashless*./

Adv Tab = use "Skip further profile checks on match"

 

AO2:

Hostname = Servername

Severity = Clear

Message Counter = Greater than or Equal to 1

NMS Domain = mgmri

Message String = /.*cashless*./

Adv Tab = use "Skip further profile checks on match"

 

The only new consideration now when using the "Skip further profile checks on match" option, is the order of your AO's. They get processed top down and stop when there's a match. So have your AO's with more specific filter options at the top, and the more general ones down the bottom.

[Patrick Murtey]

I have gone ahead and started ordering as you recommended and using the Skip further checks feature. A question we have about this. We are doing both Spectrum notifications and Email notifications. When ordering in NAS AO and using the 'Skip Further' does the system look at emails as a group and you just order the emails and group the spectrum alerts separately. Or do you order by probe, Emails and spectrum alerts as a whole?

[Max Lim]

I'm not familiar with Spectrum and not quite sure I fully understand your question. But in terms of how you order or group your AO's, there's no right or wrong way. It depends what works best for you. In my environment, I group the AO's by probe and then sort accordingly within that probe category. Actually, I use quite a few fields to filter on. Typically I use Origin, Probe Name, Sybsystem String, Sybsystem ID, User Tag 1, User Tag 2, Message string, and Hostname in each AO I have. The order in which I just mentioned them would also be typically be Generic to Specific if that makes sense. So basically I group by Probe type, then sort it so the more specific is at the top and less specific at the bottom. I also make sure the naming convention is consistent to make managing and administration easier. 

E.g.

AO-Disk1 (Origin, Probe, UserTag1/2, Message String, Hostname)

AO-Disk2 (Origin, Probe, UserTag1/2, Message String)

AO-Disk3 (Origin, Probe, UserTag1/2)

AO-Disk4 (Origin, Probe)

AO-CPU1 (Origin, Probe, UserTag1/2, Message String, Hostname)

AO-CPU2 (Origin, Probe, UserTag1/2, Message String)

AO-CPU3 (Origin, Probe, UserTag1/2)

AO-CPU4 (Origin, Probe)

AO-Service1 (Origin, Probe, UserTag1/2, Message String, Hostname)

AO-Service2 (Origin, Probe, UserTag1/2, Message String)

AO-Service3 (Origin, Probe, UserTag1/2)

AO-Service4 (Origin, Probe)

[Patrick Murtey]

The Spectrum AO's utilize a "repost" option as opposed to the "email"  in the action type.  We use utilize most of the actions, email, repost, command, assign, script, and close.