Stop Spectrum from sending multiple alerts for same device

[Anon Anon]

I need to get Spectrum to quit emailing me multiple times for the same event.  I have my SANM working but when a router or switch goes down, I don't get just one email but sometimes 10 or 20 one right after another.

[Legacy User]

You say sometimes you get 10 or 20. Is there ever the case where you just get 1 email or is it always more than 1?

 

Are the emails exactly the same each time? Same timestamp, for the same device model, same alarm detail, etc?

 

I don't suppose the devices that go down are going down and up in quick succession? You could check the Spectrum event view to see if anything like that is happening.

 

John

[Anon Anon]

multiple emails with exactly same time for the same event whether the alarm occurred or cleared.

[Anon Anon]

Do you have the ClearScript and UpdateScript sending emails as well ?

I only have the SetScript sending emails, and not ClearScript or UpdateScript

[Legacy User]

Is it always the same number of emails?

 

Do you get multiple emails for the clear as well as for new alarms?

 

I presume you have only one AlarmNotifier running? Maybe share screenshots of your SANM Applications and Policies.

 

Can you setup an AlarmNotifier with the default setscript, clearscript, etc and see if that displays alarm details multiple times? That should tell whether the problem is with your scripts or with the Spectrum/SANM end.

[Anon Anon]

I have both ClearScript and UpdateScript email set to False.  Whenever Spectrum polls a device and that device show down I will sometimes get up to 10 or more alarms with exactly the same time for the same alarm. It's getting very annoying as little things like this prevent me from widely deploying Spectrum to other users,

[Dan White]

So you also get a high occurrence number for the alarm showing in the Alarm View - or it only shows 1 there - but you still get multiple emails out of Notifier?

[Anon Anon]

Exactly. One alarm in the alarm viewer but the same alarm multiple times in the Notifier resulting in multiple alerts in my inbox

[Anon Anon]

Sounds like you have multiple SANM policies that catch the same alarm and send the alerts to you.

 

Can you post a screenshot of your policy configurations ?

[Dan White]

...but with Occurrence column showing only 1 ? (just to be sure - sometimes my colleagues hide that column in their Alarm view - sorry to keep labouring the point!).

[Anon Anon]

Here is an example.  If I have my WAN routers in a global collection called WAN Routers and I want to be alerted by SAMN to email me whenever a critical, major, or minor event occurs with any router in that collection. In setting up a new filter in the default policy in SAMN after naming the filter and putting in my email address for the notification data the only filters that I need to use are the landscape, severity, and collection.  Is that correct?

[Dan White]

OK. I see nothing wrong with that SANM set-up as you describe it.

Is the Spectrum in a standalone or distributed set up?

And you see that as far as the alarms are concerned... the "occurrences" column for one of these alarms in OneClick only has a  1 against it?

[dhapr07]

In addition to "danpaulljwhite"  If it is Distributed Setup let us know on which SpectroServer the Alarm Notifier is running.

let us know the Spectrum version.

Can you also check the Notifier.out file and check how many entries of same alarm are present.

[Legacy User]

Of course if it is a distributed environment and you have AlarmNotifier running on multiple servers that would explain why you get multiple emails. It wouldn't explain why you don't always get the same number of emails though ...

[Anon Anon]

I'm not on a distributed environment but it seems that if I only use the following filters: landscape, collection, and criticality I should get ANY alarm that is triggered on ANY device within a given collection. Do I need to also specify the alarm type if I am choosing all criticalities?

[dhapr07]

Yes you need to also add the alarm type in the filter. The options in a filter is evaluated on AND operation bases.

Can you please share screenshots of  SANM- All application , SANM- All Policies and SANM filters associated to active policy.

Also the Spectrum version with the hotfix level.

[Anon Anon]

Can't share the screenshot right now as I am at the airport. But, if I include the alarm type does that still require me to also the matching criticality? If I wanted to be alerted on all critical alerts in a global collection why do I have to include the alarm type?

[Dan White]

(Sorry I missed seeing your last - as not refreshed the conversation)

For all severity = critical alerts from devices in your global collection - no - you don't need to filter on specific alarm type.

The suggestion to filter on Alarm type was just to limit the alarms being notified to some specific types - as it seemed to us that Spectrum was detecting several different alarm conditions for each device.

 

The SetScript doesn't include what the actual problem was (Probable cause text) in the email Subject line by default - so you just see a bunch of emails and can't distinguish very easily in your in-box what is what! Unless you do a custom version of the SetScript with a few small edits - so I usually change that Subject  - shorten it, but put in the first line of the  $PCAUSE.

 

There are reasons why you may get more alarms through SANM/Notifier than you see in OneClick.

OneClick alarm view's filter  hides certain alarms if there is a higher severity alarm posted for the device.

Also - with default filter settings  - OneClick hides symptom alarms  - only showing root cause alarms. Whereas probably these all get passed to SANM so more alarms get notified than you might see in OneClick with default filter settings. That is something you can click on or off in the 'State' tab of the OneClick alarm filter, but that tab is not present in SANM's filters.

[dhapr07]

These are filters and used to filter based on customers requirement.

 

Lets say customer is interested to see only "Device has been stopped responding to polls alarm" which is of type critical. he is not interested to see other alarms.

In that case In the Severity you select the Criticality as "Critical" and Also in the alarm Type you specify "Device has been stopped responding to polls alarm" alarm type.

 

Case 2 : Lets say customer is interested in all critical alarms on the spectroServer. and want to include other alarms.

Customer has to select the Criticality/Severity as Critical and also in the alarm types customer has to select the alarms in the include list. (They can include all alarms, and Spectrum can do and operation between the "Critical" Severity and the available alarms.

 

 

In Simple words.. it is the AND operation that applies for the options inside a filter.

 

================

from the Alarm notification document.

Note: If you create a filter with multiple parameters, you create an AND condition. As a result, all of the parameters must return TRUE for the filter to return any results. To create an OR condition, create two filters, each with a different filter parameter.

===============

 

 

If you DONT add any alarms in the INCLUDE list of alarm type then you wont  receive any email/result. 

 

You can do a test on your machine.

[Dan White]

Prem,

In your case 1 - there would not be a need for the Severity filter would there? - It could be left at default, i.e. all severities allowed through - because there is a  filter for the specific Alarm Type anyway. Less processing!.

[dhapr07]

Yes I agree with you.. IN the first case we can include all the severities and add just one alarm on the alarm type..
but

 

 

It should be safe to say that if I want to be alerted on ANY critical alert in a global collection then the only filters i need are landscape,collection, critical alerts. All other filters I can exclude

this will not work.. You need to have alarms in the include list. Else it will not return any result.

[Dan White]

Prem,

You are, of course, correct.  But I think, by saying "All other filters I can exclude"  - Opnet simply meant  "All other filter tabs I can ignore" -i.e. not worry about configuring them. As, by default every Alarm Type is included and allowed through, there's  no need to configure that tab of the filter in the scenario in question.

The word 'exclude' needs careful use in the context of SANM filters, I guess!

[dhapr07]

If that is the case, yes it will work..

[Dan White]

So that is probably the cause of your multiple notifications - there were several different alarms for the devices concerned - whereas you are only interested in certain specific alarm type(s) that can now be added to the SANM filter - (in which case no need for the severity filter either, really).

 

Let us know if it now works?  (Prem's request for detailed diagnostics probably not needed if so - good luck!).

[Anon Anon]

It should be safe to say that if I want to be alerted on ANY critical alert in a global collection then the only filters i need are landscape,collection, critical alerts. All other filters I can exclude. Do I have to do a ./AlarmNoifier -r .alarmrcevery time  I make a change to a filter?

[Dan White]

No you shouldn't  have to restart AlarmNotifier when you change a SANM filter.

( If you're using the default-named .alarmrc file you don't need to specify it on the command line either.)

[Robert Kettles]

When you say "do ./AlarmNotifier .alarmrc", are you shutting down the running one before you do that? Is it possible you have multiple AlarmNotifiers running?

[Anon Anon]

I believe now I have it. At a minimum I should have the following filters: landscape, criticality, to include critical, major, minor, etc, collection, and alarm type to include the specific alarm type that I want to be alerted on 

[Dan White]

Yes that will work - may be more filtering than you need now!...

If on the left hand side of the Alarm Type tab you ONLY have the specific ones you want - (by default that list has every alarm type in Spectrum) and you've moved the rest over to the exclude side then you don't even have to worry about severity and can leave that at default .

Here is example where I only want the 40 or so custom alarms concerning some ALTEON kit - the rest of Spectrum's alarms are excluded.  I don't need a severity filter - it's left at defaults to include all severities -  as the Alarm type filter on its own is sufficiently specific.

HTH

cheers, Dan.

[Anon Anon]

This is insane.  Why is this so flippin ridiculous?  I just got 32 alerts for EXACTLY the same alarm.  Here is as best as I can describe my setup.  Within the SANM, I have a filter to alert me whenever a Cisco 871 or a Cisco ASR 9000LTE router has an alert. So I have a alert based on the following filters:

1. Landscape: I only have one

2. Device Type: Here I have selected the type of Cisco router I want to include_ Cisco 871, CiscoASR900LC

3. Collection: Default Domain; Every device that is discovered and modeled by Spectrum gets placed in this collection by default.

4. Alarm Type:  Here I included the critical alarms that I want to be alerted on.

5. Model Type: I included Rtr_Cisco.  Not sure why I would even need to include this.  Isn't the Device Type enough for Spectrum???

 

Anyway, with just these filters I can happily say I can get up 32 emails per device to my inbox for the same **** alert.

[Legacy User]

As you say that list of filters is probably overkill, however I don’t expect your problem is with your SANM filters. I expect it is more likely with the AlarmNotifier end.

 

You’ve been asked a couple of times for a screenshot of the list of SANM Applications. Why can’t you provide those?

Can you at least confirm that there is only 1 application in the list when you run a Locater search for SANM Applications?

 

You’ve also been asked for screenshots of your policies and filters and that would also be useful.

 

I asked you much earlier to set up an AlarmNotifier to use the default AlarmNotifier scripts, have you done that yet?

 

The issue here might be a bug in Spectrum but might also be in your configuration, however I’m not sure how we can help you if you don’t do the things we ask.

[Anon Anon]

Thanks John. I'll post those screenshots to this post when I get back in the office. I'll also include the AlarmNotifier. Somehow I don't think it's in any of the filters I am using as I have tried to stay as minimal as I could. You mentioned something about the AlarmNotifer is there something in there I should take a second look at?

[Legacy User]

If you run an AlarmNotifier with the default scripts then that should just display a screenful of data for each alarm.

If it is doing that and your AlarmNotifier that is customised to send email is sending multiple emails then you know the problem is with your customised scripts.

If however the default AlarmNotifier displays the data multiple times, similar to your emails, then the problem is somewhere in Spectrum/SANM.

 

If you establish the issue is with the scripts we can take a closer look at them then. Feel free to share the SetScript in the meantime if you wish.

 

Note you can run the default AlarmNotifier in parallel with your custom version, to do that you'll need a separate .alarmrc that references the default scripts rather than your custom ones. Also make sure the same policy is applied to both.

[dhapr07]

In addition to John's update, Can you also provide us with he Spectrum Exact version. So i can check if there are any known issues in the past.

Also please share screenshots of two emails that you received for one alarm.

[Anon Anon]

Ok here are some of the conditions of my Spectrum  and my quest to stop getting all these excessive alarms.

 

Spectrum version: 9.4.1

Distributed Environment: No

Custom Alert Policies: NO, I am using the default policy within the SAMN

Custom AlarmNotifier: No

Custom SetScript: YES,  Includes physical location of device in alert showing street address. I can provide this upon request.

Goal from Spectrum SAMN: To be alerted when a critical condition is alerted on a Cisco router

 

My question is this: I want to be alerted whenever the following conditions are met on either a Cisco 871 router or a Cisco ASR9000LC router

• Device has stopped responding to polls
• Chassis Down
• HSRP Change State Notification

I am using what I believe are the least amount of filters needed

 

---------------------------------------------------------------------------------------------------------------------------------------------------------

 

My ultimate goal to to have Spectrum alert me  whenever a critical or major condition occurs on a Cisco router and alert me when that condition clears and the router is back to normal but I only want the alert once not 32 times like I am getting now.  Please share any SAMN tips on how you use it to be alerted on your Cisco routers as I have routers ranging from Cisco 871 routers all the way to Nexus 7000's

[Anon Anon]

Did you check the Notifier.out file to see  how many instances of the alarm are logged? Did you see AN processing it multiple times?

 

I suggest you to raise a support call if nothing works as i see lot's of suggestions which didn't help you.

 

Kalyan

[dhapr07]

Things looks good to me..

I remember you have stated that you start the alarmnotifier using the following command.

./AlarmNotifier.exe -r .alarmrc

 

Can you check if you have multiple command prompts open or check from the Task manager if you have multiple instance of alarmnotifier.

 

 

I would suggest you to run the AlarmNotifier with the processd so we can have the alarmnotifier log file "Notifier.out"

First open task manager and kill all the alarmnotifier processes.

 

Steps to start the alarmnotifier using processd.

- go to $SPECROOT/lib/SDPM/Partlist.

- Open AlarmNotifier.IDB file in word pad or notepad

- change the below options to true.

AUTORESTART;y;

AUTOBOOTSTART;y;

- Save the file

 

-Now restart the processd.

- Open command prompt or bash window

- Naviigate to $SPECROOT/lib/SDPM

- Run the below command to restart the processd.

./processd.exe --restart

 

Now check the task manager if the AlarmNotifier application is started. Also check if there is file with name Notifier.OUT created in $SPECROOT/Notifier directory.

 

Now try recreating the problem(let any critical alarm get generated on the devices). Check how many emails you receive.

If you receive multiple emails for same alert, then check the Notifier.OUT, see how many entries of same alarms are present.

It will also tell you to which email address the alarm was sent.

 

If the issue reappears again. Please send screenshots of two sample duplicate emails and also upload the NOTIFIER.OUT file.

 

Note: For a quick resolution we would request to open a support ticket so we can remote to diagnose the problem.

 

thanks...

[Anon Anon]

I have a support call for this issue today. Thanks for everyone's help. As you can imagine this issue is a major pain right now as I can't really move on to other things I need to do if I can get spectrum do a basic alarm right.

[dhapr07]

can you share the support ticket number. So i can have a look.

[Anon Anon]

22001925

[Anon Anon]

I will be in the office at 8:30am EST. You can email me the webex invite and I will sign in as soon as I get in if that would work.

[Legacy User]

Couple of things before you head off to support ...

 

You still haven't shown us all running applications ... can you go to Locater search, expand SANM and run All Applications search. Can you either post the output or else confirm that there is only 1 entry?

 

Did you get round to running a 2nd AlarmNotifier using the default scripts? What was the outcome of that?

 

Can you post the SetScript file in case there's something obvious in there?

[Anon Anon]

I can send those when I get to work

[Anon Anon]

By the way, do you work for CA?

[Legacy User]

Nope

[Anon Anon]

It looks like I am finally fixed now and I am getting the alerts as I would expect them however I do believe the frequency of the alerts may be to much.  I want to increase the DCM Timeout and Retry from 3000ms and 3 to something like 6000ms and 3.  Where can I do this globally so all devices will get that new count?

[Legacy User]

You can do that using the Attribute Editor. (Do a Locater search for All Devices, select them all, then right click and under Utilities select the Attribute Editor)

 

Do you mind telling us what the problem was that caused the multiple emails?

[Anon Anon]

The problem was the idiot sitting in the chair

[Legacy User]

Great answer ... made me smile

[Anon Anon]

Now that I have the alert issue taken care of, I do believe I am getting a lot of false negatives from Spectrum indicating that a device is down when it actually isn't.  I think the time intervals that Spectrum uses may be too sensitive.  I'd like to be notified that a device is truly down only when Spectrum fails to poll that device within a 5 min time frame.  Is this set with the DCM Timeout and DCM Retry Count?

[Robert Kettles]

While you could do this with DCM Timeout and DCM Retry Count, I generally have only changed those on individual devices that had fragile management agent implementations (old DEC GigaSwitches if folks still remember those) or where the connectivity was poor.  If you want to cut back on the ticket/email volume, I would instead introduce a delay in the SANM policy:

 

 

Then you can investigate individual time outs that may be due to packet loss or something else.

[Anon Anon]

Can you explain exactly what the age time means?  Thanks so much for all your help on this.

[Robert Kettles]

In my example, it means for alarms that match the filter criteria, wait for it to be 5 minutes old before passing it to AlarmNotifier.  If the alarm clears within that 5 minute wait period, it's never sent to AlarmNotifier.  The alarms still show up in OneClick and you can report on it and investigate but it won't generate emails/tickets unless the condition is active for 5 minutes.

[Anon Anon]

never mind.  I figured it out.

[dhapr07]

John,

 

the SpectroServer was on Linux and there were many instance of alarm notifiers running in the back end from so long time.

Killed all the process. and then used the best processd way to start the alarmnotifier

things are good now.

 

you were right on multiple instance of alarm notifier application... Good thought

 

 

cheers

[dhapr07]

you can use the same procedure as John replied to change the DCM timeouts and DCM retries.