CA Performance Management Center - Configure Notification for Non-Threshold violation Events

[Tao Yang]

We have multiple perl/python script put under /PerformanceCenter/NotificationScripts/ and they worked pretty well for sending alerts to other systems when violation to defined threshold profiles detected.

Today I tried to use the similar way to trigger alerts for some other events such as device rebooted. At first I used email notification and put something like 
{ItemName}, {EventDesc} {EventType} {EventSubType}and I did receive email as expected when devices rebooted.

Then I create a new script to do same thing(follow the same way we did for threshold profile), and as the TechDoc instructed, reference some values such as :

$node = $ENV{'CAPM_ItemParentName'};
$subcomponent = $ENV{'CAPM_ItemName'};
$message = $ENV{'CAPM_EventDesc'};
$eventtype = $ENV{'CAPM_EventType'};
$eventsubtype = $ENV{'CAPM_EventSubType'};

However I didn't get any alerts. Before sending alerts, my script write the info to my log file, however, I could not find such events in log file either. That means it didn't even trigger the new script I put under /PerformanceCenter/NotificationScripts/

I'm wondering if NetOps uses different way for other event types such as "reconfiguration",  "Administration Event"... ?

[Jeffrey Pinard]

If you click on the Threshold Profile tab, are there any profiles/folders add to the right side?

If so, remove them.  If they are there, it means only events for those threshold profiles will pass the filters.

And the group you applied to the notification has the devices that have the reconfig events?

[Tao Yang]

Hi Jeff,

We do have a lot of folders/profiles at "Threshold Profile" page. They are main alerting for us so I cannot remove them. They are using similar script and have been working for years. 

I do apply all groups/devices for testing purpose and send to a testing queue to avoid annoying others. And I did observed "rebooted" events from CA PM Event page. Also if I use email as notification action, it works and I get emails.

Is there a way to trigger such notification w/o deleting our existing threshold profiles for "reconfiguration" types of events?

[Jeffrey Pinard]

Sorry, not asking to remove actual Threshold Profiles from DA admin views.   I was referring to the Threshold Profile tab in the notification creation/editor under stage "3 Conditions".  See it the above screenshot you provided?

We want to make sure NO threshold profile/folder is selected for that tab in notifications. 
We should not be touching Threshold Profiles under DA Administration pages.

So you enabled Email on the same 1 notification entry?  and email works but not script?

Check the EMService.log around the time the script should run.  It should print a message if it was able to run successfully or not.

[Tao Yang]

Hi Jeff,
sorry for misunderstanding your question.
Yes when I checked the threshold tab from notification "condition" config, they are actually grayed out(additional screenshot attached here). I believe because I selected the event type like the screenshot I put with my question above, make them grayed out automatically.

Yes I enables email on the exactly same notification entry, only difference is, I use this format to insert into email: {EventDesc}. But for script, I used this format to be same with other working scripts: $ENV{'CAPM_EventDesc'}; I will check EMService.log then update as soon as I can. Thanks again Jeff!

[Tao Yang]

Hi Jeff:

I found at the same time when I picked one of the latest reboot event from PM GUI,  the event was detected from EMService.log. I marked some info, such as itemID with 123456. (the actual ID is different from NetOps device context page URL device ID, I think that is normal)

--------------------------------------
INFO | qtp12345678-00 | 2021-07-28 11:53:43,802 | org.apache.cxf.interceptor.LoggingInInterceptor
| Inbound Message
----------------------------
ID: 654321
Address: http://mypc.mydomain.com:xxxx/EventManager/EventManagerWS.asmx
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml; charset=UTF-8
Headers: {Accept=[*/*], Cache-Control=[no-cache], connection=[keep-alive], Content-Length=[1800], content-type=[text/xml; charset=UTF-8], Host=[mypc.mydomain.com:xxxx], Pragma=[no-cache], SOAPAction=["http://netqos.com/nqevents/EventManager/Push"], User-Agent=[Apache CXF 2.7.11]}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns5:Push xmlns:ns2="http://netqos.com/nqevents" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns4="http://netqos.com/nqevents/filter" xmlns:ns5="http://netqos.com/nqevents/EventManager" xmlns:ns6="http://schemas.xmlsoap.org/soap/encoding/"><ns5:events><ns5:Event><ns2:Id>040f5a21-7a21-4501-bdfc-3fad9253a3ca</ns2:Id><ns2:Type>ReconfigEvent</ns2:Type><ns2:SubType>DeviceReboot</ns2:SubType><ns2:Category>Config</ns2:Category><ns2:Producer>3</ns2:Producer><ns2:Items><ns2:EventItem><ns2:Id>123456</ns2:Id><ns2:Type>device</ns2:Type><ns2:SubType>router</ns2:SubType><ns2:IsIndependent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></ns2:EventItem></ns2:Items><ns2:States><ns2:EventState><ns2:Id>040f5a21-7a21-4501-bdfc-3fad9253a3ca</ns2:Id><ns2:State>Closed</ns2:State><ns2:Description>A device reboot was detected during this poll period. The device last restarted 0:00:07.70 ago.</ns2:Description><ns2:TimeStamp>1627487621</ns2:TimeStamp><ns2:Comment>A device reboot was detected during this poll period. The device last restarted 0:00:07.70 ago.</ns2:Comment><ns2:LocalizationKey>com.ca.im.logging.InventoryDiscovery.130.DeviceRebootLessThan1Day</ns2:LocalizationKey></ns2:EventState></ns2:States><ns2:Properties><ns2:EventProperty><ns2:Name>1</ns2:Name><ns2:string>0:00:07.70</ns2:string><ns2:LocalizedStrings/></ns2:EventProperty><ns2:EventProperty><ns2:Name>DataSourceItemId</ns2:Name><ns2:long>123456</ns2:long><ns2:LocalizedStrings/></ns2:EventProperty><ns2:EventProperty><ns2:Name>DataSourceItemParentId</ns2:Name><ns2:long>123456</ns2:long><ns2:LocalizedStrings/></ns2:EventProperty></ns2:Properties><ns2:References/></ns5:Event></ns5:events></ns5:Push></soap:Body></soap:Envelope>

[Jeffrey Pinard]

Any ScriptHandlerImpl messages ( The script <scriptName> did not execute...  OR  The exit code of the executed script <scriptname>...) around EmailHandlerImpl message ( Sending e-mail ... ) that sends out the email for this event?

Or maybe a "Failed to execute script " with an exception?

[Tao Yang]

Thanks Jeff, with your hint I guess one step closer now, at about the same time I found this from EMService.log:

ID: 111111
Encoding: UTF-8
Content-Type: text/xml
Headers: {}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns5:PushResponse xmlns:ns2="http://netqos.com/nqevents" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns4="http://netqos.com/nqevents/filter" xmlns:ns5="http://netqos.com/nqevents/EventManager" xmlns:ns6="http://schemas.xmlsoap.org/soap/encoding/"/></soap:Body></soap:Envelope>
--------------------------------------
ERROR | pool-4-thread-1 | 2021-07-28 11:53:43,809 | com.ca.im.portal.em.notifications.ScriptHandlerImpl
| Failed to execute script java.io.IOException: Cannot run program "/****/***/****/PerformanceCenter/NotificationScripts/test_aaaaa_abcdefghijk.pl": error=2, No such file or directory
INFO | qtp1234567-11 | 2021-07-28 11:54:38,803 | org.apache.cxf.interceptor.LoggingInInterceptor
| Inbound Message

[Tao Yang]

And from there on the CA PC server notificationscript folder, I searched the script name by copy paste to the log file, and matched identical, all lower case. Also compare to other working scripts(the red scratch one is the one I am working on), file attributes are same:

[Jeffrey Pinard]

It's a perl script, so is the first line #!/bin/perl  ?

Does the /path/perl you are using exist?  If not, then the script will fail to run, maybe with same error.​

If you run the perl script from command line, does it run?  print something or throw an error?

The file perms look fine.  The PC runs are root?  
If you change the notification config to run one of the other scripts, does it throw the same error or work?

For security, I would suggest doing 700 for perms on those, if PC runs as root.  Only root should be able to read/execute them from web server, or commandline.

[Tao Yang]

Yes 1st line is #!/bin/perl ;

/path/perl exists, because other scripts on the same path work well;

yes services are running as root, and some of other scripts has root ownership and work fine;

changed perms to 700 from 755, just wondering isn't 700 is more strict than 755?

[Jeffrey Pinard]

Yes, it's more restrictive.  It's not gonna fix things, but make things more secure.   It should not break the other notifications using the other scripts.

This returns syntax ok ?
/bin/perl -c <scriptname>

Running out of ideas why not working...  Is there a space at the start or end of the script name in the notification configuration dialog?  remove any leading or trailing spaces, if any.

[Tao Yang]

I ran some tests like this:

uder ..../PerformanceManagement/NotificationScripts/ folder:

->perl ./myworkingscript.pl
->perl ./mynonworkingscript.pl

Both return nothing, assume no interpreter errors. 

Then I tried this under .../PerformanceManagement/NotificationScripts/ folder:
->./myworkingscript.pl
it returns nothing

->./mynonworkingscript.pl
it returned error:

-ksh ./mynonworkingscript.pl: not found [No Such file or directory]

[Jeffrey Pinard]

ksh ./nonworkingscript.pl not working may make sense since ksh is not the right script interpreter.

What did you say was the first line of the script?  complete line.

I think java may be reading the first line to determine what to use to run the script, and it's not exactly and ONLY:

#!/bin/perl

​It may fail to run.

Did you try running from commandline (no ksh, no perl before it)

-> ./nonworkingscript.pl

What does that say?

[Tao Yang]

my first line is (I have a bunch of other perl scripts have such 1st lines, and they are working fine):
#!/usr/bin/perl

Yes when I ran -> ./nonworkingscript.pl , no perl before it. working script returned nothing, but only this non-working one returned "-ksh ./mynonworkingscript.pl: not found [No Such file or directory]"

[Jeffrey Pinard]

Not sure what to try next.  Clearly when run without perl before the name, it won't run.
Did you try using dos2unix on the file maybe?

perl script.pl is bypassing using the first line of the script to determine interpreter to use.  Appears for some reason ksh is under able to determine to use /usr/bin/perl.

Once you can get the script to work on the command line without error, Notifications will use it correctly.

[Tao Yang]

actually I tried dos2unix on this script after reading your reply, and now that the error message disappeared when I run ->./nonworkingscript.pl

and now it returned nothing, which might be a good sign!!!

I will observe until next reboot event caught and watch our alerting processing systems, I will update here as soon as I can.
Really appreciate your efforts today on this question !!!

[Tao Yang]

alert from our event mgmt system has been detected. Again really appreciate! you've spent a lot of time to help me solving this issue!!!

[Tao Yang]

When I tried ->perl -c ./xxxxxxxxxx.pl

both returned :
./xxxxxxxxx.pl syntax OK