DX NetOps

Expand all | Collapse all

CA Performance Management Center - Configure Notification for Non-Threshold violation Events

Jump to Best Answer
  • 1.  CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-27-2021 10:40 PM
    We have multiple perl/python script put under /PerformanceCenter/NotificationScripts/ and they worked pretty well for sending alerts to other systems when violation to defined threshold profiles detected.

    Today I tried to use the similar way to trigger alerts for some other events such as device rebooted. At first I used email notification and put something like
    {ItemName}, {EventDesc} {EventType} {EventSubType}and I did receive email as expected when devices rebooted.

    Then I create a new script to do same thing(follow the same way we did for threshold profile), and as the TechDoc instructed, reference some values such as :

    $node = $ENV{'CAPM_ItemParentName'};
    $subcomponent = $ENV{'CAPM_ItemName'};
    $message = $ENV{'CAPM_EventDesc'};
    $eventtype = $ENV{'CAPM_EventType'};
    $eventsubtype = $ENV{'CAPM_EventSubType'};

    However I didn't get any alerts. Before sending alerts, my script write the info to my log file, however, I could not find such events in log file either. That means it didn't even trigger the new script I put under /PerformanceCenter/NotificationScripts/

    I'm wondering if NetOps uses different way for other event types such as "reconfiguration",  "Administration Event"... ?











  • 2.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Broadcom Employee
    Posted 07-28-2021 09:48 AM
    If you click on the Threshold Profile tab, are there any profiles/folders add to the right side?

    If so, remove them.  If they are there, it means only events for those threshold profiles will pass the filters.

    And the group you applied to the notification has the devices that have the reconfig events?


  • 3.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 10:18 AM
    Hi Jeff,

    We do have a lot of folders/profiles at "Threshold Profile" page. They are main alerting for us so I cannot remove them. They are using similar script and have been working for years.

    I do apply all groups/devices for testing purpose and send to a testing queue to avoid annoying others. And I did observed "rebooted" events from CA PM Event page. Also if I use email as notification action, it works and I get emails.

    Is there a way to trigger such notification w/o deleting our existing threshold profiles for "reconfiguration" types of events?


  • 4.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Broadcom Employee
    Posted 07-28-2021 10:28 AM

    Sorry, not asking to remove actual Threshold Profiles from DA admin views.   I was referring to the Threshold Profile tab in the notification creation/editor under stage "3 Conditions".  See it the above screenshot you provided?

    We want to make sure NO threshold profile/folder is selected for that tab in notifications.
    We should not be touching Threshold Profiles under DA Administration pages.

    So you enabled Email on the same 1 notification entry?  and email works but not script?

    Check the EMService.log around the time the script should run.  It should print a message if it was able to run successfully or not.




  • 5.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 11:08 AM
    Hi Jeff,
    sorry for misunderstanding your question.
    Yes when I checked the threshold tab from notification "condition" config, they are actually grayed out(additional screenshot attached here). I believe because I selected the event type like the screenshot I put with my question above, make them grayed out automatically.

    Yes I enables email on the exactly same notification entry, only difference is, I use this format to insert into email: {EventDesc}. But for script, I used this format to be same with other working scripts: $ENV{'CAPM_EventDesc'}; I will check EMService.log then update as soon as I can. Thanks again Jeff!






  • 6.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 02:08 PM
    Hi Jeff:

    I found at the same time when I picked one of the latest reboot event from PM GUI,  the event was detected from EMService.log. I marked some info, such as itemID with 123456. (the actual ID is different from NetOps device context page URL device ID, I think that is normal)

    --------------------------------------
    INFO | qtp12345678-00 | 2021-07-28 11:53:43,802 | org.apache.cxf.interceptor.LoggingInInterceptor
    | Inbound Message
    ----------------------------
    ID: 654321
    Address: http://mypc.mydomain.com:xxxx/EventManager/EventManagerWS.asmx
    Encoding: UTF-8
    Http-Method: POST
    Content-Type: text/xml; charset=UTF-8
    Headers: {Accept=[*/*], Cache-Control=[no-cache], connection=[keep-alive], Content-Length=[1800], content-type=[text/xml; charset=UTF-8], Host=[mypc.mydomain.com:xxxx], Pragma=[no-cache], SOAPAction=["http://netqos.com/nqevents/EventManager/Push"], User-Agent=[Apache CXF 2.7.11]}
    Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns5:Push xmlns:ns2="http://netqos.com/nqevents" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns4="http://netqos.com/nqevents/filter" xmlns:ns5="http://netqos.com/nqevents/EventManager" xmlns:ns6="http://schemas.xmlsoap.org/soap/encoding/"><ns5:events><ns5:Event><ns2:Id>040f5a21-7a21-4501-bdfc-3fad9253a3ca</ns2:Id><ns2:Type>ReconfigEvent</ns2:Type><ns2:SubType>DeviceReboot</ns2:SubType><ns2:Category>Config</ns2:Category><ns2:Producer>3</ns2:Producer><ns2:Items><ns2:EventItem><ns2:Id>123456</ns2:Id><ns2:Type>device</ns2:Type><ns2:SubType>router</ns2:SubType><ns2:IsIndependent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/></ns2:EventItem></ns2:Items><ns2:States><ns2:EventState><ns2:Id>040f5a21-7a21-4501-bdfc-3fad9253a3ca</ns2:Id><ns2:State>Closed</ns2:State><ns2:Description>A device reboot was detected during this poll period. The device last restarted 0:00:07.70 ago.</ns2:Description><ns2:TimeStamp>1627487621</ns2:TimeStamp><ns2:Comment>A device reboot was detected during this poll period. The device last restarted 0:00:07.70 ago.</ns2:Comment><ns2:LocalizationKey>com.ca.im.logging.InventoryDiscovery.130.DeviceRebootLessThan1Day</ns2:LocalizationKey></ns2:EventState></ns2:States><ns2:Properties><ns2:EventProperty><ns2:Name>1</ns2:Name><ns2:string>0:00:07.70</ns2:string><ns2:LocalizedStrings/></ns2:EventProperty><ns2:EventProperty><ns2:Name>DataSourceItemId</ns2:Name><ns2:long>123456</ns2:long><ns2:LocalizedStrings/></ns2:EventProperty><ns2:EventProperty><ns2:Name>DataSourceItemParentId</ns2:Name><ns2:long>123456</ns2:long><ns2:LocalizedStrings/></ns2:EventProperty></ns2:Properties><ns2:References/></ns5:Event></ns5:events></ns5:Push></soap:Body></soap:Envelope>



  • 7.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Broadcom Employee
    Posted 07-28-2021 02:30 PM

    Any ScriptHandlerImpl messages ( The script <scriptName> did not execute...  OR  The exit code of the executed script <scriptname>...) around EmailHandlerImpl message ( Sending e-mail ... ) that sends out the email for this event?

    Or maybe a "Failed to execute script " with an exception?




  • 8.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 02:41 PM
    Thanks Jeff, with your hint I guess one step closer now, at about the same time I found this from EMService.log:

    ID: 111111
    Encoding: UTF-8
    Content-Type: text/xml
    Headers: {}
    Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns5:PushResponse xmlns:ns2="http://netqos.com/nqevents" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns4="http://netqos.com/nqevents/filter" xmlns:ns5="http://netqos.com/nqevents/EventManager" xmlns:ns6="http://schemas.xmlsoap.org/soap/encoding/"/></soap:Body></soap:Envelope>
    --------------------------------------
    ERROR | pool-4-thread-1 | 2021-07-28 11:53:43,809 | com.ca.im.portal.em.notifications.ScriptHandlerImpl
    | Failed to execute script java.io.IOException: Cannot run program "/****/***/****/PerformanceCenter/NotificationScripts/test_aaaaa_abcdefghijk.pl": error=2, No such file or directory
    INFO | qtp1234567-11 | 2021-07-28 11:54:38,803 | org.apache.cxf.interceptor.LoggingInInterceptor
    | Inbound Message



  • 9.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 02:50 PM
    And from there on the CA PC server notificationscript folder, I searched the script name by copy paste to the log file, and matched identical, all lower case. Also compare to other working scripts(the red scratch one is the one I am working on), file attributes are same:



  • 10.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Broadcom Employee
    Posted 07-28-2021 02:56 PM

    It's a perl script, so is the first line #!/bin/perl  ?

    Does the /path/perl you are using exist?  If not, then the script will fail to run, maybe with same error.​

    If you run the perl script from command line, does it run?  print something or throw an error?

    The file perms look fine.  The PC runs are root?
    If you change the notification config to run one of the other scripts, does it throw the same error or work?

    For security, I would suggest doing 700 for perms on those, if PC runs as root.  Only root should be able to read/execute them from web server, or commandline.




  • 11.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 03:17 PM
    Yes 1st line is #!/bin/perl ;

    /path/perl exists, because other scripts on the same path work well;

    yes services are running as root, and some of other scripts has root ownership and work fine;

    changed perms to 700 from 755, just wondering isn't 700 is more strict than 755?




  • 12.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Broadcom Employee
    Posted 07-28-2021 03:32 PM

    Yes, it's more restrictive.  It's not gonna fix things, but make things more secure.   It should not break the other notifications using the other scripts.

    This returns syntax ok ?
    /bin/perl -c <scriptname>

    Running out of ideas why not working...  Is there a space at the start or end of the script name in the notification configuration dialog?  remove any leading or trailing spaces, if any.




  • 13.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 04:52 PM
    I ran some tests like this:

    uder ..../PerformanceManagement/NotificationScripts/ folder:

    ->perl ./myworkingscript.pl
    ->perl ./mynonworkingscript.pl

    Both return nothing, assume no interpreter errors.

    Then I tried this under .../PerformanceManagement/NotificationScripts/ folder:
    ->./myworkingscript.pl
    it returns nothing

    ->./mynonworkingscript.pl
    it returned error:

    -ksh ./mynonworkingscript.pl: not found [No Such file or directory]







  • 14.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Broadcom Employee
    Posted 07-28-2021 05:01 PM

    ksh ./nonworkingscript.pl not working may make sense since ksh is not the right script interpreter.

    What did you say was the first line of the script?  complete line.

    I think java may be reading the first line to determine what to use to run the script, and it's not exactly and ONLY:

    #!/bin/perl

    ​It may fail to run.

    Did you try running from commandline (no ksh, no perl before it)

    -> ./nonworkingscript.pl

    What does that say?




  • 15.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 05:11 PM
    my first line is (I have a bunch of other perl scripts have such 1st lines, and they are working fine):
    #!/usr/bin/perl

    Yes when I ran -> ./nonworkingscript.pl , no perl before it. working script returned nothing, but only this non-working one returned "-ksh ./mynonworkingscript.pl: not found [No Such file or directory]"




  • 16.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events
    Best Answer

    Broadcom Employee
    Posted 07-28-2021 05:30 PM
    Not sure what to try next.  Clearly when run without perl before the name, it won't run.
    Did you try using dos2unix on the file maybe?

    perl script.pl is bypassing using the first line of the script to determine interpreter to use.  Appears for some reason ksh is under able to determine to use /usr/bin/perl.

    Once you can get the script to work on the command line without error, Notifications will use it correctly.



  • 17.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 05:49 PM
    actually I tried dos2unix on this script after reading your reply, and now that the error message disappeared when I run ->./nonworkingscript.pl

    and now it returned nothing, which might be a good sign!!!

    I will observe until next reboot event caught and watch our alerting processing systems, I will update here as soon as I can.
    Really appreciate your efforts today on this question !!!




  • 18.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-29-2021 12:16 AM
    alert from our event mgmt system has been detected. Again really appreciate! you've spent a lot of time to help me solving this issue!!!


  • 19.  RE: CA Performance Management Center - Configure Notification for Non-Threshold violation Events

    Posted 07-28-2021 04:54 PM
    When I tried ->perl -c ./xxxxxxxxxx.pl

    both returned :
    ./xxxxxxxxx.pl syntax OK