DX NetOps

 View Only
Expand all | Collapse all

Spectrum Alarm Notifier Fault Tolerance?

  • 1.  Spectrum Alarm Notifier Fault Tolerance?

    Posted Oct 17, 2011 11:51 AM
    Has anyone implemented fault tolerance for Alarm Notifier? Is it even possible? I'd like to hear your insight and thoughts.

    I currently run Alarm Notifier on the primary SS server. But when it fails and the secondary SS takes over we do not get alarm notifications because it is not running on the secondary.

    Thanks
    Chris


  • 2.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Oct 17, 2011 11:58 AM
    HI,

    Alarmnotifier is not a fault tolerant piece. We ran into that when we set up fault tolerance in spectrum 8.1. I'm pretty sure I had logged an enhancement for this but I don't know where it's at wth CA. Guess I should look into it.

    What we ended up doing is modifying our scripts. We now have alarmnotiifer running on every DSS in the envrionment. Each DSS sends its alarms for its own server to our internal sytem to have tickets logged. It was the only way to get it running when fault tolerance would take over.

    Doris.


  • 3.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Oct 17, 2011 01:03 PM
    I have the same issue, I did find this , but have not tried it yet:
    http://www.dachsug.ch/wiki/index.php/Alarming#Redundant_AlarmNotifier


    Thanks,
    Rob


  • 4.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Oct 19, 2011 07:33 AM
    I am the creator of the Wiki entry.
    I use the solution decribed in there for 2 customers without problems.
    If anybody has some hints or useful extensions, please post them in the Wiki.

    Thanks.


  • 5.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Oct 24, 2011 11:42 AM
    Hi Chris:

    Did any of the responses below answer your question? If so, please check the "Mark as Accepted Solution" box in the posts that helped. This will indicate that your issue is Resolved.

    Thanks,
    Mary


  • 6.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Jul 20, 2012 04:34 PM
    Well that was a funny moment. I google searched for this very same issue today and came across this post. I read it and all the replies not noticing at first who wrote the original question. Apparently I forgot I posted this question and am now just coming back around to it.

    At this time I am unable to reach the Wiki page, it could be blocked but it just spins, so I will try to access it later tonight from home and see if it actually answers my question.

    Sorry for such a delayed response.

    Thanks :)


  • 7.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Broadcom Employee
    Posted Jul 22, 2012 10:40 AM
    There have been many ER's logged for AlarmNotifier fault tolerance but to be honest, I am not sure if that is something that we will be doing out of the box in a future release. Here is a tech note that we have to make AlarmNotifier fault tolerant ....


    Alarm Notifier can be configured to run in a fault tolerant environment so that alarm notification still occurs when the primary server is down and the secondary server is active. In order to do this, the AlarmNotifier application must be running on both the primary and secondary SpectroSERVERs. The SetScript file (located in the /Notifier directory) will need to be modified on both the primary and the secondary machine to include an if/then statement that utilizes the $SERVER parameter.



    On the primary machine, you will need to add an if/then statement to the SetScript file so that if the $SERVER parameter equals the name of the primary SpectroSERVER, then send the email notification.

    On the secondary machine, you will need to add an if/then statement to the SetScript file so that if the $SERVER parameter equals the name of the secondary SpectroSERVER, then send the email notification.

    If you are using the UpdateScript and ClearScript, the same modifications will need to be made.

    Related Issues/Questions:
    How can the SPECTRUM Alarm Notifier be configured to run in a fault tolerant environment?
    AlarmNotifier fault tolerance
    Alarm Notifier does not fail over when the primary server fails. How can the Alarm Notifier be configured so that alarm notifications will still occur whent the secondary server is active?

    Causes of this problem:
    Alarm Notifier is not a fault tolerant application.

    Additional Information:
    SPECTRUM Support is not responsible for creating scripts for customers so modifications are done at the customers own risk.
    Alarm Notifier can be configured to run in a fault tolerant environment so that alarm notification still occurs when the primary server is down and the secondary server is active. In order to do this, the AlarmNotifier application must be running on both the primary and secondary SpectroSERVERs. The SetScript file (located in the /Notifier directory) will need to be modified on both the primary and the secondary machine to include an if/then statement that utilizes the $SERVER parameter.



    On the primary machine, you will need to add an if/then statement to the SetScript file so that if the $SERVER parameter equals the name of the primary SpectroSERVER, then send the email notification.

    On the secondary machine, you will need to add an if/then statement to the SetScript file so that if the $SERVER parameter equals the name of the secondary SpectroSERVER, then send the email notification.

    If you are using the UpdateScript and ClearScript, the same modifications will need to be made.

    Related Issues/Questions:
    How can the SPECTRUM Alarm Notifier be configured to run in a fault tolerant environment?
    AlarmNotifier fault tolerance
    Alarm Notifier does not fail over when the primary server fails. How can the Alarm Notifier be configured so that alarm notifications will still occur whent the secondary server is active?

    Causes of this problem:
    Alarm Notifier is not a fault tolerant application.

    Additional Information:
    SPECTRUM Support is not responsible for creating scripts for customers so modifications are done at the customers own risk.


    .... I just want to reinforce - writing custom scripts is beyond the scope of support so keep that in mind if you want to try this

    HTH,
    Joe


  • 8.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Jul 23, 2012 02:37 AM
    Hi All,

    The one posted in the link and the other which is given by Joe should work.

    1. The link uses precedence as an attribute.
    2. Tec doc uses server as an attribute in the .alarmrc.

    Let me know if you still have the issue.

    kalyan


  • 9.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Broadcom Employee
    Posted Jul 23, 2012 03:42 AM
    Joe is correct. The PM team have received a few ERs in the Alarm Notified area. But given the other prioritiy areas that have been identified, we havent been able to accept the Alarm Notifier ones.
    To confirm on Joe's point below, the upcoming release does not have any updates to the alarm notifier area.

    Regards,
    Kiran Diwakar


  • 10.  RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Jul 26, 2012 07:05 PM
    Here is my solution which works on the assumption that if primary server is down (and alarm notifier) then secondary server and alarm notifier will alert - as an aside I set alarm notifer to autostart and autorestart, I run scripts to check tthe status of Alarm notifier and if that files I use paging systems to alert me

    Anyway
    Your primary spec DB will have precedence 10 (thats attribute 0x12c0a in every model)
    Assume your secondary Spectrum is precendence 20

    On both servers add this to $specroot/Notifier/.alarmrc
    EXTRA_ATTRS_AS_ENVVARS=0X12C0A

    In setscript and clearscript in the bit just after

    if [ "$SENDMAIL" = "True" ]
    then
    RECIPIENTS=$VARFORMAIL
    ........."
    RECIPIENTS="NotificationData/RepairPerson"
    fi

    On primary add this -
    if [[ "$SANM_0X12C0A" = "20" ]]
    then
    echo "SS Secondary is running"
    echo "Precedence = $SANM_0X12C0A"
    exit 0
    fi

    On secondary add this

    if [[ "$SANM_0X12C0A" = "10" ]]
    then
    echo "SS Primaryis running"
    echo "Precedence = $SANM_0X12C0A"
    exit 0
    fi





    What this will do is
    whenever an alarm is generated the model in the DB is checked and the attribute 0x12x0a is read - if its 10 (primary server precedence) then Primary Alarm notifier sends it and secondary will write a line to the notifier log file saying primary is running
    If 0x12x0a is 20 then Seconday server Alarm notifier will send the mail and the primary would write to the notifier file saying seconday is running

    Been using this for 4 years now no issues


  • 11.  Re: [CA Spectrum] RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Jul 27, 2012 02:03 AM
    That is fantastic thanks for sharing


    Sent from Blackberry please excuse Brevity.


    From: CA Infrastructure Management Global User Community (eHealth/Spectrum/NetQoS) [mailto:CommunityAdmin@communities-mail.ca.com]
    Sent: Friday, July 27, 2012 12:05 AM
    To: mb.2239493.98650223@myca-email.ca.com <mb.2239493.98650223@myca-email.ca.com>
    Subject: [CA Spectrum] RE: Spectrum Alarm Notifier Fault Tolerance?


    Here is my solution which works on the assumption that if primary server is down (and alarm notifier) then secondary server and alarm notifier will alert - as an aside I set alarm notifer to autostart and autorestart, I run scripts to check tthe status of Alarm notifier and if that files I use paging systems to alert me

    Anyway
    Your primary spec DB will have precedence 10 (thats attribute 0x12c0a in every model)
    Assume your secondary Spectrum is precendence 20

    On both servers add this to $specroot/Notifier/.alarmrc
    EXTRA_ATTRS_AS_ENVVARS=0X12C0A

    In setscript and clearscript in the bit just after

    if [ "$SENDMAIL" = "True" ]
    then
    RECIPIENTS=$VARFORMAIL
    ........."
    RECIPIENTS="NotificationData/RepairPerson"
    fi

    On primary add this -
    if [[ "$SANM_0X12C0A" = "20" ]]
    then
    echo "SS Secondary is running"
    echo "Precedence = $SANM_0X12C0A"
    exit 0
    fi

    On secondary add this

    if [[ "$SANM_0X12C0A" = "10" ]]
    then
    echo "SS Primaryis running"
    echo "Precedence = $SANM_0X12C0A"
    exit 0
    fi





    What this will do is
    whenever an alarm is generated the model in the DB is checked and the attribute 0x12x0a is read - if its 10 (primary server precedence) then Primary Alarm notifier sends it and secondary will write a line to the notifier log file saying primary is running
    If 0x12x0a is 20 then Seconday server Alarm notifier will send the mail and the primary would write to the notifier file saying seconday is running

    Been using this for 4 years now no issues
    Posted by:jasonheb
    --
    CA Communities Message Boards
    https://communities.ca.com/web/ca-ehealth-and-ca-spectrum-global-user-community/message-boards/-/message_boards/view_message/98650223
    mb.2239493.98650223@myca-email.ca.com
    https://communities.ca.com
    Please be aware that Logicalis UK Ltd may monitor email traffic data and also email content for security purposes.
    ______________________________________________________________________
    This email has been scanned by the Symantec Email Security.cloud service.
    For more information please visit http://www.symanteccloud.com
    ______________________________________________________________________


    This email communication does not create or vary any contractual relationship between Logicalis and you. Internet communications are not secure and accordingly Logicalis does not accept any legal liability for the contents of this message. The contents of this email are confidential to the intended recipient at the email address to which it has been addressed. It may not be disclosed to or used by anyone other than this addressee, nor may it be copied in any way. If received in error, please contact Logicalis on the above switchboard number quoting the name of the sender and the addressee and then delete it from your system. Please note that neither Logicalis nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any).

    Please be aware that Logicalis UK Ltd may monitor email traffic data and also email content for security purposes.

    Logicalis UK Ltd, Registered in England and Wales No: 3732397, Registered Office: 110 Buckingham Avenue, Slough. Berkshire, SL1 4PF

    ______________________________________________________________________
    This email has been scanned by the Symantec Email Security.cloud service.
    For more information please visit http://www.symanteccloud.com
    ______________________________________________________________________


  • 12.  RE: Re: [CA Spectrum] RE: Spectrum Alarm Notifier Fault Tolerance?

    Posted Aug 01, 2012 11:31 PM
    Thanks for the reply. I will give that a try next week as soon as I am done getting through some audit requirements that take priority.

    Also of note, a Google search of this turns up something about Alarm Notifier Fault Tolerance on site www.dachsug.ch/wiki/... but I thought I would mention the site no longer comes up. Either way, the solution above makes sense and I hope to give it a try.

    Its too bad that complete fault tolerance is not higher on the priority list at CA for Spectrum. Given the secondary ever has to take over we have lost all automated notification communications from Spectrum to engineers and many other functions are lost until the primary is back online.

    Thanks :)