DX NetOps

Expand all | Collapse all

Spectrum Connector issue with 2.0.0.238

  • 1.  Spectrum Connector issue with 2.0.0.238

    Posted 03-17-2018 12:57 PM

    Hi All,

     

    I have ran into a Spectrum connector issue, The connector services stops, reinitializes every two or 3 minutes. After few hours it go to a closing state. I have an case open with SOI and Spectrum to look into this. I has suggested by Spectrum team to Uninstall and reinstall the connector. Before doing that, I wana make sure, I am not missing any thing out.

     

    Note: Connector server is a cluster environment

     

    Kindly help me with the same.

     

    When I looked into the Spectro server's spectro CORBA communication debug log I found this:

     

    Mar 17 12:46:47 CORBA TRACE at CModelDomainItcN.cc(1534): Client SpectrumConnector@BRIPWVIMAPP02 (app uid 5aad31fd-0000-1000-1fa9-a9fe02eb0000)  is starting a heartbeat callback.
    Mar 17 12:46:47 CORBA TRACE at CModelDomainItcN.cc(1153): Heartbeat callback thread for client SpectrumConnector@BRIPWVIMAPP02 has been started.
    Mar 17 12:47:10 CORBA TRACE at CsWatchNotificationQueue.cc(150): An exception was raised by a(n) alarm callback , client svccaim@bripwvimapp02 : CORBA::OBJECT_NOT_EXIST - the client has been removed.
    Mar 17 12:47:53 CORBA TRACE at CModelDomainItcN.cc(1345): An exception was raised by a heartbeat callback to client SpectrumConnector@BRIPWVIMAPP02: CORBA::OBJECT_NOT_EXIST - trying again...
    Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1601): Client SpectrumConnector@BRIPWVIMAPP02 (app uid 5aad31fd-0000-1000-1fa9-a9fe02eb0000) is stopping a heartbeat callback.
    Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1613): Informing the moot about the disconnected client
    Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1329): An exception was raised by a heartbeat callback to client SpectrumConnector@BRIPWVIMAPP02 indicating the client is no longer running: CORBA::OBJECT_NOT_EXIST - the client has been removed.
    Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1439): Heartbeat callback thread for client SpectrumConnector@BRIPWVIMAPP02 has been stopped.
    Mar 17 12:48:17 CORBA TRACE at CsCExcptHlpr.cc(580): Throwing exception : CsCModelDomain.stopWatchAttrValsOfModels.callback.DOES_NOT_EXIST.DOES_NOT_EXIST
    Mar 17 12:48:17 CORBA TRACE at CsCExcptHlpr.cc(580): Throwing exception : CsCModelDomain.stopWatchModels.callback.DOES_NOT_EXIST.DOES_NOT_EXIST
    Mar 17 12:48:17 CORBA TRACE at CsCExcptHlpr.cc(726): Throwing exception : CsCAlarmDomain.stopWatchAlarms.return.DOES_NOT_EXIST

     

     

    The below log is from SAMIntegration service logs in the Connector server :

     

    CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org MdrMonitor thread exiting...
    INFO   | jvm 1    | 2018/03/17 12:49:48 | CatalogDaemon.main:interrupted exception:sleep interrupted
    INFO   | jvm 1    | 2018/03/17 12:49:48 | java.lang.InterruptedException: sleep interrupted
    INFO   | jvm 1    | 2018/03/17 12:49:48 |  at java.lang.Thread.sleep(Native Method)
    INFO   | jvm 1    | 2018/03/17 12:49:48 |  at com.ca.sam.ifw.eventplus.catalog.CatalogDaemon$2.run(CatalogDaemon.java:307)
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : OFFLINE - Connector is restarting due to critical error: MDR host bripwvimapp03 is unavailable : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : OFFLINE - Connector is restarting due to critical error: MDR host bripwvimapp03 is unavailable : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | ********** CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org Stopped.  Connector is restarting due to critical error: MDR host bripwvimapp03 is unavailable
    INFO   | jvm 1    | 2018/03/17 12:49:48 | name of Silo is CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org
    INFO   | jvm 1    | 2018/03/17 12:49:48 | name of Silo siloInst.implementationClass before load is com.ca.usm.ucf.spectrum.SpectrumUCFConnector
    INFO   | jvm 1    | 2018/03/17 12:49:48 | Attempting to start silo adaptor: CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : OFFLINE - Connector is loading : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : OFFLINE - Connector is loading : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | null:Waiting for Event Processing to Finish...
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : HANDSHAKE - Attempting to register connector with SOI manager : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : HANDSHAKE - Attempting to register connector with SOI manager : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : HANDSHAKE - Waiting for connector-specific initialization : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org : HANDSHAKE - Waiting for connector-specific initialization : Broadcasting HeartBeat Message
    INFO   | jvm 1    | 2018/03/17 12:49:48 | ---------> calling CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org: initialize
    INFO   | jvm 1    | 2018/03/17 12:49:48 | <--------- CA:00005_Spectrum-bripwvimapp03@bripwvnmapc01.ad.bcbsri.org: initialize returned in 93 ms.
    INFO   | jvm 1    | 2018/03/17 12:49:48 |



  • 2.  Re: Spectrum Connector issue with 2.0.0.238

    Broadcom Employee
    Posted 03-18-2018 09:56 PM

    Hi Saju,

     

    This looks like a communications issue between the SS and connector machine. Check the below KB and see if it helps.

     

    Spectrum connector restarts frequently - CA Knowledge 



  • 3.  Re: Spectrum Connector issue with 2.0.0.238

    Posted 03-19-2018 09:15 AM

    Hi Tarun,

     

    – Port 14002 (SpectroSERVER) 

    – Port 14004 (Location Server) 

    – Port 14006 (CORBA Name Server) 

     

    All the three components are running on same server.

     

    Only the connector is running on separate... So, the Spectro server should be able to telnet connector server using the port 14001 right? Do, I need to bother about other ports, as It will be communicating?

     

    Regards,

    Saju Mathew



  • 4.  Re: Spectrum Connector issue with 2.0.0.238

    Posted 03-19-2018 11:57 AM

    Hi Tarun,

     

    Increasing the Heartbeat also didn't help. I did cross check, the ports seems fine I am able to telnet from Spectro server to connector server using 14001 port. Also, alerts come in when the connectors reinitialize everytime. 

     

    Just for Information, The connector is in a cluster environment.

     

    Regards,

    Saju Mathew



  • 5.  Re: Spectrum Connector issue with 2.0.0.238

    Broadcom Employee
    Posted 03-20-2018 10:37 AM

    Hello Saju,

     

    You have already opened 2 cases for this with support one in the Spectrum team and one in the SOI team- This will need to be addressed via those cases.

     

    Regards

     

    Declan



  • 6.  Re: Spectrum Connector issue with 2.0.0.238

    Posted 03-20-2018 12:09 PM

    I understand Declan,

     

    Post re-configuration I just wanted to be sure, If I can get any input from anyone who might have got the same issue and fixed the same instead of re-installing the connectors.

     

    Regards,

    Saju Mathew