Spectrum Connector issue with 2.0.0.238

[Saju Mathew]

Hi All,

 

I have ran into a Spectrum connector issue, The connector services stops, reinitializes every two or 3 minutes. After few hours it go to a closing state. I have an case open with SOI and Spectrum to look into this. I has suggested by Spectrum team to Uninstall and reinstall the connector. Before doing that, I wana make sure, I am not missing any thing out.

 

Note: Connector server is a cluster environment

 

Kindly help me with the same.

 

When I looked into the Spectro server's spectro CORBA communication debug log I found this:

 

Mar 17 12:46:47 CORBA TRACE at CModelDomainItcN.cc(1534): Client SpectrumConnector@BRIPWVIMAPP02 (app uid 5aad31fd-0000-1000-1fa9-a9fe02eb0000)  is starting a heartbeat callback.
Mar 17 12:46:47 CORBA TRACE at CModelDomainItcN.cc(1153): Heartbeat callback thread for client SpectrumConnector@BRIPWVIMAPP02 has been started.
Mar 17 12:47:10 CORBA TRACE at CsWatchNotificationQueue.cc(150): An exception was raised by a(n) alarm callback , client svccaim@bripwvimapp02 : CORBA::OBJECT_NOT_EXIST - the client has been removed.
Mar 17 12:47:53 CORBA TRACE at CModelDomainItcN.cc(1345): An exception was raised by a heartbeat callback to client SpectrumConnector@BRIPWVIMAPP02: CORBA::OBJECT_NOT_EXIST - trying again...
Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1601): Client SpectrumConnector@BRIPWVIMAPP02 (app uid 5aad31fd-0000-1000-1fa9-a9fe02eb0000) is stopping a heartbeat callback.
Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1613): Informing the moot about the disconnected client
Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1329): An exception was raised by a heartbeat callback to client SpectrumConnector@BRIPWVIMAPP02 indicating the client is no longer running: CORBA::OBJECT_NOT_EXIST - the client has been removed.
Mar 17 12:48:14 CORBA TRACE at CModelDomainItcN.cc(1439): Heartbeat callback thread for client SpectrumConnector@BRIPWVIMAPP02 has been stopped.
Mar 17 12:48:17 CORBA TRACE at CsCExcptHlpr.cc(580): Throwing exception : CsCModelDomain.stopWatchAttrValsOfModels.callback.DOES_NOT_EXIST.DOES_NOT_EXIST
Mar 17 12:48:17 CORBA TRACE at CsCExcptHlpr.cc(580): Throwing exception : CsCModelDomain.stopWatchModels.callback.DOES_NOT_EXIST.DOES_NOT_EXIST
Mar 17 12:48:17 CORBA TRACE at CsCExcptHlpr.cc(726): Throwing exception : CsCAlarmDomain.stopWatchAlarms.return.DOES_NOT_EXIST

 

 

The below log is from SAMIntegration service logs in the Connector server :

 

CA:[e-mail] MdrMonitor thread exiting...
INFO   | jvm 1    | 2018/03/17 12:49:48 | CatalogDaemon.main:interrupted exception:sleep interrupted
INFO   | jvm 1    | 2018/03/17 12:49:48 | java.lang.InterruptedException: sleep interrupted
INFO   | jvm 1    | 2018/03/17 12:49:48 |  at java.lang.Thread.sleep(Native Method)
INFO   | jvm 1    | 2018/03/17 12:49:48 |  at com.ca.sam.ifw.eventplus.catalog.CatalogDaemon$2.run(CatalogDaemon.java:307)
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : OFFLINE - Connector is restarting due to critical error: MDR host bripwvimapp03 is unavailable : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : OFFLINE - Connector is restarting due to critical error: MDR host bripwvimapp03 is unavailable : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | ********** CA:[e-mail] Stopped.  Connector is restarting due to critical error: MDR host bripwvimapp03 is unavailable
INFO   | jvm 1    | 2018/03/17 12:49:48 | name of Silo is CA:[e-mail]
INFO   | jvm 1    | 2018/03/17 12:49:48 | name of Silo siloInst.implementationClass before load is com.ca.usm.ucf.spectrum.SpectrumUCFConnector
INFO   | jvm 1    | 2018/03/17 12:49:48 | Attempting to start silo adaptor: CA:[e-mail]
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : OFFLINE - Connector is loading : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : OFFLINE - Connector is loading : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | null:Waiting for Event Processing to Finish...
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : HANDSHAKE - Attempting to register connector with SOI manager : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : HANDSHAKE - Attempting to register connector with SOI manager : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : HANDSHAKE - Waiting for connector-specific initialization : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | 2018/03/17 12:49:47 : bripwvnmapc01.ad.bcbsri.org : CA:[e-mail] : HANDSHAKE - Waiting for connector-specific initialization : Broadcasting HeartBeat Message
INFO   | jvm 1    | 2018/03/17 12:49:48 | ---------> calling CA:[e-mail]: initialize
INFO   | jvm 1    | 2018/03/17 12:49:48 | <--------- CA:[e-mail]: initialize returned in 93 ms.
INFO   | jvm 1    | 2018/03/17 12:49:48 |

[Tarun Pamu]

Hi Saju,

 

This looks like a communications issue between the SS and connector machine. Check the below KB and see if it helps.

 

Spectrum connector restarts frequently - CA Knowledge 

[Saju Mathew]

Hi Tarun,

 

– Port 14002 (SpectroSERVER) 

– Port 14004 (Location Server) 

– Port 14006 (CORBA Name Server) 

 

All the three components are running on same server.

 

Only the connector is running on separate... So, the Spectro server should be able to telnet connector server using the port 14001 right? Do, I need to bother about other ports, as It will be communicating?

 

Regards,

Saju Mathew

[Saju Mathew]

Hi Tarun,

 

Increasing the Heartbeat also didn't help. I did cross check, the ports seems fine I am able to telnet from Spectro server to connector server using 14001 port. Also, alerts come in when the connectors reinitialize everytime. 

 

Just for Information, The connector is in a cluster environment.

 

Regards,

Saju Mathew

[Legacy User]

Hello Saju,

 

You have already opened 2 cases for this with support one in the Spectrum team and one in the SOI team- This will need to be addressed via those cases.

 

Regards

 

Declan

[Saju Mathew]

I understand Declan,

 

Post re-configuration I just wanted to be sure, If I can get any input from anyone who might have got the same issue and fixed the same instead of re-installing the connectors.

 

Regards,

Saju Mathew