Steps OC client takes to connect to failover FT SS when primary is not available?

[Jon Velazquez]

Recently setup FT Spectroserver.  The DB synchronization is working.  However, the OneClick client does not turn yellow when I shutdown the primary SS. 

 

Would like to understand the steps taken by OC client to connect to the secondary in order to troubleshoot? 

 

Typically OC connects to TCP port 14002 on the Spectroserver.

 

On the secondary FT Spectroserver do not see TCP 14002 listening.  Do not know if this is normal or FT SS starts listening on 14002 when failover occurs.  When we stopped primary SS we still did not see TCP 14002 on FT SS.

[Wael AbdelWahab]

Hello

I have implemented that serval time with no issue, as you follow these About SpectroSERVER Fault Tolerance - CA Spectrum - 10.2 to 10.2.3 - CA Technologies Documentation  

please note that the fastest method that you make the SpectroServe to be HOT by adding the  following line to the .vnmrc file: 

secondary_polling=yes. 

 

anyway do you have firewall between the primary SS and the failover SS ? or between them and the OC ?

 

Regards

[Jon Velazquez]

No firewalls.  We were trying the warm standby.  On your FT SS do you see TCP 14002 listening?  It does look fairly simple so surprised it is not failing over right after checking our config.

[Wael AbdelWahab]

as Phani.Devulapalli said, you should have the OC server on the host security (can be done from the SS system applet (/../Spectrum/bin/SCP in linux)

the simple way add "+" in the host security to allow all SS to access this server and this include the OC server too

[Phani Devulapalli]

You just need to have the below ports open on both the primary and secondary SS and OneClick servers for the communication to work. 

 

Most importantly, make sure you add the OneClick server host name to the Host Security file on secondary,may be this could be the issue in your case 

 

 

[Jon Velazquez]

We do have the + in Host Security on the FT SS.  So since we are doing warm standby secondary_polling=no will the OC client be delayed in turning yellow until the FT SS is actually ready? 

[Phani Devulapalli]

If you are on warm  the secondary SS would be running already, just that it would not be polling until the switch happens and the FT switch over should not take more than a minute or so if the primary goes down....in your case what happens when the primary SS is down? Does the OneClick just turn red without switching to secondary ?

[Jon Velazquez]

The OC does connection status does not even turn red when primary is down (including processd down).  There is no red border either.  In connection status it only shows the primary SS server name for landscape and events both green.  So it's like the OC client doesn't even understand the primary went away nor that it can't connect to the secondary.  Further, the FT SS is not listening on TCP 14002 at all it does not have that port up which to me seems to be an application thing. 

[Silvio Okamoto]

Hi JonV,

 

The secondary SpectroSERVER must be listening on port TCP 14002 and it also must have established a connection to the OneClick web server machine on port TCP 14002.

If the SpectroSERVER is not listening on port TCP 14002, check if the local firewall is enabled. Have you tried restarting the SpectroSERVER application and the Processd?

 

Thanks,

Silvio

[Jon Velazquez]

Restarted the whole host after shutting down the FT SS properly.  It's still not using that tcp port 14002.  Wonder if I just need to deinstall Spectrum and all patches and resintall if there is not a way to debug further.

 

SS FT Host

netstat -a -o -n | grep 140
  TCP    0.0.0.0:14003          0.0.0.0:0              LISTENING       5168
  TCP    0.0.0.0:14004          0.0.0.0:0              LISTENING       2864
  TCP    0.0.0.0:14006          0.0.0.0:0              LISTENING       696
  TCP    10.1.9.60:14004        10.1.9.60:57237        ESTABLISHED     2864
  TCP    10.1.9.60:14004        10.1.9.141:63217       ESTABLISHED     2864
  TCP    10.1.9.60:14004        10.1.9.141:63218       ESTABLISHED     2864
  TCP    10.1.9.60:57237        10.1.9.60:14004        ESTABLISHED     2068
  TCP    10.1.9.60:57238        10.1.9.141:14004       ESTABLISHED     2068
  TCP    10.1.9.60:57239        10.1.9.141:14004       ESTABLISHED     2068
  TCP    [::]:14003             [::]:0                 LISTENING       5168
  TCP    [::]:14004             [::]:0                 LISTENING       2864
  TCP    [::]:14006             [::]:0                 LISTENING       696

[Silvio Okamoto]

Hi JonV,

 

Please check the $SPECROOT/SS/.vnmrc file if the following entry is not either truncated or missing:

orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -Dvbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadStackSize=1048576 -ORBpropStorage ../.corbarc

 

This specific entry will make the SpectroSERVER listening on TCP port 14002.

 

Note: The entry above is from CA Spectrum 10.2.2 in Linux.

 

Thanks,

Silvio

[Jon Velazquez]

Ahh.. thanks so much...secondary does not have a space between = and -Dvbroker.  Could that be it?

Primary  SS .vnmrc
orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -ORBpropStorage ../.corbarc
Secondary SS .vnmrc
orb_args=-Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002   -ORBpropStorage ../.corbarc

[Silvio Okamoto]

Hi JonV,

 

It looks like you have a truncated entry on both SSs machines.

Please copy and paste my example in your .vnmrc file and start the SpectroSERVER application.

 

orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -Dvbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadStackSize=1048576 -ORBpropStorage ../.corbarc

 

Thanks,

Silvio

[Jon Velazquez]

Interesting.  We are on a Windows 2012r2 VM for reference.

[Silvio Okamoto]

ah.. Windows, then should be like this:

 

orb_args=-Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002   -ORBpropStorage ../.corbarc

[Jon Velazquez]

That's what I have to begin with.  Interesting.

[Silvio Okamoto]

We need additional information to further investigate this problem.

As this script will gather sensitive data from your environment I would recommend to open a ticket at CA Technical Support.

 

Please provide the output of the "getSpectrumInfo.sh lite" script:

 

 

1. Open a bash shell (bash -login)

2. Run:
bin/support/getSpectrumInfo.sh lite

3. Gather the $SPECROOT/logs-<hostname>-yyyymmdd-hhmm.tar.gz

Note: You must run the getSpectrumInfo.sh script from the $SPECROOT directory. It does not run from the $SPECROOT/bin/ directory.

[Jon Velazquez]

CA Case Management

00977988

 

Case opened this morning.  Attached getSpectrunInfo.sh logs file generated from Specroot on FT SS.

[Jon Velazquez]

The breakthrough came when CA support happened to take my .vnmrc file and put it on his test server.  It would not bring up TCP port 14002 either.  That's when we knew it was related to something amiss in the .vnmrc file.  I had one mistake in a statement "ftasv_debug" instead of ftasv_debug= or ftasv_debug=true.  Other statement of interest is wait_activate=yes we changed that to no.  The other factor at play is whether we are getting stuck by some models never finishing activation.  I see one such model now as we still have 1 problematic model with voice.  It's the same model each time but main Spectroserver will finally activate it but the FT SS is still working on it since the restarting from the sync from yesterday at 7pm.  I think we are over the hump now.  Thanks Silvio, Phani, Wael, and any other for the troubleshooing ideas!