Recently setup FT Spectroserver. The DB synchronization is working. However, the OneClick client does not turn yellow when I shutdown the primary SS.
Would like to understand the steps taken by OC client to connect to the secondary in order to troubleshoot?
Typically OC connects to TCP port 14002 on the Spectroserver.
On the secondary FT Spectroserver do not see TCP 14002 listening. Do not know if this is normal or FT SS starts listening on 14002 when failover occurs. When we stopped primary SS we still did not see TCP 14002 on FT SS.
I have implemented that serval time with no issue, as you follow these About SpectroSERVER Fault Tolerance - CA Spectrum - 10.2 to 10.2.3 - CA Technologies Documentation
please note that the fastest method that you make the SpectroServe to be HOT by adding the following line to the .vnmrc file:
anyway do you have firewall between the primary SS and the failover SS ? or between them and the OC ?
No firewalls. We were trying the warm standby. On your FT SS do you see TCP 14002 listening? It does look fairly simple so surprised it is not failing over right after checking our config.
as Phani.Devulapalli said, you should have the OC server on the host security (can be done from the SS system applet (/../Spectrum/bin/SCP in linux)
the simple way add "+" in the host security to allow all SS to access this server and this include the OC server too
You just need to have the below ports open on both the primary and secondary SS and OneClick servers for the communication to work.
Most importantly, make sure you add the OneClick server host name to the Host Security file on secondary,may be this could be the issue in your case
We do have the + in Host Security on the FT SS. So since we are doing warm standby secondary_polling=no will the OC client be delayed in turning yellow until the FT SS is actually ready?
If you are on warm the secondary SS would be running already, just that it would not be polling until the switch happens and the FT switch over should not take more than a minute or so if the primary goes down....in your case what happens when the primary SS is down? Does the OneClick just turn red without switching to secondary ?
The OC does connection status does not even turn red when primary is down (including processd down). There is no red border either. In connection status it only shows the primary SS server name for landscape and events both green. So it's like the OC client doesn't even understand the primary went away nor that it can't connect to the secondary. Further, the FT SS is not listening on TCP 14002 at all it does not have that port up which to me seems to be an application thing.
The secondary SpectroSERVER must be listening on port TCP 14002 and it also must have established a connection to the OneClick web server machine on port TCP 14002.
If the SpectroSERVER is not listening on port TCP 14002, check if the local firewall is enabled. Have you tried restarting the SpectroSERVER application and the Processd?
Restarted the whole host after shutting down the FT SS properly. It's still not using that tcp port 14002. Wonder if I just need to deinstall Spectrum and all patches and resintall if there is not a way to debug further.
SS FT Host
netstat -a -o -n | grep 140 TCP 0.0.0.0:14003 0.0.0.0:0 LISTENING 5168 TCP 0.0.0.0:14004 0.0.0.0:0 LISTENING 2864 TCP 0.0.0.0:14006 0.0.0.0:0 LISTENING 696 TCP 10.1.9.60:14004 10.1.9.60:57237 ESTABLISHED 2864 TCP 10.1.9.60:14004 10.1.9.141:63217 ESTABLISHED 2864 TCP 10.1.9.60:14004 10.1.9.141:63218 ESTABLISHED 2864 TCP 10.1.9.60:57237 10.1.9.60:14004 ESTABLISHED 2068 TCP 10.1.9.60:57238 10.1.9.141:14004 ESTABLISHED 2068 TCP 10.1.9.60:57239 10.1.9.141:14004 ESTABLISHED 2068 TCP [::]:14003 [::]:0 LISTENING 5168 TCP [::]:14004 [::]:0 LISTENING 2864 TCP [::]:14006 [::]:0 LISTENING 696
Please check the $SPECROOT/SS/.vnmrc file if the following entry is not either truncated or missing:
orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -Dvbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadStackSize=1048576 -ORBpropStorage ../.corbarc
This specific entry will make the SpectroSERVER listening on TCP port 14002.
Note: The entry above is from CA Spectrum 10.2.2 in Linux.
Ahh.. thanks so much...secondary does not have a space between = and -Dvbroker. Could that be it?
Primary SS .vnmrcorb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -ORBpropStorage ../.corbarcSecondary SS .vnmrcorb_args=-Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -ORBpropStorage ../.corbarc
It looks like you have a truncated entry on both SSs machines.
Please copy and paste my example in your .vnmrc file and start the SpectroSERVER application.
Interesting. We are on a Windows 2012r2 VM for reference.
ah.. Windows, then should be like this:
orb_args=-Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -ORBpropStorage ../.corbarc
That's what I have to begin with. Interesting.
We need additional information to further investigate this problem.
As this script will gather sensitive data from your environment I would recommend to open a ticket at CA Technical Support.
Please provide the output of the "getSpectrumInfo.sh lite" script:
1. Open a bash shell (bash -login)
2. Run: bin/support/getSpectrumInfo.sh lite
3. Gather the $SPECROOT/logs-<hostname>-yyyymmdd-hhmm.tar.gz
Note: You must run the getSpectrumInfo.sh script from the $SPECROOT directory. It does not run from the $SPECROOT/bin/ directory.
CA Case Management
Case opened this morning. Attached getSpectrunInfo.sh logs file generated from Specroot on FT SS.
The breakthrough came when CA support happened to take my .vnmrc file and put it on his test server. It would not bring up TCP port 14002 either. That's when we knew it was related to something amiss in the .vnmrc file. I had one mistake in a statement "ftasv_debug" instead of ftasv_debug= or ftasv_debug=true. Other statement of interest is wait_activate=yes we changed that to no. The other factor at play is whether we are getting stuck by some models never finishing activation. I see one such model now as we still have 1 problematic model with voice. It's the same model each time but main Spectroserver will finally activate it but the FT SS is still working on it since the restarting from the sync from yesterday at 7pm. I think we are over the hump now. Thanks Silvio, Phani, Wael, and any other for the troubleshooing ideas!