DX NetOps

Expand all | Collapse all

Steps OC client takes to connect to failover FT SS when primary is not available?

Jump to Best Answer
  • 1.  Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 04:09 PM

    Recently setup FT Spectroserver.  The DB synchronization is working.  However, the OneClick client does not turn yellow when I shutdown the primary SS. 

     

    Would like to understand the steps taken by OC client to connect to the secondary in order to troubleshoot? 

     

    Typically OC connects to TCP port 14002 on the Spectroserver.

     

    On the secondary FT Spectroserver do not see TCP 14002 listening.  Do not know if this is normal or FT SS starts listening on 14002 when failover occurs.  When we stopped primary SS we still did not see TCP 14002 on FT SS.



  • 2.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 04:46 PM

    Hello

    I have implemented that serval time with no issue, as you follow these About SpectroSERVER Fault Tolerance - CA Spectrum - 10.2 to 10.2.3 - CA Technologies Documentation  

    please note that the fastest method that you make the SpectroServe to be HOT by adding the  following line to the .vnmrc file: 

    secondary_polling=yes. 

     

    anyway do you have firewall between the primary SS and the failover SS ? or between them and the OC ?

     

    Regards



  • 3.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 05:11 PM

    No firewalls.  We were trying the warm standby.  On your FT SS do you see TCP 14002 listening?  It does look fairly simple so surprised it is not failing over right after checking our config.



  • 4.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 05:33 PM

    as Phani.Devulapalli said, you should have the OC server on the host security (can be done from the SS system applet (/../Spectrum/bin/SCP in linux)

    the simple way add "+" in the host security to allow all SS to access this server and this include the OC server too



  • 5.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 05:21 PM

    You just need to have the below ports open on both the primary and secondary SS and OneClick servers for the communication to work. 

     

    Most importantly, make sure you add the OneClick server host name to the Host Security file on secondary,may be this could be the issue in your case 

     

    SPEC--ss_OC_across_firewall (9.2)

     

    SPEC--Firewall_Prim_Sec_SS



  • 6.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 09:01 PM

    We do have the + in Host Security on the FT SS.  So since we are doing warm standby secondary_polling=no will the OC client be delayed in turning yellow until the FT SS is actually ready? 



  • 7.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-01-2018 10:24 PM

    If you are on warm  the secondary SS would be running already, just that it would not be polling until the switch happens and the FT switch over should not take more than a minute or so if the primary goes down....in your case what happens when the primary SS is down? Does the OneClick just turn red without switching to secondary ?



  • 8.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 10:43 AM

    The OC does connection status does not even turn red when primary is down (including processd down).  There is no red border either.  In connection status it only shows the primary SS server name for landscape and events both green.  So it's like the OC client doesn't even understand the primary went away nor that it can't connect to the secondary.  Further, the FT SS is not listening on TCP 14002 at all it does not have that port up which to me seems to be an application thing. 



  • 9.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 08:43 AM

    Hi JonV,

     

    The secondary SpectroSERVER must be listening on port TCP 14002 and it also must have established a connection to the OneClick web server machine on port TCP 14002.

    If the SpectroSERVER is not listening on port TCP 14002, check if the local firewall is enabled. Have you tried restarting the SpectroSERVER application and the Processd?

     

    Thanks,

    Silvio



  • 10.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 10:50 AM

    Restarted the whole host after shutting down the FT SS properly.  It's still not using that tcp port 14002.  Wonder if I just need to deinstall Spectrum and all patches and resintall if there is not a way to debug further.

     

    SS FT Host

    netstat -a -o -n | grep 140
      TCP    0.0.0.0:14003          0.0.0.0:0              LISTENING       5168
      TCP    0.0.0.0:14004          0.0.0.0:0              LISTENING       2864
      TCP    0.0.0.0:14006          0.0.0.0:0              LISTENING       696
      TCP    10.1.9.60:14004        10.1.9.60:57237        ESTABLISHED     2864
      TCP    10.1.9.60:14004        10.1.9.141:63217       ESTABLISHED     2864
      TCP    10.1.9.60:14004        10.1.9.141:63218       ESTABLISHED     2864
      TCP    10.1.9.60:57237        10.1.9.60:14004        ESTABLISHED     2068
      TCP    10.1.9.60:57238        10.1.9.141:14004       ESTABLISHED     2068
      TCP    10.1.9.60:57239        10.1.9.141:14004       ESTABLISHED     2068
      TCP    [::]:14003             [::]:0                 LISTENING       5168
      TCP    [::]:14004             [::]:0                 LISTENING       2864
      TCP    [::]:14006             [::]:0                 LISTENING       696



  • 11.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 12:02 PM

    Hi JonV,

     

    Please check the $SPECROOT/SS/.vnmrc file if the following entry is not either truncated or missing:

    orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -Dvbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadStackSize=1048576 -ORBpropStorage ../.corbarc

     

    This specific entry will make the SpectroSERVER listening on TCP port 14002.

     

    Note: The entry above is from CA Spectrum 10.2.2 in Linux.

     

    Thanks,

    Silvio



  • 12.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 01:28 PM

    Ahh.. thanks so much...secondary does not have a space between = and -Dvbroker.  Could that be it?

    Primary  SS .vnmrc
    orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -ORBpropStorage ../.corbarc
    Secondary SS .vnmrc
    orb_args=-Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002   -ORBpropStorage ../.corbarc



  • 13.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 01:52 PM

    Hi JonV,

     

    It looks like you have a truncated entry on both SSs machines.

    Please copy and paste my example in your .vnmrc file and start the SpectroSERVER application.

     

    orb_args= -Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002 -Dvbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadStackSize=1048576 -ORBpropStorage ../.corbarc

     

    Thanks,

    Silvio



  • 14.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 02:12 PM

    Interesting.  We are on a Windows 2012r2 VM for reference.



  • 15.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 02:39 PM

    ah.. Windows, then should be like this:

     

    orb_args=-Dvbroker.se.iiop_tp.scm.iiop_tp.listener.port=14002   -ORBpropStorage ../.corbarc



  • 16.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 02:45 PM

    That's what I have to begin with.  Interesting.



  • 17.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-02-2018 07:14 PM

    We need additional information to further investigate this problem.

    As this script will gather sensitive data from your environment I would recommend to open a ticket at CA Technical Support.

     

    Please provide the output of the "getSpectrumInfo.sh lite" script:

     

     

    1. Open a bash shell (bash -login)

    2. Run:
    bin/support/getSpectrumInfo.sh lite

    3. Gather the $SPECROOT/logs-<hostname>-yyyymmdd-hhmm.tar.gz


    Note: You must run the getSpectrumInfo.sh script from the $SPECROOT directory. It does not run from the $SPECROOT/bin/ directory.



  • 18.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?

    Posted 03-05-2018 09:14 AM

    CA Case Management

    00977988

     

    Case opened this morning.  Attached getSpectrunInfo.sh logs file generated from Specroot on FT SS.



  • 19.  Re: Steps OC client takes to connect to failover FT SS when primary is not available?
    Best Answer

    Posted 03-06-2018 09:40 AM

    The breakthrough came when CA support happened to take my .vnmrc file and put it on his test server.  It would not bring up TCP port 14002 either.  That's when we knew it was related to something amiss in the .vnmrc file.  I had one mistake in a statement "ftasv_debug" instead of ftasv_debug= or ftasv_debug=true.  Other statement of interest is wait_activate=yes we changed that to no.  The other factor at play is whether we are getting stuck by some models never finishing activation.  I see one such model now as we still have 1 problematic model with voice.  It's the same model each time but main Spectroserver will finally activate it but the FT SS is still working on it since the restarting from the sync from yesterday at 7pm.  I think we are over the hump now.  Thanks Silvio, Phani, Wael, and any other for the troubleshooing ideas!