DX NetOps

 View Only

Upgrading CA Spectrum FT-paired DSS - OC-server hang issue

  • 1.  Upgrading CA Spectrum FT-paired DSS - OC-server hang issue

    Broadcom Employee
    Posted Jun 30, 2017 10:15 AM

    Dear all, 


    for bigger CA Spectrum monitoring the default implementation is a DSS/Distributed SpectroSERVER setup - and here most likely running the SpectroSERVERs in a FT/Fault-Tolerant setup (paired to primary and secondary). 

    In addition it is recommended to use multiple OneClick-Server. 


    Let's try to cover best practice upgrade details to this discussion.


    For many setup/runtime installs, doing the migration for the Primary_SpectroSERVER set first - while having the secondary_SpectroSERVER set running. As the CA Spectrum sets/installs should be at matching releases, this would require to "split" the SpectroSERVER sets. Potentially this could be done by firewall services - or i.e. by IPtables configuration on Linux hosts.


    So - per administrative actions you may achieve to have sets of  Spectrum running:

    • All primary_SpectroSERVERs and additional OC-server
    • All secondary_SpectroSERVERs and additional OC-server


    Most likely per default configuration the OC-server setup may cover the MLS and a BackupLocationServer configuration. So once the two sets are "split" this may cause the OC-server from the "primary_set" not being able to contact the BackupLS from the "secondary_set" and versa. 


    Now - depending on the firewall or IP-tables behaviour - ie. to drop the session request, this can cause the OC-server during service startup to "wait" for TCP-session establishment. Due to this it may appear the OC-server is in a hang condition - and will not complete for service startup. So the OC-service will not become available. infact this means, the configured MLS and BackupLS have to be available at time of startup for the OC-service (here better: TCP-session establishment should not run into a "SYN_SENT" wait condition). This is also true in case the MLS is correctly configured and the BackupLS is not reachable - or versa.


    So it is important before "splitting" the two sets to reconfigure the OC-servers to have references to MLS and BackupLS which are in the same set/connectable. Having the primary_OC refering to primary_SpectroSERVERs only - and the secondary_set OC-server to refer to the secondary_set SpectroSERVERs. Once done - you can split then the both runtimes/sets. 


    hth / Regards, Joerg