Clarity

 View Only
  • 1.  Beacons can't communicate after NSA upgrade

    Posted Sep 26, 2016 06:52 PM

    I'm upgrading from 13.2.0.8 -> 13.2.0.9 -> 14.3.0 -> 14.3.0.8.  In both the DEV and TEST clusters, the beacon on the NSA cannot communicate with any of the beacons on the non-NSA boxes after the NSA is upgraded.  I ran a health check before each upgrade and there were no issues.  The multicast address hasn't changed, I've tried resetting the admin password, to no avail.  IGMP snooping has not been disabled.  IP addresses have not changed - all cluster members are in the same subnet.  In the TEST upgrade I checked communication at the 14.3.0 stage, and was unsuccessful.  Tech Support has not been able to find a solution.  One test suggested was for Jgroups:

     

    java org.jgroups.tests.McastSenderTest1_4 -mcast_addr <multicast_address> -port <mcast port> -use_all_interfaces

     

    but this doesn't work on the upgraded machine (but does on the non-upgraded boxes).  I'm running RHEL 6.8 and JDK1.8.0_101 for 14.3.

     

    For DEV and TEST I ended up replacing the $NIKU_HOME directory on the non-NSA boxes with the one from the NSA, and editing properties.xml.  This resolves the problem. 

     

    The PROD cluster upgrade is scheduled for two weeks, so I need a resolution.  Is copying  $NIKU_HOME a supported approach?

     

    Thanks



  • 2.  Re: Beacons can't communicate after NSA upgrade

    Broadcom Employee
    Posted Sep 26, 2016 07:06 PM

    Hi Chris,

     

    I hope you upgraded the server which hosts the NSA and then you distributed to other app servers in cluster? Health check doesn't show if your multicasting is working fine or not. 

     

    Regards

    Suman Pramanik 



  • 3.  Re: Beacons can't communicate after NSA upgrade

    Posted Sep 26, 2016 07:51 PM
      |   view attached

    Hi,

     

    I did the upgrade on the NSA box and then checked admin tower on both the NSA and non-NSA boxes.  The NSA couldn’t see the non-NSA boxes and vice versa.  Since the beacons weren’t communicating, I couldn’t do the distribution

     

    Thanks

    chris



  • 4.  Re: Beacons can't communicate after NSA upgrade

    Broadcom Employee
    Posted Sep 26, 2016 07:58 PM

    Hi Chris,

     

    You can proceed to copy the upgraded folder to different nodes in the cluster and it the same which distribution does, however after that if you run the multicast test and all tests doesn't go through then there is some multicast issues where network team needs to go deep into investigation.

     

    Regards

    Suman Pramanik 



  • 5.  Re: Beacons can't communicate after NSA upgrade

    Posted Sep 28, 2016 11:52 PM
      |   view attached

    Hi Suman,

     

    After copying $NIKU_HOME and editing properties.xml on each non-NSA cluster node, each node can see all the others in “admin tower”.  Doesn’t this verify that multicasting is working correctly?

     

    Thanks for your help



  • 6.  Re: Beacons can't communicate after NSA upgrade

     
    Posted Oct 17, 2016 04:38 PM

    Hi chris.parker - We're you ever able to figure this one out? Thanks! Chris



  • 7.  Re: Beacons can't communicate after NSA upgrade

    Posted Oct 17, 2016 05:25 PM
      |   view attached

    Hi,

     

    No – I got the same behaviour in each of three environments.  But after copying $NIKU_HOME from the NSA box to the others and editing properties.xml, everything works correctly.

     

    Thanks

    Chris



  • 8.  RE: Re: Beacons can't communicate after NSA upgrade

    Posted Mar 12, 2021 03:17 AM
    Hello all,

    I have got the same symptoms on beacon after teledistribution.
    Context : Red Hat 7.x, Oracle, Upgrade from 15.7.0 to 15.9.0, cluster of 3 servers, multicast on IP 239.0.0.1

    • Server 1 : CSA running, upgrade is done. beacon start correctly after upgrade
    • Server 2 : Code teledistributed. Error beacon : may have failed to start.
    • Server 3 : Code teledistributed. Error beacon : may have failed to start.
    Note that nothing changed on the network level : multicast on same IP / ports.

    Logs beacon-system on servers 2 and 3 :

    Launching a JVM...
    WrapperManager: Initializing...
    Failed to communicate: 239.0.0.1 : 9090
    on_exit trigger matched.  Restarting the JVM.  (Exit code: 1)
    Launching a JVM...
    WrapperManager: Initializing...
    Failed to communicate: 239.0.0.1 : 9090
    on_exit trigger matched.  Restarting the JVM.  (Exit code: 1)
    Launching a JVM...
    WrapperManager: Initializing...
    Failed to communicate: 239.0.0.1 : 9090
    on_exit trigger matched.  Restarting the JVM.  (Exit code: 1)
    Launching a JVM...
    WrapperManager: Initializing...
    Failed to communicate: 239.0.0.1 : 9090
    on_exit trigger matched.  Restarting the JVM.  (Exit code: 1)
    Launching a JVM...
    WrapperManager: Initializing...
    Failed to communicate: 239.0.0.1 : 9090
    on_exit trigger matched.  Restarting the JVM.  (Exit code: 1)
    There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
       There may be a configuration problem: please check the logs.


    I will try the following actions :
    • Restart the servers
    • Copy-paste the folder $NIKU_HOME as mentioned above
    Regards,
    David

    ------------------------------
    Consultant
    CoPrime
    ------------------------------



  • 9.  RE: Re: Beacons can't communicate after NSA upgrade

    Broadcom Employee
    Posted Mar 12, 2021 04:04 AM

    Hi David

    Yes we have seen this issue and copy pasting to cluster node is the best way to overcome this. I will try to investigate this and it certainly looks like a code bug where distribution is not working properly from CSA.

    Regards
    Suman Pramanik 



    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 10.  RE: Re: Beacons can't communicate after NSA upgrade

    Posted Mar 14, 2021 04:11 PM
    For information, a server restart could solve the problem.

    Regards,
    David

    Another symptom with the utility admintower :
    • The server 1 could see servers 2 and 3
    • Servers 2 and 3 could only see themselves


    ------------------------------
    Consultant
    CoPrime
    ------------------------------