Clarity

 View Only
  • 1.  Potential NSA/Tower issue in 16.1.0

    Posted Nov 28, 2022 10:13 AM
    Edited by Paul Schofield Nov 28, 2022 11:29 AM
    Hello everyone,

    We've upgraded 2 environments (DEV and TEST) to 16.1.0 and we've noticed that the admin tower command no longer works as expected, and the servers and services display strangely in the NSA.

    I'll start with the NSA. When we login we sometimes see all servers in the cluster, sometimes none, and sometimes just a couple of them. The same is true of the services. On the individual servers the services show as running (service status all gives the expected results), and the load-balancer is showing all app services as healthy. Clarity seems to be working OK though.

    If we execute the admin tower command set set trace on, we don't see anything on any of the app servers. Similarly the list clients option just shows "none", even after explicitly setting autodiscovery on

    These used to work fine before we upgraded, although I'll admit that the NSA behaviour of now you see it / now you don't is a feature I've seen many times before. It does seem to have worsened though.

    Another thing we've seen is that the cmn_jgroups_ping table isn't being populated (we can't use multicast so JDBCPing is in use).

    I've looked in the release note but didn't see anything that might explain the above. In the beacon-system.log on all servers the "Validated by server" messages show for all 4 servers in the cluster so I think things are sort of working.

    Has anyone else seen this behaviour?

    In closing, everything is fine in production (which is still at 16.0.3).

    As always any and all suggestions will be gratefully received.

    Paul


  • 2.  RE: Potential NSA/Tower issue in 16.1.0

    Posted Nov 29, 2022 01:58 AM
    I'm, updating this thread as things have started to work. Restarting each server in sequence didn't fix things and neither did restarting the NSA. What did do the trick is removing and redeploying the NSA in both systems.

    The admin tower command now displays the trace as expected and the cmn_jgroups_ping table is populated. The servers and services are still missing sometimes but that's a cosmetic feature we can live with.


  • 3.  RE: Potential NSA/Tower issue in 16.1.0

    Broadcom Employee
    Posted Nov 29, 2022 04:02 AM
    Hi Paul 

    Couple of questions 

    • Was the clarity folder copied from another enviornment 
    • If it was copied the environment from where it was copied, was it running on auto discovery or using the host file. 


    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 4.  RE: Potential NSA/Tower issue in 16.1.0

    Posted Nov 29, 2022 04:19 AM
    Hi Suman,

    The problem happened after an in-situ upgrade where auto-discovery was set on before we started and that setting was carried through. In the way AWS has been setup here the IP addresses of the servers change periodically so auto discovery is the best way. 

    As I noted above, after a lot of restarting of servers and services, redeploying the NSA did the trick. I have no idea why.


  • 5.  RE: Potential NSA/Tower issue in 16.1.0

    Broadcom Employee
    Posted Nov 29, 2022 08:10 AM
    when you redeploy the service it will take the files from template and deploy so your host.xml should have been repopulated which fixed this. I understand your IP changes in AWS, however the hostname should be constant right, in the bind address is the host name populated or the IP address

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 6.  RE: Potential NSA/Tower issue in 16.1.0

    Posted Nov 29, 2022 08:32 AM
    Hi again,

    Sadly no, the host name changes too, although during the upgrade it didn't, but periodically the servers are redeployed and that's when the addresses change.

    For example, if the first app server has an IP of 10.168.1.1 its hostname is IP-10.168.1.1. The next time (after redeployment) the same server might have an IP of 10.168.1.2 and its hostname changes to IP-10.168.1.2. This is how AWS is working at this client. For now, I think I have the steps to fix things after each upgrade and/or redeployment.

    Thanks for getting back,

    Paul


  • 7.  RE: Potential NSA/Tower issue in 16.1.0

    Broadcom Employee
    Posted Nov 29, 2022 08:52 AM
    Sadly I dont think clarity support these configurations yet, the other challenge is if the hostname changes specially for BG when the app and BG are running the processes can get stuck. Just wanted to let you know that this can cause additional issues too.

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------