Clarity

 View Only
Expand all | Collapse all

Clarity Cluster - JDBC Ping CSA dont see all servers

  • 1.  Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 02, 2021 11:23 AM
    Edited by Naman Patel Mar 02, 2021 11:28 AM
    I am setting up a new clarity server cluster. I have currently 2 servers, one with CSA. I am not able to see 2 servers on the CSA servers list, I can see only the node on which CSA is running. I have tried autodiscovery as well as clarity hosts.xml file, without success.

    Setup Done:

    Version: Clarity 15.8.1
    OS: CentOS

    Sever 1:
    Added NSA, App, App2, BG, Beacon
    Enabled JDBC Ping using  (CSA: Configure JDBC Ping As An Alternative to Multicast (broadcom.com))
    OS Hosts file has the entry for 127.0.0.1 localhost with hostname
    Multicast Setting:
      <nsa bindAddress="10.105.165.75" multicastAddress="230.0.10.6" multicastPort="9090" clientPort="9091" serviceName="Clarity PPM Beacon" serviceUser="" servicePassword="" autoStartService="true" autoDiscovery="true" useJDBCPing="true"/>


    Server 2:
    Added App, App2, Bg, Beacon
    OS Hosts file has the entry for 127.0.0.1 localhost with hostname
    Enabled JDBC Ping

      <nsa bindAddress="10.105.165.78" multicastAddress="230.0.10.6" multicastPort="9090" clientPort="9091" serviceName="Clarity PPM Beacon" serviceUser="" servicePassword="" autoStartService="true" autoDiscovery="true" useJDBCPing="true"/>​​


    I can do admin tower and trace works in both directions.

    Server 1 Traces:



    Server 2 Traces:

    JDBC Ping database entries:



    I have referred following documents:

    Clarity: Fact Sheet: Clarity Beacon with Multicasting - include FAQs. (broadcom.com)
    CA Clarity PPM on VMWare: avoid multicast issues when using vSwitch (broadcom.com)
    Testing for Active Nodes in the Cluster - Clarity (broadcom.com)


    Am I missing anything?


  • 2.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 02, 2021 03:28 PM
    I believe you have done everything correct, does the other server has the same CSA password as that needs to be same. We might need to check the host file for both the servers to if that has correct IP address

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 3.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 02, 2021 03:32 PM
    Yes CSA Password is the same on all servers. I did make sure the host file has the correct IP addresses.


  • 4.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 02, 2021 03:37 PM
    Hi Naman,
    You have to include both servers in the bind address, separated by comma with no spaces. Try adding them both, and see if that helps?

    ------------------------------
    Nika Hadzhikidi
    Sr Principal Support Engineer
    Broadcom
    ------------------------------



  • 5.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 02, 2021 03:49 PM
    I tried adding both IPs in bind addresses, but the beacon starts to fail after adding comma-separated IPs.

    NSA log error when I add comma-separated IPs.

    Caused by: java.lang.Exception: Property assignment of bind_addr in TCP with original property value 10.105.165.78,10.105.165.75 and converted to null could not be assigned
    	at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1154)
    	at org.jgroups.stack.Configurator.createLayer(Configurator.java:445)
    	at org.jgroups.stack.Configurator.createProtocols(Configurator.java:398)
    	at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:90)
    	at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:57)
    	at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:476)
    	at org.jgroups.JChannel.init(JChannel.java:852)
    	at org.jgroups.JChannel.<init>(JChannel.java:159)
    	at org.jgroups.JChannel.<init>(JChannel.java:120)
    	at com.niku.union.utility.JgroupMessengerUtil.createChannel(JgroupMessengerUtil.java:218)
    	... 22 more​



  • 6.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 02, 2021 04:27 PM
    One of the IPs may not be correct/valid as Suman said above. I'd try the Jgroups test as per here:
    https://knowledge.broadcom.com/external/article?articleId=201202 
    see if that goes through.

    ------------------------------
    Nika Hadzhikidi
    Sr Principal Support Engineer
    Broadcom
    ------------------------------



  • 7.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 02, 2021 05:20 PM
    I have double-checked, triple-checked, I don't see the wrong IPs. I SSH in using these IPs you can also see these IPs in the screenshots.


  • 8.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 03, 2021 04:51 AM
    Hi Naman

    I believe you are using this in Amazon and I guessed by looking at your IP address. I also see you have app and app 2. For the test can you stop and remove the app  2 and remove the same tag from properties.xml and see.

    From the stack I think some where its reading the local host which is causing this 
    Caused by: java.lang.Exception: Property assignment of bind_addr in TCP with original property value 10.105.165.78,10.105.165.75 and converted to null could not be assigned
    at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1154)
    at org.jgroups.stack.Configurator.createLayer(Configurator.java:445)
    at org.jgroups.stack.Configurator.createProtocols(Configurator.java:398)
    at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:90)
    at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:57)
    at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:476)
    at org.jgroups.JChannel.init(JChannel.java:852)
    at org.jgroups.JChannel.<init>(JChannel.java:159)
    at org.jgroups.JChannel.<init>(JChannel.java:120)
    at com.niku.union.utility.JgroupMessengerUtil.createChannel(JgroupMessengerUtil.java:218)
    ... 22 more​

    Similar error stack while using the Jgroup cache messager is there on other threads  https://bugzilla.redhat.com/show_bug.cgi?id=900707

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 9.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 03, 2021 09:31 AM
    You are correct I am on AWS (that's why JDBC Ping). I removed app2 from both servers and still the same issue.

    Trace is still working in both directions, it strongly suggests there is no IP-related issue.





  • 10.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 03, 2021 10:01 AM
      |   view attached
    Uploading document for better image quality.

    Attachment(s)

    docx
    multicast test.docx   172 KB 1 version


  • 11.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 03, 2021 09:50 AM
    Edited by Naman Patel Mar 03, 2021 09:51 AM
    I also tried multicasting test steps in the documents, there is an issue with the class name in the document (Multicast tests (full recommendations and best practices in getting Multicast to work with Clarity) (broadcom.com)) after digging into Jar I found the correct class name doesn't include 1_4. But anyway, tests were successful in both directions, and I still can't see both servers on NSA.






  • 12.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 03, 2021 10:06 AM
    Hi Naman

    I understand and I am not saying its IP but there is some issue we need to debug, I believe you have a case raised too

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 13.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 03, 2021 10:11 AM
    Yes, I have raised the case and getting support. I got unique ideas from you and Nika. So trying everything and documenting as much as I can on the community site.


  • 14.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers
    Best Answer

    Posted Mar 03, 2021 04:35 PM
    This is very comprehensive documentation on setting up and testing multicast. It's not clarity but still applies.

    Multicast Configuration (gigaspaces.com)

    It has helped me solve the problem. I changed my host file 

    Server 1
    From:
    127.0.0.1 localhost myhost1

    To
    127.0.0.1 localhost
    10.105.165.78 myhost1

    server 2
    From:
    127.0.0.1 localhost myhost2

    To
    127.0.0.1 localhost
    10.105.165.75 myhost2

    The second row in the host file is the eth0 IP address on the local machine with its hostname.

    Both should work, and it seems that both settings work at a low level when I am testing network traffic. But for some reason, in my case, NSA doesn't pick up other servers if they don't have their own IP entry with their own hostname.



  • 15.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 04, 2021 02:40 AM
    Glad to hear, if you seen my earlier reply my suggestion was to check etc host and the error from stack showed that it was picking up local host.

    Below was the stack confirmed that. 

    From the stack I think some where its reading the local host which is causing this
    Caused by: java.lang.Exception: Property assignment of bind_addr in TCP with original property value 10.105.165.78,10.105.165.75 and converted to null could not be assigned
    at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1154)
    at org.jgroups.stack.Configurator.createLayer(Configurator.java:445)


    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 16.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 04, 2021 07:52 AM
    @Suman Pramanik, agree your suggestion made me dig more into hosts. Ultimately following product documentation made it abundantly clear. I hope we can add this to the Clarity documentation. 

     ​


  • 17.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 04, 2021 07:59 AM
    Thank you for the feedback Naman and we can certainly incorporate that.@Akshay Sainath Iyer lets discuss and get this done. ​

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 18.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 09, 2021 01:00 PM
    Hi Naman,

    Besides the tracing aspect, please note some other things to double check:

    1 - Don't use the default multicast address, there could be other applications using that address

    2 - Make sure changes are propagated to the Clarity Database
    If your using and uploading the properties.xml in the DB, validate the files & consistency on both servers where you can force the sync with the following command:
    – admin general upload-config –info

    3 - In a clustered system, sometimes the bind addresses should be entered on each server as comma separated values with no space between IPs
    e.g. 199.11.51.1,199.12.51.2

    4 - All servers in the Clarity cluster MUST be able to send/receive multicast traffic.
    4.1 Use a common multicast port for each server 30.0.0.1),
    4.2 NSA password on all servers must be same
    4.3 Common RHEL problems, check with the network people that MULTICAST flag on "ifconfig –a" is setup
    4.4 Ensure "Distributed" option is set within the NSA

    Cdt,
    Philip


  • 19.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 09, 2021 01:13 PM
    @Philip PROSS agree with all the points, Except Point 3, I tried doing that several times (in call with Broadcom Support) services won't start with comma-separated values. I think comma-separated values need to be used when you have multiple NIC on the same server ​we can put IP addresses of both NIC interfaces as bind addresses.

    Also, I would add point 5.

    Make sure hosts file has the following entries.

    127.0.0.1 localhost
    <SERVER_IP> <SERVER_HOSTNAME>

    Replace server_ip and server_hostname using values found using OS level commands like hostname and ipconfig(Win) or ifconfig(Unix).


  • 20.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 10, 2021 04:03 AM
    One correction we dont need comma separated IP address in bind address that was needed when we supported WebSphere but with Tomcat being a managed service with clarity we dont need that.

    ------------------------------
    Thanks & Regards
    Suman Pramanik
    Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
    Broadcom
    ------------------------------



  • 21.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Broadcom Employee
    Posted Mar 10, 2021 10:25 AM
    Philip/Suman - Just to clarify:
    Comma separated bind addresses are used for multicast with Tomcat with multiple NICs.
    For JDBC Ping this is not required as the change in hosts with the IPs should work.

    ------------------------------
    Nika Hadzhikidi
    Sr Principal Support Engineer
    Broadcom
    ------------------------------



  • 22.  RE: Clarity Cluster - JDBC Ping CSA dont see all servers

    Posted Mar 10, 2021 10:30 AM
    Yup agree with Nika, that means multiple NIC from the same host not from different hosts. We should not combine IP addresses of 2 different cluster nodes in the bind-address with comma.