JDBCPing Configuration Issue

[Devendra Jakhmola]

We have recently moved from multicast to JDBC ping. We can see the list of services in table - CMN_JGROUPS_PING.

However, we are seeing that the processes are now running slower and many event based processes don't even trigger. We have done necessary settings changes for JDBC Ping:

1. Setting auto discovery to TRUE.
2. Setting same admin password on all servers.
3. Keeping Bind Address on each of the servers' properties.xml pointing to individual server's IP address.
4. Added attribute "useJDBCPing="true" on each server's properties.xml.

Could you please guide us on how to ensure that enabling JDBC ping is working as expected?

Also, when we tried to go back to Multicasting by removing JDBCping tab on all the servers and changing the "Bind Address" to have all server's IPs, all the servers started showing Offline on CSA. We performed JGroups testing on multicast address and beacon port, and all servers do commmunicating to each other.

------------------------------
Regards,
Devendra Jakhmola
------------------------------

[Suman Pramanik]

Hi Devendra 
Moving to JDBC ping doesn't cause performance issues and to prove that out you can switch back to multicast and see. JDBC does the cache replication from the database where as Multicast use network protocols to do the cache sync. 

To eliminate all these i would start looking few things 

1. If you can share the properties.xml and I can quickly review 
2. Since the process is running slow, can you provide few additional information 

• How many process instance you have in the system?
• What is the size of BPM_ERRORS table? 
• Did you check the NMS_MESSAGES & NMS_MESSAGE_DELIVERY tables to track the time of the process event

3. I believe you are using the database as oracle, can you share the init.ora parameters and also you should monitor any tables if they are fragmented you should work with your DBA to de fragment them. 

Attaching a document to troubleshoot if you JDBC Ping is working on not. Hope this helps

------------------------------
Thanks & Regards
Suman Pramanik
Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
Broadcom
------------------------------

Attachment(s)

JDBCPing Tests v1.docx   149 KB 1 version

[Devendra Jakhmola]

Hi Suman,

I'll peform the JDBCPing testing as per the document and will share my results.

1. For your quick review, i have attached properties.xml for 2 of our 4 servers.
2. With our analysis – we have found that the bg events are triggered but not received i.e. data is being inserted in to NMS_MESSAGES table the moment user perform any action on application but not reflected/takes too much time into NMS_MESSAGE_DELIVERY table. We have observed that the last 20 received event list in the process engine is always blank where in we see the values are in last 20 sent event list . This is when there were not more than 40 running instances of processes.
3. I have attached the fragmentation data on all our tables in Oracle DB.

When we tried to switch back to Multicast, all the non-NSA servers in cluster started appearing offline (disconnected). We have tried to remove, add, deploy start beacon on all servers already.

Attachment(s)

Prod_NIKU_FRAGMENTATION.txt   122 KB 1 version

[Suman Pramanik]

Thanks Devendra

The properties.xml looks ok however your tables are heavily fragmented and you definitely need to ask your DBA to export the tables and import table to regain the wasted space and perform a reindexing. Also we might need to focus on database optimization 

Also how many process instance you have in the system (all status) 

Regards
Suman Pramanik

------------------------------
Thanks & Regards
Suman Pramanik
Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
Broadcom
------------------------------

[Devendra Jakhmola]

Hi Suman,
Thanks for your guidance so far. 

We have reached out to DBAs with the recommendation for database de-fragmentation and they are reviewing this recommendation. Also, we have performed the JDBCping testing as per your document and trace on' shows servers are communicating with each others.

Regarding JDBCPing,  we noticed in the process engine history page that every time we restart BG services, the bg-hostname is being generated each time.Example, the name of BG engines is now showing as and everytime a new ID is generated...

bg-f6308d9fe2ae8ca0:612ae6a7:1721d09a437:-7ffe
bg-4f7c22c6bf9f5783:-5aec1590:1721d1a9ef6:-7ffe

Even after we restored clarity home folder and cleaned up CMN_JGROUPS_PING table, this behavior didn't change. We are suspecting that there exists a configuration outside clarity home folder which could be causing this. As part of JDBC ping configuration, we got hostname added in /etc/hosts file as shown below. Do you think the 'hosts' file has wrong configuration -

127.0.0.1 vdurppmas001.wilm.***.com localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

***.**.**.69 vdurppmas001.wilm.***.com vdurppmas001

Regards,
Devendra

[Stefano Lazzarotto]

Hi Devendra.
We had a similar issue (Process engine list growing) and the cause was a network configuration error.
Nslookup hostname it wasn't resolved.
Also, due to this problem, we had million of entries in cmn_sessions table.

Hope it helps.

Stefano

[Devendra Jakhmola]

Hi Suman,

It was found out that $HOSTNAME was entered as actual hostname of the server in /etc/hosts. So the issue got fixed after we replaced it with "$HOSTNAME". So now JDBCping is configured correctly.

However, the overall issue with slowness of system is still not resolved. DBA has following comments -

"Fragmentation in a database is a normal and expected occurrence.  In a locally instead of dictionary managed table space  (ours are locally managed) rebuilds due to fragmentation are rare.  There can be scenarios where a rebuild is warranted but only in response to identified and degraded performance resulting from the fragmentation.  The results of the analysis of the trace file for the session in the database that performed the job did not identify any degradation in performance from fragmentation.  Additionally, defragmenting unnecessarily can actually cause performance degradation for a period of time after the defragmentation activity. "

We provided trace files to DBAs and they confirmed that the jobs are not taking time in processing inside database and moreover database session in production was idle (SQL*Net message from client) for approx. 29 of the approx. 33 minutes of session time.

We have around 72 active processes in system and at certain point of time, there could be around 1000 process instances concurrently running in system. However, same load tested in lower environments work just fine.

Regards,
Devendra Jakhmola

[Suman Pramanik]

Hi Devendra

Glad you were able to ping point the HOST NAME issue, I do agree with your DBA that fragmentation is not only the cause however when we look at performance issue there is every other aspect needs to be looked . Now you also said the session were idle at database so its needs to be looked by taking probably multiple thread dumps to see if the processes are waiting and unable to get the adequate CPU threads

Now coming to active process you said 72, and it could be 1000 too. However my question was about total process in the system that include even completed as well as error ones .

I also see you have a case opened and we will work with you over the case to troubleshoot because i dont think there is anything further can be offered by me on the community without looking into details. 


------------------------------
Thanks & Regards
Suman Pramanik
Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
Broadcom
------------------------------

[Devendra Jakhmola]

Hi Suman,

The issue appeared to be caused due to configuration change in Database for Oracle Dataguard (used for disaster recover). Apart from Clarity upgrade, we had upgraded Oracle 12.1 to 12.2 and during this migration, the Dataguard's setting was changed from "Maximum Performance" to "Maximum Availability".

After DBA restored the setting, application performance drastically improved.

Regards,
Devendra Jakhmola

[Suman Pramanik]

Maximum Availability can still work well if you can manage the redo logs well, so i would say to test this in non production and check the log switching and gradually increase size of redo logs 

------------------------------
Thanks & Regards
Suman Pramanik
Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
Broadcom
------------------------------

[Paul Schofield]

Hi Suman, I'm trying to get JDBCPing working on AWS. Everything seems to work except I can't see the 2nd server in the cluster in the NSA and hence I can't see any traffic between the nodes in admin tower. Can you give an example of how the $HOSTNAME in the hosts file looks on a Windows platform? I'm assuming it should be localhost but can't decide if it should be the server name, the IP address or just local host (without any quotes).

[Suman Pramanik]

Is both the server on same subnet and RMI ports are communicable, there are IP restrictions in AWS zone something that we need to consider. Ideally if we add IP address to etc/host file that should suffice

------------------------------
Thanks & Regards
Suman Pramanik
Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
Broadcom
------------------------------

[Paul Schofield]

Hi Suman,

 

The problem turned out to be not having ephemeral ports open, coupled with an error in the guide, which states the multicast port  (9090 by default) is used for the TCP traffic between the nodes. It's not. It's the Beacon/Client port (9091 by default) that's used. After fixing these two points the JDBCPing is working fine.

 

We've not made any changes to the hosts file (even though the guide says they are required).

 

Thanks for following up!

 

 

Paul Schofield

Clarity Technical Consultant 
_____________________________________ 
 
1010 Cambourne Business Park | Cambourne | Cambridge | CB23 6DP|UK

m: [phone] | www.ignite-tec.com 

 

 

 

 

[Suman Pramanik]

Paul - I will review and get the document corrected, yes its not multicast port but beacon port as beacon service sync the messaging between each node in the cluster. 

The etc host is optional and not mandatory, it's the tip we have added for troubleshooting.

------------------------------
Thanks & Regards
Suman Pramanik
Sr. Principal Support Engineer | Customer Success & Support, Enterprise Software Division
Broadcom
------------------------------