Clustering keeps getting out-of-sync

[Jorghy Misnan]

Product: CA PAM 3.2.4

Issue: Secondary node keep getting out-of-sync for no apparent reason. I'll provide more information if there's anyone who can help me solve this issue. Thank you.

[JAVIER HERRERO]

Hi, We have similar problem whith cluster in Version 3.2.4. (Already opened a case)
Records like this in cluster logs:
"PAM-CM-0451: ServerReachabilityMonitor.run Server '***.***.***.***' reachability changed: 443=<removed> (filtered), 3306=<removed> (filtered), 5900=<removed> "
Finally at any time, we don´t know , the cluster gets down.

As a work around, CA technologies Techinician has change the next parameter : Configuration/Clustering/tuning/Max number of...from 10000 to 20000
But still going down sudenly.

I´ll tell you all if it gets solved

[Jorghy Misnan]

In my case, i have 3 nodes on three different sites with one act as primary. I've tried playing around with Cluster tuning with no result, it still getting out-of-sync, either the session manager or credential manager got red cross mark. At that point, resync no longer works, and i have to restart the cluster (which on production).

[Ralf Prigl]

Hello, Cluster synchronization problems can have multiple causes and almost always require log review. This cannot be done in the context of a community discussion. Please open a support case and attach system and session logs from all cluster nodes for review.

[Jorghy Misnan]

I've had also raised this issue to support, they promise me to fix it on version 3.2.4 yet the problem have not go away.

[JAVIER HERRERO]

We are in same situation, they told us in ver.3.2.4 should be solved but it is not true. 

[Adolfo Navarro]

Hello Ralf,

In this moment, we have a open case about this issue 3 weeks ago, we do not have a satisfactory answer yet and the issue persist and the customer (Core account) complain about that. Please can you tell me where can i setting next: "As a work around, CA technologies Techinician has change the next parameter : Configuration/Clustering/tuning/Max number of...from 10000 to 20000"

In the image below do not appear that route "Configuration/Clustering/tuning/Max number" where i can found it.

Yesterday we upgrade to version 3.3 and the message persist.

Thank for you help.

Adolfo.

[Ralf Prigl]

Hi Adolfo, Cluster tuning parameters, and how to make them visible in the UI, are discussed in our online documentation, see https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-3/deploying/set-up-a-cluster/configure-a-cluster.html#concept.dita_30f62b5da45eda51a0c59e3b30699fbe1d7588aa_ClusterTuningClusterTuning

[Ralf Prigl]

​I received a reply from Adolfo, but don't see it here. It's stating that changing the configuration setting doesn't eliminate messages like

WARNING:PAM-CM-0451: ServerReachabilityMonitor.run Server '10.xx.xx.xx' reachability changed: 443=<removed> (filtered), 3307=<removed> (filtered), 5900=<removed> (filtered), 7900=<removed> (filtered), 7901=<removed> (filtered)

Such messages would not be affected by any tuning parameters. They come from a watchdog that periodically checks connections to other cluster nodes in the same site for the ports involved in cluster synchronization. None of the cluster tuning parameters are of any interest to that watchdog. They also are not concerned with the actual synchronization of data. As long as the databases keep showing active on the cluster status pages there is no real problem with the synchronization.

[Adolfo Navarro]

Hi Ralf, 

I made an access policy with a privileged IAM AWS account but when entering the AWS console i don't see the resources, however if I login with the same IAM account directly in the AWS console if it is possible to see the resources. Why does this happen? What would it take for what through PAM if it is possible to view AWS resources with the same IAM account?

Maybe editing AWS user access policies? I'm sure.

Please your comments.

Thank You.

Adolfo.

[Ralf Prigl]

Hi Adolfo, Sorry, I don't understand how this is related to the cluster out-of-sync problem.

[Jorghy Misnan]

The troublesome part is, there's barely any pattern as to when this behaviour happen and the cluster logs simply too lacking to help me troubleshooting. The most i can do is aside of increasing the replication to 20000, also the connection timeout in each site to 90 sec and the poll freq to 300 sec. That will reduce the interval for out-of-sync in my experience.