Hi Ralf,
From the documentation its clear that
within the Primary site, member appliances communicate over ports
TCP/443, TCP/3306 (MySQL), TCP/5900 (Hazelcast), TCP/7900 (JGroups), TCP/7901 (JGroups heartbeat).
But it is not clear what communication ports are required for communication between Primary and Secondary site appliances (either/ both ways). Is port 443 the only port required for any synchronization?
ThanksOriginal Message:
Sent: 10-16-2019 11:44 AM
From: Ralf Prigl
Subject: PAM 3.2.x Multi-master clustering
Hi Sandeep, On pages like
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-2-6/deploying/ip-addresses-and-ports-for-network-connectivity.html you find the following entry under port 443:
|
Appliance: cluster member
|
Appliance: cluster member
|
Required bi-directional communication between members of a cluster.
|
So port 443 is required for communication between any two cluster members. For communication between primary site nodes you found the additional requirements already.
Original Message:
Sent: 10-16-2019 01:52 AM
From: Sandeep Sharma
Subject: PAM 3.2.x Multi-master clustering
Thanks for your response Ralf. Looks like multi-site cluster is the only approach for DR (across WAN) as per the way product (syncing mechanism among nodes, functionality of nodes at primary vs. secondary) is designed.
The product documentation for v.3.2.x does not clearly state the network port requirements between primary site nodes and secondary site nodes.
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-2-6/deploying/set-up-a-cluster/cluster-deployment-requirements.html"Clustered appliance
: In the primary site, these ports are required: TCP/443, TCP/3306 (MySQL), TCP/5900 (Hazelcast), TCP/7900 (JGroups), TCP/7901 (JGroups heartbeat). (For a standalone appliance, only TCP/443 is necessary.)"
It only states the port requirement for nodes in the primary site.
Please advise what specific ports are required between nodes in primary and secondary site for PAM v3.2.x.
Thanks
Original Message:
Sent: 10-09-2019 10:02 AM
From: Ralf Prigl
Subject: PAM 3.2.x Multi-master clustering
Hi Sandeep, Network stability is very important for multi-master clusters. Frequent packet losses, or temporary disconnects lasting 10s of seconds can cause the cluster to go out of sync, and you have to stop and start the cluster to get all nodes back in sync in PAM 3.2.X. A potentially even bigger problem is that when you loose connectivity for an extended period of time, nodes in two different sites will each regard the other site as down and start acting as masters. This is what is called a split-brain scenario. When you restart the cluster later on, you have to pick one of the nodes as new master, and any changes made by other nodes during the disconnect will be lost, potentially causing the loss of new passwords for some target accounts.
The new PAM 3.3 release is more tolerant in this regard. If you have three nodes in a site and one drops out temporarily, it will come back ok with no need to restart the cluster, and as long as you have three or more nodes in the primary site, a split-brain problem is much less of a concern.
All nodes in a multi-master cluster can execute scheduled jobs. That's why it's called multi-master, and an option to disable jobs on one of the nodes would run counter to the concept of this type of cluster. You clearly want a multi-site cluster. I don't regard your note as a valid point. As mentioned above, when a node goes out of sync temporarily in a multi-master site, a cluster restart will also be necessary to recover from it. It's not better than a multi-site cluster in that regard. Also, all communication across sites occurs on the HTTPS ports in PAM 3.2.
Original Message:
Sent: 10-08-2019 10:22 PM
From: Sandeep Sharma
Subject: PAM 3.2.x Multi-master clustering
Hi,
I have a few queries about setting up a
multi-master cluster (2 nodes in Production and 1 node in DR) in
PAM v3.2.6:-
Requirements to setup clustering are as per the documentation -
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-2-6/deploying/set-up-a-cluster/cluster-deployment-requirements.html#concept.dita_38e1c0583c6eb0be966b80c4c66116131592358f_NetworkRequirements
1. Is configuring a
multi-master cluster with nodes across 2 sites over the WAN link i.e. Production and DR supported? Is there any specific guidance on minimum recommended bandwidth and network latency?
Note: From the documentation - there are specific requirements under high network availability and WAN section - "
If the network is down, each cluster member times out after 20 seconds and that member is deactivated." and "Be mindful that network packet loss is not excessive."
2. For multi-master cluster i.e. primary site - "The first cluster member that is listed in a Primary site is the data synchronization source for all cluster members."
Do ALL members of multi-master cluster active i.e. perform password change activity and execute any scheduled jobs in the background? Is there any supported way to make any of the members passive i.e. not execute any of the activities in the background etc.?
If not, is it alright to block all traffic between the Production cluster nodes and DR cluster node and allow only data replication/ sync ports traffic?
Note: The reason for not going ahead with multi-site (with secondary site in DR) approach is due to known issues with clustering, which might be unresolved even in v3.2.6 i.e. cluster node(s) at secondary site get out-of-sync, "Secondary members can "self-heal" after being disconnected." not working as designed which requires restarting the cluster for syncing
Thanks,
Sandeep