Minimizing downtime in Multi-site cluster configuration

[Chris Scott]

Hello,

Based on the requirements that all appliances must be on the same software version, I'm looking for a good strategy for upgrading while minimizing service downtime in a multi-site cluster configuration (given there are two sites, two members in each site).

I'm interested in any specific strategies the user community would like to share.

Thanks

Chris Scott

[Ralf Prigl]

Hello Chris, A good strategy is found in our online documentation at https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-3-2/upgrading/upgrading-across-a-multi-site-cluster.html

There are a couple of additional items not discussed specifically on this page:
- In 3.3 you can add nodes to the primary site while the cluster is on
- In 3.3 you should NOT have two nodes in the primary site. When one of them has a problem there will be quorum loss, and the other node will become passive, not a good situation to be in. Please consider adding a third node. See page https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-3-2/deploying/set-up-a-cluster.html and the Cluster Synchronization, Promotion, and Recovery page under it

For your case the procedure would be as follows:
- Take one secondary site node out of the cluster and upgrade it.
- Stop the cluster.
- Upgrade your master node (first node in the primary site)
- On the master node update the cluster configuration to only include itself and the upgraded secondary node, then turn the cluster back on.
- Upgrade the remaining two nodes and join them back into the cluster.
- Add a third node to the primary site.

If adding a third node to the primary site is not an option, you might want to change the cluster configuration and create a new secondary site for your current second primary node, resulting in a 1-1-2 cluster instead of a 2-2 cluster.

[Chris Scott]

Ralf,

First, appreciate your thoughts on cluster site setup. We are definitely working on the third appliance in the primary site.

Secondly, just to clarify your remarks - would you please verify my understanding .

> For your case the procedure would be as follows:
> - Take one secondary site node out of the cluster and upgrade it.
My understanding:> Eject the secondary site leader and upgrade it while end users are still using the primary site VIP.

> - Stop the cluster.
My understanding:> Assuming you meant primary cluster, end users will have to either logoff or get kicked off the primary site VIP and unable to entirely use PAM

> - Upgrade your master node (first node in the primary site)
My understanding: End users will continue to be unable to entirely use PAM while maintenance is being performed on the primary site.

> - On the master node update the cluster configuration to only include itself and the upgraded secondary node, then turn the cluster back on.
My understanding: End users will now be able to use the primary site VIP

> - Upgrade the remaining nodes and join them back into the cluster.
My understanding: End users will continue to be able to use the primary site VIP

If my understandings are correct, then would it be accurate to state this setup requires an approximate complete service outage of about an hour or two, as long as all goes well?  

If not, would it be possible to clarify my understanding?

Thanks 

Chris Scott

[Ralf Prigl]

Yes, that looks right. If you run in Operationally Save mode, you could unlock the secondary site node that is still on the older release right after stopping the cluster, and that would allow it to be used by PAM users while you upgrade the master node. You can do it with the second node in the primary site too, but only if there is no danger that it will run password update jobs, which would cause target accounts to go out of sync with the master.