Problem with upgrading from 11.0 to 11.1 Without downtime

[Pasquale Gianluca Costa]

Hi,

 

I need to upgrade the patches without downtime, as per the documentation

 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/11-0/install-configure-upgrade/configure-a-gateway-cluster/deactivating-a-cluster-node.html

 

where downtime is expected.

 

The infrastructure in question consists of a cluster of 4 machines, 2 with databases and 2 without databases. The two database machines are replicated.

 

How can I proceed?

Thank you

[Gustavo Azolas]

Hi Pasquale :

I do not see a secure way to achieve a Gw upgrade without downtime on this use case , because the ssg replication in place.
- to stop and restart that replication will have a  downtime due need to stop ssg service too.
- Patching Gw requires also a gateway server reboot. 
Will be more secure if do you find a window time to apply the upgrade and in case of replication broken you will be able to re-initialize properly too.

-------------------------------------------

[Vince Baker]

Hi Pasquale,

Maybe I'm missing something, but why can't this be done the way we've always upgraded a single cluster in the past? i.e. Declustering, or in your case splitting the cluster into just 2 parts. The following would be based on you being able to provide service with 2 out of 4 gateways running. If you don't have enough capacity I'm sure Broadcom will help with temporary upgrade licensing to add more resources for a few hours only.

I'd do the following:

1. Ensure replication is in sync.
2. Break replication
3. check, or if needed, point nodes 1 & 3 to Node 1's DB (call it upg_cluster_1 for this purpose)
4. point nodes 2 & 4 to Node 2's DB ((call it upg_cluster_2 for this purpose)
5. Cut traffic to nodes 2 & 4 using load balancer or iptables (Leaving traffic only going to upg_cluster_1)
6. Do upgrade on upg_cluster_2 
7. Do hidden live tests in upg_cluster_2
8. Enable upg_cluster_2 in load balancer
9. Repeat steps 5 onward for 1 & 3 (upg_cluster_1) 
10. Re-enable replication
11. Point nodes 2 & 4 to Node 1's DB
12. Restart Nodes 2 & 4
13. Put all nodes back in load balance pool.

We'd be happy to help you further in an Expert Services engagement if needed.

Regards

Vince Baker

-------------------------------------------

[Ryan Rank]

Hi Pasquale,

I've done what Vince said and it works really well. It's tedious and you have to be very careful, but can absolutely be done. As Vince said, one of the biggest things is to make sure your load balancer is solid. Either have some solid health monitors (I have something in place where I can manually mark a node down by changing a cluster wide property) or mark the unavailable node administratively down/remove it from the pool at the load balancer level. You don't want to risk live traffic going to a non-functional node.

One thing to consider is capacity. If your 4 node cluster is at 75% total capacity or higher, you'll create a partial downtime anyway. So you'll have to be mindful of how many nodes are down at any given point in time. 

-------------------------------------------