Need to re-ip vSAN hosts on different network and physical uplinks

[MarcLaf]

I've received some guidance around this elsewhere and just want to double check (second opinion).

We need to re-ip multiple vSAN hosts (a 3 host cluster and a 4 host cluster). These IP's are on a different network which require using different uplinks.

We are using VDS and I have already configured the new port groups using the new uplinks.

Here is the plan I have so far.

1. Create new vmk adapters for new management IP and override default gateway on the TCP/IP stack.
2. Assign this vmk to the new PG using the new uplinks - make sure this is Ephemeral binding
3. Migrate existing Management vmk to a new PG that uses Ephemeral binding
4. Disconnect and remove host from inventory
5. Update DNS and ensure vCenter can resolve new IP
6. Re-add host using hostname with new IP
7. Rejoin to cluster.

What I have mixed replies on is whether or not I need to use maintenance mode for this. If a host is a member of a vSAN cluster and it gets disconnected from vCenter (and removed), since the vSAN network is still up and the VM network is still up, these VM's should still run. I don't want vSAN to freak out and prevent the host from rejoining the cluster when I add it back.

There is also the aspect of needing to override the default gateway for the TCP/IP stack permanently. The only thing that is using this stack is the current management vmk but I can't change the stack without losing connectivity. I WOULD also just change the IP using the DCUI however it appears to be permanently bound to using vmNIC0. I can't change the NIC in the DCUI (the option is greyed out). I can only change the IP but changing that means it's updating vmk0 which as I mentioned, is using the old physical NIC and this is not on the right network.

This leads me to my last unanswered question - what is the relationship between setting management in DCUI and what we have in vCenter? Meaning, when you initially set up management in DCUI, you pick a physical NIC, set IP, VLAN, GW, etc. But later if you are using say vmk4 for management, does the DCUI know this? So in my scenario if I add vmk4 as management, then remove and rejoin using this new IP, and then go into DCUI, will it be linked? (Sorry if that's not explained very well, I'm having a hard time articulating what I am asking here). DCUI sets an IP on a physical NIC whereas in vCenter you are setting an IP on a VMK which is a member of a PG which has an uplink assigned to it. It's apples to oranges here and I don't know how they're linked.

[TheBobkin]

@MarcLaf, Most of your question here should be asked in the ESXi/Networking Community as these aspects are not really vSAN-related at all.

But with regard to the vSAN aspect and the not wanting "vSAN to freak out" - the cluster won't care (or even notice really) a node leaving the vSphere cluster and coming back with a different Management IP address, however the action of it being removed from vSphere inventory may/will cause the node to be evicted from the cluster (because vCenter told the other nodes that ESXi-01 is gone now).

So you should use MM if possible when changing each node, if for whatever reason this isn't possible (e.g. lack of compute resources or god-forbid....FTT=0 data...) then you can prevent this from being possible by disabling unicastagent updates on all nodes before starting the overall change (and of course reverting it after all are changed).

Check current configuration:

# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates

Enable:

# esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates

Disable:

# esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates