Recover vSAN after ESXi and vCenter Reinstall

[btwood]

We have a cluster that is in a bad state following a power failure. Nobody had the credentials, and we had to reinstall and reconfigure.

What is the proper way to reconnect/reestablish vSAN?

We have already configured ESXi hosts for networking, and installed vCenter on a local datastore on one of the hosts.

This article by William Lam outlines some of the process. However creating a new vSAN ID, joining the hosts, and adding them to vCenter results in failing to boot VMs due to data issues.

Thanks.

[TheBobkin]

 , you created a new vSAN cluster and hence a new vsanDatastore - your VMs won't power on because they are stored at /vmfs/volumes/vsan:OldUUID/namespaceUUID/VM.vmx but you only now have /vmfs/volumes/vsan:NewUUID/namespaceUUID/VM.vmx available.

This is quite easily resolved by joining all the nodes back to the original vSAN cluster and hence pathing to vsan:OldUUID will be back.
As you have reinstalled the nodes there won't be any logs to get this cluster UUID back from (other than if you had external logging server), but this is easily obtained by looking at basically any object (other than ones created since you reformed the cluster).

If you look in the output of 'esxcli vsan debug object list --all' run on a node or objtool lookup of any object you can see the path listed like so:
Path: /vmfs/volumes/vsan:OldUUID/namespaceUUID/VM.vmdk

The vsanDatastore part is actually the cluster UUID in a different format, e.g. if you see 'vsan:123c163babcd1234-567860e1d4bdea01' then the original cluster UUID is 123c163b-abcd-1234-5678-60e1d4bdea01, so just run cluster leave on each node and rejoin them to that cluster with that UUID.

If the unicastagent lists aren't correct and you don't want to manually fix them (https://kb.vmware.com/s/article/2150303) and/or vCenter is fighting you over cluster UUIDs, then you can just remove all the nodes from the vSphere cluster (and confirm vSAN disabled on all of them), turn off vSAN on that cluster object, manually join vSAN cluster with correct UUID from command line on one node then move that node back into the vSphere cluster, then enable vSAN on that vSphere cluster and then move the remainder of nodes into the vSphere cluster (in exactly this order ).

[btwood]

This is very helpful. Thank you.

For anyone else that needs to go through reinstalls, don't mess with your cluster UUID. It seems that the recommended action after (re)installing ESXi, is to configure your vSAN networking, then add them to a cluster using vCenter. It seems that vCenter is able to pick up on the prior vSAN config and complete the vSAN configuration without much help.

This article was also useful:
https://kb.vmware.com/s/article/2151610

Setting a new vSAN UUID was definitely a mistake, but we were able to resolve that.

Thanks again. I'm marking yours as the answer as it helped to get us to the solution.