Recover orphan VMs of vSAN cluster

[sistemas-cmic]

I explain the catastrophe. I had a cluster with 3 hosts. Everything working perfectly, until I changed the MTU of the vmk and totally lost communication. I rebooted all 3 hosts and no virtual machines booted. No more vcenter vsphere (just failed to power on allways). All vm's show up with pathnames like /vmfs/volumes/vsan:xxxxx... nothing works.
So I installed another vcenter from scratch, and added the hosts to that cluster, configured everything again (this time with default MTU). But it turns out that all virtual machine objects seem to reference an old uuid from the previous vsan cluster. Disks are automatically claimed but with errors, ie inaccessible state.
Please i need to recover those virtual machines, at least the vmdk files that seem to have disappeared into nothing.

Don't ask me why I did such a thing in production. Nobody here knows anything and I have to guess and learn as I go.

[Digory34]

Hi Sistemas-cmic,

As you have rollback your MTU setting? could you try to power on the old VCenter?

Also, could you do a esxcli vsan cluster get from command line on one host ? and check the unicast agent list?

Configuring vSAN Unicast networking from the command line (2150303) (vmware.com)

I guess, If the vsan hosts can communicate between them from VSAN traffic, the VMs and vsan datastore should be accessible as your rollback your MTU settings.

[sistemas-cmic]

U are a genius! that was exactly what i needed, I had already seen it but I had not understood it. After that all the vm's went back to their original names and i was able to boot vcenter.

Then i had another problem with the vds network. I had to reset the network configuration of all hosts from the ILO (with help from Creating a vSAN Cluster without a vCenter Server) and do the unicastagent thing again. Everything returned to normal.

Tthank you very much really!

[Digory34]

Glad to have assist you on this.

Have a nice day!

[Jamie Gravatte]

We're so glad this answered your question! Could you kindly click the "verified solution" button in order to better help your peers find this information?

Thanks!

Jamie - Digital Support Team

[sistemas-cmic]

Yeah, of course. Thanks a lot!

[TheBobkin]

, two main points here:

 

MTU: if you change MTU to something mismatched or higher than switch/end-to-end supports, vSAN performance will degrade horribly, BUT, it won't isolate the nodes...until you reboot (or anything that causes leave+rejoin). The prudent action would have been to just set this back to 1500 on the vmk.

 

vSAN cluster & vsanDatastore have a 1:1 relationship: you have probably created a new vSAN cluster here and thus a new vsanDatastore - your OSFS path of all objects is probably now incorrect.

 

This can be fixed in 2 ways:

1. Rejoin the original cluster with the Sub-cluster UUID - if you don't have persistent logging then this can be pulled from an old log bundle or other resource. If you do have logs from before reboot then check vsansystem.log.

2. Change all Objects OSFS path to reference the new vsanDatastore ID instead of old.

 

While this issue was caused by human error, please don't worry about it, if you don't understand how to action the above, please open a case with vSAN GS so my colleagues can assist with 1./2. .