Invalid VMs Datastores

[Alysson Diaz]

We have an ESXi v6.7 server on Cisco UCS C220 M5 that has 10 hard drives with 2.4TB each and in HBA mode.

1 hard drive is failing, we just realized it when we were trying to upgrade a VM's storage from 100GB to 500GB, however, we were unable to because of this hard drive issue.

When we attempted to troubleshoot it, we ran /etc/init.d/hostd restart, this triggered all the VMs to go to "Invalid" status.

datastore1 is gone, Storage tab displays "0".

Without trying to restart the whole server in doubt that if we restart it, all VMs will be gone, thru SSH, we managed to pinpoint the failing drive, it has "failed H:0x5 D:0x0 P0x0 Invalid sense data: 0x0 0x6e 0x73" message and it is in slot 7, by cross checking with our ESXi's GUI, we noticed that this failing disk (naa.5000039a882017c9) has only 187.62GB of capacity instead of 2.4TB.

We ended up planning a scheduled VMs backup operation to a backup server, managed to backup only 2 VMs but unable to do the same with the remaining VMs because we are unable to terminate their other processes before backing them up.
We have already tried kill -t soft, hard and force commands to terminate these processes but to no avail, is there still any other way to do this?

All screenshots were uploaded, hoping they will help me discussing our issue.

Any help would be greatly appreciated, thank you so much!

[Nathan Watts]

I have no answers for you. 

I do recall working with 6.7U3 waaayyyy back then, and we would get into situations similar to this where the host would become unresponsive and you could not cleanly failover the VMs etc, the only thing to do was hard reset the host with a force power off from the out of band MGMT. This was an issue in 6.7, I believe was resolved at some point.
We had our VMs on shared NFS storage though and the VMs would restart on other host nodes, this looks stand alone to me?

In this situation you could not do anything with the VMs from the Host level, backup etc. the host would gradually get worse over time (days) until we had no choice but to reset the host.

I'd recommend that you get that host upgraded to vSphere 7 then 8 (to be on a supported version) assuming that you have maintenance on that license key.

[Alysson Diaz]

Hi Nathan, thank you for the response. Yes, this is a standalone ESXi 6.7 host and the VMs are stored on a local multi-extent VMFS datastore across 10 SAS disks in HBA/JBOD mode. One of the disks is already returning hardware I/O errors and is part of the datastore.

At the moment, several VMs have already been shut down at the guest level, but their VMX/vCPU/vmm worlds remain and cannot be terminated. vim-cmd hangs, localcli vm process list returns Sysinfo errors, and the Host Client times out.

We are concerned that a hard reset may cause the multi-extent datastore not to mount again because of the failing disk. In your experience, would you still recommend a host reset in this condition, or would you first clone/replace the failing physical disk before powering the host back on?

We are also considering using GNU ddrescue to sector-clone the failing SAS disk to an equal-or-larger replacement before the next ESXi boot.

Any guidance would be greatly appreciated.

[Nathan Watts]

Hi Alysson,

Please keep in mind I am not responding for Global Support and not providing any VMware/Broadcom recommendations here.

Being this is an extent/JBOD datastore, the loss of a single disk can jeopardise the entire volume, shutting down that host in it's current state is going to be a "hold your breath" moment, the host should come back fine, the datastore may return failures and not mount, I simply do not know.
as you have a volume that is spanning multiple disks without any form of redundancy it's highly likely you will see multi VM data loss or none, just depends on where the data is written, and how that volume will behave on remount.
if you have a method to clone the drive to a new disk and replace it, I'd do that first.

When we had the issue (VMs unresponsive, unable to kill process) GS was unable to give us a method to kill the VM worlds, the only way we could get the VMs running healthy again was to reset the host.
How recent and good are you backups?
be prepared with them should you need them.
Also great time to get some redundant storage, even a simple NAS with some form of RAID would be better than an internal JBOD. Assuming that 
The C220 does have a RAID option but it sounds like when this was built they went for max capacity and failures to tolerate.
I wish I had better news and options for you, but sadly I do not.