vmware esxi virtual machine invalid

[nazim109]

Dear ,

Thanks in advance.

I am facing issue in my vmware ESXI 6.5.0.

I have two datastore. 256GB ssd for OS and 1.6TB storage for virtual machines.  in the storage menu showing only 256GB storage.

But 1.6 TB showing at storage>Device also my all virtual machine status is invalid.

Can you please assist me how can i recover it without losing data?

 

name: vmfs/volumes/5aa2a95d-897c4df3-ae49-002590c62e7c

status : Invalid

 

Attached log.

 

Thanks

 

 

 

[hayoken]

Did you figure out how to do this? This just happened to me to day when I lost power.

[a_p_]

Dos the command esxcli storage vmfs snapshot list show any results?

André

[hayoken]

I found out that it corrupted on the storage. I am just redoing the vm. Thanks!

[nazim109]

André,

[root@localhost:~] esxcli storage vmfs snapshot list
[root@localhost:~]

 

 

No snapshot available.

[hayoken]

The command he asked for was if you take snapshots of your vms. If you did, then you could have restored from one of those.

[nazim109]

No snapshot is available.

Now what can i do now?

 

[a_p_]

Please rescan the storage adapter, and then check whether the host's vmkernel.log contains related entries.

André

[nazim109]

Dear André,

Already tried by rescan but not getting in data store.

Is there any other suggestion?

Thanks

[nazim109]

this is the log of rescan

 

2015-01-19T17:42:37.977Z cpu1:66292)ScsiDeviceIO: 2948: Cmd(0x439509ac7500) 0x1a, CmdSN 0x5084 from world 0 to dev "naa.61866da07306b2001c4bb566bc470bf1" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2015-01-19T17:42:39.858Z cpu4:65773)FSS: 5749: No FS driver claimed device 'naa.61866da07306b2001c4bb566bc470bf1:1': No filesystem on the device
2015-01-19T17:42:39.867Z cpu21:67822 opID=98db16a)World: 12230: VC opID 85e2cd14 maps to vmkernel opID 98db16a
2015-01-19T17:42:39.867Z cpu21:67822 opID=98db16a)VC: 4511: Device rescan time 180 msec (total number of devices 6)
2015-01-19T17:42:39.867Z cpu21:67822 opID=98db16a)VC: 4514: Filesystem probe time 18 msec (devices probed 4 of 6)
2015-01-19T17:42:39.867Z cpu21:67822 opID=98db16a)VC: 4516: Refresh open volume time 1 msec

[a_p_]

Does the following command (a single command line) show details about the missing datastore?

offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done

André

[nazim109]

Dear 

Please have the output of this. 

[root@localhost:~] offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { fo
r i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128
-s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------";done
/vmfs/devices/disks/naa.61866da07306b2001c4bb566bc470bf1
Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)?
Warning: Not all of the space available to /dev/disks/naa.61866da07306b2001c4bb566bc470bf1 appears to be used, you can fix the GPT to use all of the space (an extra 4300800 blocks) or continue with the current setting? This will also move the backup table at the end if is is not at the end already. diskSize (2924216320) AlternateLBA (2919915519) LastUsableLBA (2919915486) NewLastUsableLBA (2924216286)
gpt
182024 255 63 2924216320
1 128 2919915480 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
---------------------
/vmfs/devices/disks/t10.ATA_____SanDisk_SDSSDA240G______________________173566803571________
gpt
29185 255 63 468862128
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
3 15472640 468862094 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
---------------------

[a_p_]

Can you please run the following 4 commands, and post the output?

hexdump -C "/dev/disks/naa.61866da07306b2001c4bb566bc470bf1" | grep -i 01400000 -A20 -B20
hexdump -C "/dev/disks/naa.61866da07306b2001c4bb566bc470bf1" | grep -i 01310000 -A20 -B20

partedUtil getptbl "/vmfs/devices/disks/naa.61866da07306b2001c4bb566bc470bf1"
partedUtil getUsableSectors "/vmfs/devices/disks/naa.61866da07306b2001c4bb566bc470bf1"

André

[nazim109]

Dear,

 

This message is showing after few minutes of hex dump.

 

error:

 

Remote side sent SSH2_MSG_EXT_ INFO after

USERAUTH_SUCCESS

[nazim109]

Here the another 2 command output:

[root@localhost:~] partedUtil getptbl "/vmfs/devices/disks/naa.61866da07306b2001c4bb566bc470bf1"
Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)?
Warning: Not all of the space available to /dev/disks/naa.61866da07306b2001c4bb566bc470bf1 appears to be used, you can fix the GPT to use all of the space (an extra 4300800 blocks) or continue with the current setting? This will also move the backup table at the end if is is not at the end already. diskSize (2924216320) AlternateLBA (2919915519) LastUsableLBA (2919915486) NewLastUsableLBA (2924216286)
gpt
182024 255 63 2924216320
1 128 2919915480 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

 

[root@localhost:~] partedUtil getUsableSectors "/vmfs/devices/disks/naa.61866da07306b2001c4bb566bc470bf1"
Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)?
Warning: Not all of the space available to /dev/disks/naa.61866da07306b2001c4bb566bc470bf1 appears to be used, you can fix the GPT to use all of the space (an extra 4300800 blocks) or continue with the current setting? This will also move the backup table at the end if is is not at the end already. diskSize (2924216320) AlternateLBA (2919915519) LastUsableLBA (2919915486) NewLastUsableLBA (2924216286)
34 2924216286

[a_p_]

I could be wrong, but to me this looks like a disk drive issue.

To rule out an ESXi (e.g. driver) issue, you could try an boot the system using a Live Linux, to see whether the HexDump works.
You may of course change the path to the disk device for the command.

André

[nazim109]

Dear André

without hexdump is there any way to recover it?

 

[a_p_]

It's actually not about hexdump, or any other command. It's basically about finding out whether the disk can be accessed at all using a Live Linux.

André

[nazim109]

Dear,

No change last 1 hour.

 

 

[a_p_]

This at least shows that the volume is readable, so it may likely be something related to how ESXi accesses the datastore.
I'm not a data recovery expert, so please make sure you double check recommendations.

What may be something that can help with troubleshooting is the vSphere On-disk Metadata Analyzer (VOMA).
See e.g.  https://kb.vmware.com/s/article/2036767

André

[Nathan Savolskis]

This certainly reads along like a disk failure.  Which is unfortunate, but can you access this disk any other way, and migrate vmdk files somewhere else?