Datastore has disappeared from UI, yet VMs still functioning

[carpii]

My vSphere server (Esxi 6.7 U2) has two drives..

• 2G ATA (holds the 'store2' datastore)
• 2G SSD (1G for system/boot, and 1G for the 'store' datastore)

The 'store' datastore has completely disappeared, even though the drive is still shown in vSphere

Whats strange is that even though 'store' is no longer shown, all the VM's it contains are still listed in vSphere, and I can fire them up without issue.

This is true even after a reboot. Rescanning drives and datastores from vSphere does not help.

The commands below dont show any obvious problems:

[root@vsphere:~] esxcfg-scsidevs -m
t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500:3      /vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500:3      5d25f4cb-e5c87db4-bd66-80ee73e51739  0  store
t10.ATA_____ST2000DM0082D2FR102__________________________________WFL2A4N6:1 /vmfs/devices/disks/t10.ATA_____ST2000DM0082D2FR102__________________________________WFL2A4N6:1 5d260dd8-042c3326-db3d-80ee73e51739  0  store2

[root@vsphere:~] esxcli storage filesystem list
Mount Point                                        Volume Name     UUID                                 Mounted  Type              Size            Free
-------------------------------------------------  --------------  -----------------------------------  -------  ------  --------------  --------------
/vmfs/volumes/b14b3c4e-6878019f                    remo-vmbackups  b14b3c4e-6878019f                       true  NFS     53736225628160  42260340019200
/vmfs/volumes/5d25f4cb-e5c87db4-bd66-80ee73e51739  store           5d25f4cb-e5c87db4-bd66-80ee73e51739     true  VMFS-6   1016028200960    197688033280
/vmfs/volumes/5d260dd8-042c3326-db3d-80ee73e51739  store2          5d260dd8-042c3326-db3d-80ee73e51739     true  VMFS-6   2000112582656   1655821041664
/vmfs/volumes/9dc8d475-9f29c6f7-d352-b6ffd4ba5c17                  9dc8d475-9f29c6f7-d352-b6ffd4ba5c17     true  vfat         261853184       108232704
/vmfs/volumes/afd60c70-e6c42821-d19a-561968def717                  afd60c70-e6c42821-d19a-561968def717     true  vfat         261853184       261849088
/vmfs/volumes/5d25f4c5-b0ac17f0-5ca8-80ee73e51739                  5d25f4c5-b0ac17f0-5ca8-80ee73e51739     true  vfat         299712512       118415360
/vmfs/volumes/5d25f4f3-b2579642-2da5-80ee73e51739                  5d25f4f3-b2579642-2da5-80ee73e51739     true  vfat        4293591040      4259446784

 

How can I link the 'store' datastore so it shows up in vSphere UI again, and can be browsed using Datastore browser?

[carpii]

Update: I ssh'd into vSphere and moved all the vm's from the broken store datastore, onto store2

I then deleted the partition which the store datastore was on, with the intention of recreating the partition

I was able to delete the partition, but now I can;t create a new one. vSphere fails with the error "Failed to create VMFS datastore. Cannot change host configuration"

Is there a way I can create and format the partition from CLI?

[Andrew Nash]

Ive seen something like this before.

have you seen this post 

Solved: Failed to create VMFS datastore - Cannot change th... - VMware Technology Network VMTN

[a_p_]

A datastore on the installation disk can neither be created nor modified from the GUI.
However, it is possible to do that from the command line.

Please provide the output of the following two commands (as plain text please).

partedUtil getptbl "/vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500"
partedUtil getUsableSectors "/vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500"

André

 

[carpii]

Hi, thanks for the help

 

[root@vsphere:~] partedUtil getptbl "/vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500"
gpt
124519 255 63 2000409264
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

 

 

 

[root@vsphere:~] partedUtil getUsableSectors "/vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500"
34 2000409230

 

 

As a sidenote, do you know if its normal for there to be two vmkDiagnostic partitions?

I feel like when I first started troubleshooting this issue, I only noticed one, but I might be mistaken.

[a_p_]

The existing partition look fine.

In order to re-create the VMFS partition run the following three commands:

partedUtil add "/vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500" gpt "3 15474688 2000408575 AA31E02A400F11DB9590000C2911D1B8 0"
vmkfstools -C vmfs6 -S "store" "/vmfs/devices/disks/t10.NVMe____Samsung_SSD_970_PRO_1TB_________________AEAEB49155382500:3"
vmkfstools -V

Explanation:

1. create a new VMFS partition no. 3
2. format the new partition as a VMFS6 file system, and name it "store"
3. perform a storage rescan

André

[carpii]

Wow, that's some magic you did there. It worked perfectly, thankyou so much

 

Do you have any idea how the datastore got into this state in the first place?

 

I don't recall making any changes to vSphere, but I noticed my VM backups had stopped working for a week or two

I use Active Backup for Business to backup my VM's (from my Synology NAS) - Part of that backup process requires ESXi to create a snapshot of each VM.

 

But when I investigated this, I saw that vSphere was failing to snapshot every VM on 'store' with 'required file cannot be found'

Its only when I dug into this more, that I realised the datastore had gone completely awol