Local disk RDM stopped working

[kzqc]

I have a PC running ESXi 6.7, managed by vcenter running on a different server. This PC has 1 SSD for local vfms 6 datastore, and 9 HDD's mapped thru RDM and used in two VMs. This setup has being working for several years since ESXi 5.5, 6.0, and last year upgraded to 6.7.

But today, when I was doing some VM management, when I shutdown, removed the RDM HDD, and later tried to added it back, it failed with error "Incompatible device backing specified for device 0". I just cannot add the RDM disk back to the VM. At this time, the other VM with the 8 RDM HDD's is still running fine. Then I stopped that VM, and check the settings, again, the RDM disks for that VM is showing in red error as well. Now none of the RDM HDD's is working for me.

I originally used the vmkfstools -z command to create the RDM vmdk in a common @RDM subfolder in the local SSD vmfs datastore. Now I cannot rm these vmdk files. When I tried to create these RDM vmdk files in a different @RDM2 subfolder, I get the following error:

[root@... ... .../@RDM2] vmkfstools -z t10.ATA_____ST32000542AS_____________________

___________________5XW0XR9N  ST32000542AS_5XW0XR9N.vmdk

Raw device, '/vmfs/volumes/5c5c96da-3ffa9809-d5cd-bc5ff4d4f02c/@RDM2/t10.ATA_____ST32000542AS________________________________________5XW0XR9N' doesn't exist, or isn't a disk.

Now I even reinstalled the ESXi to the latest 6.7 20190104001, but the problem is same.

I searched on the web and cannot find any answers. Anyone can help me with this? Thanks.

[kzqc]

Additional information: forgot to mention, the disk devices are still in the /vmfs/devices/disks/ subdir

ls /vmfs/devices/disks/

mpx.vmhba35:C0:T0:L0

mpx.vmhba35:C0:T0:L0:1

mpx.vmhba35:C0:T0:L0:5

mpx.vmhba35:C0:T0:L0:6

mpx.vmhba35:C0:T0:L0:7

mpx.vmhba35:C0:T0:L0:8

t10.ATA_____ST32000542AS________________________________________5XW0XR9N

t10.ATA_____ST32000542AS________________________________________5XW0XR9N:1

t10.ATA_____ST32000542AS________________________________________5XW0XR9N:2

t10.ATA_____ST5000DM0002D1FK178__________________________________W4J09CF4

t10.ATA_____ST5000DM0002D1FK178__________________________________W4J09CF4:1

t10.ATA_____ST5000DM0002D1FK178__________________________________W4J09CF4:2

t10.ATA_____ST5000DM0002D1FK178__________________________________W4J09CF4:5

t10.ATA_____ST5000DM0002D1FK178__________________________________W4J09CF4:6

t10.ATA_____Samsung_SSD_860_EVO_500GB_______________S3Z1NB0K903414W_____

t10.ATA_____Samsung_SSD_860_EVO_500GB_______________S3Z1NB0K903414W_____:1

t10.ATA_____TOSHIBA_HDWA130____________________________________75H9GMNGS

t10.ATA_____TOSHIBA_HDWA130____________________________________75H9GMNGS:1

t10.ATA_____TOSHIBA_HDWA130____________________________________75H9GMNGS:2

t10.ATA_____TOSHIBA_HDWA130____________________________________75H9GMNGS:5

t10.ATA_____TOSHIBA_HDWD130____________________________________1867YNPAS

t10.ATA_____TOSHIBA_HDWD130____________________________________1867YNPAS:1

t10.ATA_____TOSHIBA_HDWD130____________________________________1867YNPAS:2

t10.ATA_____TOSHIBA_HDWD130____________________________________1867YNPAS:5

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N0E43ZW7

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N0E43ZW7:1

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N0E43ZW7:2

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N0E43ZW7:5

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N1571602

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N1571602:1

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N1571602:2

t10.ATA_____WDC_WD30EZRX2D00D8PB0_________________________WD2DWMC4N1571602:5

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX11D74RHT2H

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX11D74RHT2H:1

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX11D74RHT2H:2

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX11D74RHT2H:5

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX11D74RHT2H:6

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX31DA43KL3J

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX31DA43KL3J:1

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX31DA43KL3J:2

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX31DA43KL3J:5

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX31DA43KL3J:6

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX41D849PR06

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX41D849PR06:1

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX41D849PR06:2

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX41D849PR06:5

t10.ATA_____WDC_WD50EZRX2D00MVLB1_________________________WD2DWX41D849PR06:6

vml.0000000000766d68626133353a303a30

vml.0000000000766d68626133353a303a30:1

vml.0000000000766d68626133353a303a30:5

vml.0000000000766d68626133353a303a30:6

vml.0000000000766d68626133353a303a30:7

vml.0000000000766d68626133353a303a30:8

vml.0100000000202020202020202020202020355857305852394e535433323030

.....

.....

vml.010000000053335a314e42304b39303334313457202020202053616d73756e:1

[continuum]

>  the disk devices are still in the /vmfs/devices/disks/ subdir
??? please explain - that directory is the only directory where you should ever find those files / device nodes
> When I tried to create these RDM vmdk files in a different @RDM2 subfolder, I get the following error:
I was wondering about this when I read your post last night ....
Do you really create a directory with @RDM2 in the name and move the t10.ATA___* files / device nodes into that dir ?
If you really do that then I have no further questions.
STOP IT

[kzqc]

wait. I found my mistake.

[continuum]

do you want to leave without telling us about your mistake ???

[kzqc]

No, I original wrote a long reply to your questions, and just after posting that, I found my mistake. So I did a quick edit. Then went off to correct my mistake.

Anyways, it was stupid. I forgot to add the absolute path to the devices. I was in @RDM2 subfolder, and of course the devices do not exist in that folder.

With absolute path, for the devices, I can create the new RDM vmdks in the new @RDM2 folder, and was able to use those in the VM's.

Now I still cannot remove the old RDM vmdk files from the old @RDM subdir.

[root@KGC10:/vmfs/volumes/5c5c96da-3ffa9809-d5cd-bc5ff4d4f02c/@RDM] rm *

rm: can't remove 'ST32000542AS_5XW0XR9N-rdmp.vmdk': No such file or directory

...

...

rm: can't remove 'WD50EZRX2D00MVLB1_WD2DWX41D849PR06.vmdk': No such file or directory

However, I was able to rename the @RDM subdir to a different name @Bad, to put it aside. I had similar experience with this kind of stuck files in vmfs before. I could not find anyways to remove them, other than wiping and recreating the vmfs volume.