ESXi 7 Failed to open disk link

[MohammadRavaghi]

Hello Everyone,

there is a situation we have some VMs need to be repaired. i will try to post some usefull information and please help me to resolve this. i'm stuck at finding a solution for this cancerous Snapshots.

VM list shows files contained in VM folder. there is one snapshot which VMSD points to it  also . and there is no more Snapshots since i see it there:

Get Snapshot:
|-ROOT
--Snapshot Name : AD 96.10.24
--Snapshot Id : 8
--Snapshot Desciption :
--Snapshot Created On : 1/13/2018 16:41:5
--Snapshot State : powered on

checking Base Disk Integrity shows no errors:

vmkfstools -e ACTIVE\ DIRECTORY\ 2008\ R2.vmdk
Disk chain is consistent.

 

but checking Snapshot shows some errors:

vmkfstools -e ACTIVE\ DIRECTORY\ 2008\ R2-000003.vmdk
Failed to open disk link /vmfs/volumes/603a59bb-5476d2af-5742-d89d672b5ccc/Pishrafteh/DC/ACTIVE DIRECTORY 2008 R2-000003.vmdk :The specified feature is not supported by this version (24)Disk chain is not consistent : The specified feature is not supported by this version (24)

 

using dd command i could see the COWD output and as far as i know it is good news about the Snapshot file itself :

dd if="ACTIVE DIRECTORY 2008 R2-000003-delta.vmdk" bs=512 count=1 | hexdump -C
1+0 records in
1+0 records out
00000000 43 4f 57 44 01 00 00 00 03 00 00 00 00 00 00 0f |COWD............|
00000010 01 00 00 00 04 00 00 00 00 f0 00 00 1d 9a 0f 05 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200

 

Flat Disk vmdk descriptor file :

 

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=d5c0b21f
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 251658240 VMFS "ACTIVE DIRECTORY 2008 R2-flat.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "15665"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "7939dbb4ac24405834645245d5c0b21f"
ddb.toolsInstallType = "1"
ddb.toolsVersion = "10272"
ddb.uuid = "60 00 C2 9c 6c 9f c5 9f-ce 39 3e 46 ed c5 f5 b9"
ddb.virtualHWVersion = "13"
~
~
~

and here is the Delta Disk Descriptor file :

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=4b0939e5
parentCID=d5c0b21f
#isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="ACTIVE DIRECTORY 2008 R2.vmdk"
# Extent description
RW 251658240 VMFSSPARSE "ACTIVE DIRECTORY 2008 R2-000003-delta.vmdk"

# The Disk Data Base
#DDB

ddb.longContentID = "f500ac4755ec8a7a275f86a74b0939e5"
ddb.toolsInstallType = "1"
ddb.toolsVersion = "10272"
~
~

and lastly here is vmx file content  which i created using vmxrebuild.sh found in vmware KBs :

#!/usr/bin/vmware
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "17"
vmci0.present = "TRUE"
numvcpus = "4"
memSize = "4096"
sched.cpu.units = "mhz"
powerType.powerOff = "default"
powerType.suspend = "default"
powerType.reset = "default"
scsi0.virtualDev = "lsisas1068"
scsi0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
ide1:0.fileName = "/vmfs/volumes/5d2685e7-1b9ad381-7c55-d89d672b5ccc/issabel4-USB-DVD-x86_64-20170714.iso"
ide1:0.present = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.clientDevice = "TRUE"
floppy0.fileName = "vmware-null-remote-floppy"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
ethernet0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "ACTIVE DIRECTORY 2008 R2-000003.vmdk"
scsi0:0.present = "TRUE"
displayName = "ACTIVE DIRECTORY 2008 R2"
guestOS = "windows7srv-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "56 4d c5 a7 23 ff 1f 36-3c 23 0e e9 af 8a e0 fa"
uuid.location = "56 4d c5 a7 23 ff 1f 36-3c 23 0e e9 af 8a e0 fa"
vc.uuid = "52 6a 7a d3 e7 3f 62 89-85 6f 66 1d 7f 2e 94 0f"
chipset.onlineStandby = "FALSE"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
sched.cpu.affinity = "all"
sched.cpu.latencySensitivity = "normal"
ethernet0.generatedAddress = "00:0c:29:8a:e0:fa"
vmci0.id = "-1349852934"
tools.upgrade.policy = "manual"
svga.vramSize = "8388608"
cleanShutdown = "FALSE"
extendedConfigFile = "ACTIVE DIRECTORY 2008 R2.vmxf"
tools.syncTime = "FALSE"
nvram = "ACTIVE DIRECTORY 2008 R2.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
hpet0.present = "TRUE"
cpuid.coresPerSocket = "2"
disk.EnableUUID = "TRUE"
numa.autosize.vcpu.maxPerVirtualNode = "4"
numa.autosize.cookie = "40001"
sched.swap.derivedName = "/vmfs/volumes/603a59bb-5476d2af-5742-d89d672b5ccc/Pishrafteh/DC/ACTIVE DIRECTORY 2008 R2-806e4e1c.vswp"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 67 23 ff 1f 30"
ethernet0.generatedAddressOffset = "0"
monitor.phys_bits_used = "43"
vmotion.checkpointFBSize = "8388608"
vmotion.checkpointSVGAPrimarySize = "8388608"
softPowerOff = "FALSE"
toolsInstallManager.lastInstallError = "0"
toolsInstallManager.updateCounter = "2"
svga.guestBackedPrimaryAware = "TRUE"
migrate.hostLog = "./ACTIVE DIRECTORY 2008 R2-806e4e1c.hlog"
scsi0:0.redo = ""
toolsInstall.origFileName = "/vmfs/devices/cdrom/mpx.vmhba1:C0:T2:L0"
toolsInstall.origType = "atapi-cdrom"
ide1:0.startConnected = "FALSE"
vmxnet3.serialNumberV2 = "FALSE"

 

also attached latest vmware log files

any help would be appreciated.

 

any help would be appreciated

[MohammadRavaghi]

here is log files attached

[a_p_]

The logs show that the VM was previously running on ESXi 6.5, but you are seeing the issue on ESXi 7.

Did you copy/restore the VM with the ESXi upgrade from a VMFS-5 to VMFS-6 datastore.?
Please remember the VMFSsparse (delta) files are not supported on VMFS-6 datastores.

André

[MohammadRavaghi]

I found there is no difference between this two version of ESXI in practice. I tried both 6.5 and 7

how ever i'm trying to find out what to do for now . I have ESXI 7. tell me what can i do ?

[a_p_]

It's not about the ESXi version, but the file system version, i.e. VMFS-5 vs. VMFS-6.

André

[MohammadRavaghi]

I'll try to reformat an empty Datastore using VMFS5 and try resorting Disks and Snapshots.

thanks in advance for the time you taken on my case :))

 

[MohammadRavaghi]

Thanks bro i could resolve the main issue by chaning the VMFS to version 5

however i have some OS level issues that i am working on.

for another VM i have this issue :

 

[root@PHY01:/vmfs/volumes/608a9704-951bc9d7-7eb0-d89d672b5ccc/ARINA] vmkfstools -e ARINA-000002.vmdk
Failed to open disk link /vmfs/volumes/608a9704-951bc9d7-7eb0-d89d672b5ccc/ARINA/ARINA-000002.vmdk :Input/output error (327689)Disk chain is not consistent : Input/output error (327689)

 

may you describe it a bit more ?

and what to do against ?

[a_p_]

I can't really help you with I/O errors, as these are usually related to hardware issues. Is this a VM on the failed RAID set?

André

[MohammadRavaghi]

yes

it's been failed on a Raid 5 and recovered by PC-3000 Raid recovery in a clean room. most of the files are working properly. like logs or VMDK files. this snapshot file is the first file in two VM folder which is reporting I/O error. is there any way to check the delta file integrity itself !

I tested file using vmkfstools and here is the result ! :

[root@PHY01:/vmfs/volumes/608a9704-951bc9d7-7eb0-d89d672b5ccc/ARINA] vmkfstools -x check ARINA-000002.vmdk
Disk is error free

 

also Base disk has same situation ! :

[root@PHY01:/vmfs/volumes/608a9704-951bc9d7-7eb0-d89d672b5ccc/ARINA] vmkfstools -x check ARINA.vmdk
Disk is error free

 

and BTW Data delivered on an external hard and i imported files into datastores. so there shouldn't be any other Hardware or disk errors because the old server is gone and we replaced it with a new one.

i'm a bit confused at this moment

If anyone can help me it would be appreciated (

 

[a_p_]

This indeed confuses me too. Two things that come in my mind:

• please provide a complete list of the VM's files that you've received (i.e. the output of ls -lisa), and attach the .vmdk descriptor files (the small text files without flat/delta/sesparse in their names) to your next reply
• restore the recovered files to a PC, and try to open the VM in VMware Workstation

André

[MohammadRavaghi]

Hi again

[root@PHY01:/vmfs/volumes/608a9704-951bc9d7-7eb0-d89d672b5ccc/ARINA] ls -lisa
total 269314072
104891204 8 drwxr-xr-x 1 root root 3500 Apr 30 17:29 .
4 1024 drwxr-xr-t 1 root root 2380 Apr 30 16:57 ..
436241220 51168256 -rw-r--r-- 1 root root 52395659264 Feb 20 2019 ARINA-000002-delta.vmdk
381715268 0 -rw-r--r-- 1 root root 314 Apr 30 17:12 ARINA-000002.vmdk
188777284 0 -rw-r--r-- 1 root root 314 Apr 30 16:23 ARINA-000002.vmdk.old
192971588 4096000 -rw-r--r-- 1 root root 4194304000 Apr 29 18:09 ARINA-Snapshot5.vmem
125862724 2048 -rw-r--r-- 1 root root 1206899 Apr 29 17:55 ARINA-Snapshot5.vmsn
121668420 0 -rw-r--r-- 1 root root 51 Apr 29 17:55 ARINA-ac7e0ee8.hlog
163611460 4096000 -rw-r--r-- 1 root root 4194304000 Apr 30 07:00 ARINA-ac7e0ee8.vswp
255886148 209715200 -rw-r--r-- 1 root root 214748364800 Apr 30 15:06 ARINA-flat.vmdk
138445636 1024 -rw-r--r-- 1 root root 8684 Apr 29 17:55 ARINA.nvram
377520964 0 -rw-r--r-- 1 root root 447 Apr 30 18:49 ARINA.vmdk
146834244 0 -rw-r--r-- 1 root root 387 Apr 29 17:55 ARINA.vmsd
142639940 8 -rw-r--r-- 1 root root 3271 Apr 30 15:57 ARINA.vmx
117474116 8 -rw-r--r-- 1 root root 3182 Apr 29 17:55 ARINA.vmxf
155222852 1024 -rw-r--r-- 1 root root 428714 Apr 29 17:56 vmware-10.log
151028548 1024 -rw-r--r-- 1 root root 907541 Apr 29 17:55 vmware-11.log
130057028 1024 -rw-r--r-- 1 root root 410742 Apr 29 17:55 vmware-12.log
109085508 3072 -rw-r--r-- 1 root root 2819294 Apr 29 17:55 vmware-13.log
352355140 1024 -rw-r--r-- 1 root root 47108 Apr 30 15:57 vmware-14.log
172000068 1024 -rw-r--r-- 1 root root 320016 Apr 29 17:57 vmware-9.log
369132356 1024 -rw-r--r-- 1 root root 45871 Apr 30 16:00 vmware.log
167805764 112640 -rw-r--r-- 1 root root 115343360 Apr 29 17:56 vmx-ARINA-2893942504-1.vswp
184582980 112640 -rw-r--r-- 1 root root 115343360 Apr 29 17:57 vmx-ARINA-2893942504-2.vswp
339772228 0 -rwxr-xr-x 1 root root 279 Apr 30 15:55 vmxrebuild.sh

 

 

and here is descriptor files as i cannot attach more than a single file :

 

ARINA.vmdk content :

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 419430400 VMFS "ARINA-flat.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "26108"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "b2f82cb3d3232c212cc4d3fdfffffffe"
ddb.uuid = "60 00 C2 9c b6 9b 23 d9-04 0f 1f c3 99 d2 03 e3"
ddb.virtualHWVersion = "13"
~
~

ARINA-000002.vmdk

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fe9c0f85
parentCID=fffffffe
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="ARINA.vmdk"
# Extent description
RW 419430400 VMFSSPARSE "ARINA-000002-delta.vmdk"

# The Disk Data Base
#DDB

ddb.longContentID = "b2f82cb3d3232c212cc4d3fdfffffffe"
~
~

[a_p_]

I can't see anything wrong in the file listing as well as in the descriptor files, except for the "ddb.longContentID" in "ARINA-000002.vmdk". It looks like the descriptor files have been recreated manually rather than restored, but that should not be the cause for the issue.

Did you have a chance to try and open a copy of the VM in VMware Workstation yet?

André

 

[MohammadRavaghi]

yes i have the chance

but what is the difference ?

i will try and let u know the result asap

 

[MohammadRavaghi]

no chance at all

power on task drops without giving back any response !

i used vmware workstation latest version

[a_p_]

No response at all? Is a new vmware.log file created?

What you want to verify is that the VM's configuration looks ok in VMware Workstation. You may need to configure e.g. networking because it differs from ESXi. If this doesn't help either, then please attach the recovered .vmx file to a reply post.

André

[MohammadRavaghi]

yes i moved files after a raid failure and i tried to recover data from raid. i checked vmdk integrity and i'm sure data is healthy itself.