VM Does not start after ESXi has been replaced with version 6

[Stendec75]

Hi All,

I have run in to a scary problem after replacing my VMware ESXi version 5.1 to version 6.

My server has an Areca RAID controller that is by default supported by version 6. So, after installing (clean install ) ESXi 6 on a internal USB drive, everthing look good. It shows my Datastore stored on my RAID controller and I can see the content,  (see attachment, image of my datastore) including my VM-SBS2011.vmdk file.

I’m not a highly skilled VM pro, but the .vmdk file (about 40GB) should contain my complete Windows environment (Windows 2011 server).

Having access to my datastore on my RAID controller and access to the original .vmx file, everything seems to be fine. Right click the .vmx, add to Inventory and I’m ready to go…but….it won’t start.

It shows an Out of Resource error, with not that much additional information. My feeling is that there is something wrong with Swap location, because it looks like the swap location is located outside de VM (.vmdk) storage location. This is what my .vmx looks like:

My .vmdk is not stored on "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/ ! It points to a different location.

(VMX file also uploaded with this post)

No I have two question. What ca be the cause of this issue, not able to start the VM?

Second question. Could a solution be: Delete all files in my datastore, except the VM-SBS2011-flat.vmdk file, create a new virtual machine on my server, using an existing disk (pointing to my VM-SBS2011-flat.vmdk), will I be able to start my VM? Will it create a fresh (working) .vmx file? There where no snapshots exist when I shut my Windows OS down the last time. Will I lose any data? Please help!

At this moment I’m downloading the complete content (all files) from the datastore to my local computer as an backup. I hope my .vmdk file will be enough.

I'm really thankfull for your help!

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "8"
pciBridge0.present = "true"
pciBridge4.present = "true"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "true"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "true"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "true"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "true"
hpet0.present = "true"
nvram = "VM-SBS2011.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
displayName = "VM-SBS2011"
extendedConfigFile = "VM-SBS2011.vmxf"
floppy0.present = "true"
numvcpus = "4"
cpuid.coresPerSocket = "4"
scsi0.present = "true"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsisas1068"
memsize = "12288"
scsi0:0.present = "true"
scsi0:0.fileName = "VM-SBS2011.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "true"
ide1:0.clientDevice = "TRUE"
ide1:0.deviceType = "atapi-cdrom"
ide1:0.startConnected = "TRUE"
floppy0.startConnected = "false"
floppy0.fileName = ""
floppy0.clientDevice = "true"
ethernet0.present = "true"
ethernet0.virtualDev = "e1000e"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
chipset.onlineStandby = "false"
disk.EnableUUID = "true"
guestOS = "windows8srv-64"
uuid.location = "56 4d fe 63 9b 1a 6f c6-ba 0b 00 e8 3e 7a e8 80"
uuid.bios = "56 4d fe 63 9b 1a 6f c6-ba 0b 00 e8 3e 7a e8 80"
vc.uuid = "52 2d 6e 21 82 34 c3 04-c8 18 19 67 f8 67 4d a5"
snapshot.action = "keep"
sched.cpu.min = "0"
sched.cpu.units = "mhz"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.shares = "normal"
scsi0.pciSlotNumber = "160"
ethernet0.generatedAddress = "00:0c:29:7a:e8:80"
ethernet0.pciSlotNumber = "192"
vmci0.id = "1048242304"
vmci0.pciSlotNumber = "32"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/VM-SBS2011-snapshots/VM-SBS2011-6cbed3a5.vswp"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.sasWWID = "50 05 05 63 9b 1a 6f c0"
ethernet0.generatedAddressOffset = "0"
vm.genid = "5456487432926317901"
vm.genidX = "901288717606574007"
hostCPUID.0 = "0000000d756e65476c65746e49656e69"
hostCPUID.1 = "000206d70020080017bee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000d756e65476c65746e49656e69"
guestCPUID.1 = "000206d700040800969822031fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000d756e65476c65746e49656e69"
userCPUID.1 = "000206d700200800169822031fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
vmotion.checkpointFBSize = "4194304"
softPowerOff = "TRUE"
tools.syncTime = "FALSE"
bios.forceSetupOnce = "false"
ide1:0.fileName = ""
bios.bootDelay = "5000"
snapshot.redoNotWithParent = "True"
unity.wasCapable = "FALSE"
tools.remindInstall = "TRUE"
workingDir = "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/VM-SBS2011-snapshots"
fileSearchPath = "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/VM-SBS2011-snapshots;."
checkpoint.vmState.readOnly = "FALSE"
checkpoint.vmState = ""
scsi0:1.present = "TRUE"
scsi0:1.deviceType = "scsi-passthru"
scsi0:1.fileName = "/vmfs/devices/genscsi/naa.5000878000000000"
scsi0:1.allowGuestConnectionControl = "FALSE"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.mode = "persistent"
scsi0:2.present = "FALSE"
sched.swap.hostLocal = "disabled"

[Stendec75]

I have added a picture of what my datastore looks like!

[continuum]

Unregister the VM.
Doubleclick the vmx-file in WinSCP and delete this lines:

sched.swap.derivedName = "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/VM-SBS2011-snapshots/VM-SBS2011-6cbed3a5.vswp"
workingDir = "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/VM-SBS2011-snapshots"
fileSearchPath = "/vmfs/volumes/512b5271-cb39b672-c198-001e676870c7/VM-SBS2011-snapshots;."
scsi0:1.present = "TRUE"
scsi0:1.deviceType = "scsi-passthru"
scsi0:1.fileName = "/vmfs/devices/genscsi/naa.5000878000000000"
Readd VM to inventory and readd the scsi-passthru device.

[Stendec75]

So if I understand correct:

1. Unregister = Delete from Inventory? (without deleting datastore content ofcourse)

2. Then edit the .vmx (delete those lines).

3. Then add my VM to the Inventory again and run?

Your help is much appreciated !

[Stendec75]

Sorry, but it didn't work out...please help! Got 'Insuffient resource' error

See attachment

[a_p_]

How much physical memory do you have in your host? It may be worth reducing the VM's memory from 12 to 8GB, just to rule out memory issues.

If a vmware.log file is created in the VM's folder, then please attach it to a reply post.

André

[Stendec75]

Thanks you André! I attached the complete log-dir.

[Stendec75]

By the way, I changed my memory settings already. Without any result.

[a_p_]

The vmkernel.log contains multiple entries like this:

2016-02-20T17:27:05.383Z cpu2:35562)WARNING: Res3: 6892: 'VMHDD1': [rt 1] Cluster harvesting resulted in an unrecoverable error: Invalid metadata

2016-02-20T17:27:05.383Z cpu2:35562)WARNING: SwapExtend: 683: Failed to extend swap file from 0 KB to 65536 KB.

2016-02-20T17:27:05.383Z cpu2:35562)WARNING: SwapExtend: 686: Current swap file size is 0 KB.

2016-02-20T17:27:05.383Z cpu2:35562)WARNING: SwapExtend: vm 35562: 691: Failed to extend swap file type=regular from 0 KB to 65536 KB. status=Invalid metadata

2016-02-20T17:27:05.383Z cpu2:35562)WARNING: SwapExtend: vm 35562: 693: Current swap file size is 0 KB.

2016-02-20T17:27:05.385Z cpu2:35562)WARNING: DLX: 1054: Volume 5124937c-6bb795bc-2745-001e676870c7 ("VMHDD1") might be damaged on the disk. Corrupt lock detected at offset 2019724107704189184

2016-02-20T17:27:05.385Z cpu2:35562)WARNING: [type 1c078060 offset 2019724107704189184 v 2018940499510951936, hb offset 2022535696375365888

gen 16640, mode 16640, owner 00000004-4e8f846c-1fb3-001e676870c7 mtime 13214

I'm not a data recovery specialist like continuum, but it might be worth Checking Metadata Consistency with VOMA‌ 

Please ensure that you have an up-to-date backup of your VMs before running the command with the fix option!

André

[Stendec75]

Well...I made a copy of the complete folder (took hours) already. Will that be enough as backup?

[a_p_]

It depends on how you copied it. With the VM's virtual disk being thin provisioned, I'd suggest you export the VM to an OVF format (from the vSphere Client's File menu). In addition to this download the .vmx and .vmxf files from the datastore. This will allow you to import the VM again if necessary and restore it with the same MAC and UUID.

André

[Stendec75]

I have copied the VM-folder to a VMWare ESXi 5.5 datastore on another (older HP SAS) server en guess what...it starts! So for now I'm back in business. It seems like there is something wrong with the original server, because my .vmdk file doesn't seems to be damaged otherwise it would not start on my spare good old HP server..

I still wonder what the hell happend after I replaced the internal USB hypervisor drive and installed ESXi 6 on a fresh USB. I didn't touch the RAID array. Now, even when I boot the original server with it's original ESXi 5.1 USB drive (untouched), I got the same error when trying to boot the VM. Thats suggest there is something wrong with the RAID array but I can access the datastore on the RAID array and (after moving the content to another VM machine) all the files are just fine running smooth on the other server now. Shoot me.

Any ideas? 

[Stendec75]

Still very curious what is wrong with the original server. I ran the http://pubs.vmware.com/vsphere-60/topic/com.vmware.vsphere.troubleshooting.doc/GUID-6F991DB5-9AF0-4F9F-809C-B82D3EED7DAF.htmlMetadata Consistency check like André advised. This is the result:

~ # voma -m vmfs -f check -d /vmfs/devices/disks/eui.001b4d2083559996:1

Checking if device is actively used by other hosts

Running VMFS Checker version 1.0 in check mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'VMHDD1') with UUID:5124937c-6bb795bc-2745-001e676870c7, Version 5:58

ON-DISK ERROR: Invalid lock address 2019724107704189184

ON-DISK ERROR: lockAddr 7778304 Invalid lock type 470253664

ON-DISK ERROR: lockAddr 7778304 Invalid lock mode 16640

ON-DISK ERROR: Invalid lock :

         [type 1c078060 offset 2019724107704189184 v 2018940499510951936, hb offset 2022535696375365888

         gen 16640, mode 16640, owner 00000004-4e8f846c-1fb3-001e676870c7 mtime 13214

         num 0 gblnum 0 gblgen 0 gblbrk 0]

Phase 2: Checking VMFS heartbeat region

Phase 3: Checking all file descriptors.

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

Total Errors Found:           4

Qustion is, did this command fix anything? Because the command has parameter -f? Or do I need to run another command for the fix?

[a_p_]

I'm glad to see that the VM is up and running on the other host. Since this was the only VM on the datastore in question, I'd be curios whether voma can repair the metadata with the fix option.

André

[Stendec75]

I tried, but it won't allow me to fix....

[root@localhost:~] voma -m vmfs -f fix -d /vmfs/devices/disks/eui.001b4d2083559996:1

Checking if device is actively used by other hosts

Found 1 actively heartbeating hosts on device '/vmfs/devices/disks/eui.001b4d2083559996:1'

1): MAC address 00:1e:67:68:70:c7

I have no idea how to workaround en close the 'actively heartbeating' so the voma command can continue?

Ther isn't any VM running on the datastore. Unmounting should help (found with Google) but I'm unable to unmount te datastore. Options is grey out...

[a_p_]

Since this is a standalone host (please correct me if I'm wrong) all I can think of that causes the error message is the scratch location (.locker directory) that's on the VMFS volume. If you want you could try to temporarily set the scratch location back to its default /tmp/scratch, reboot the host and run the voma command again. If this still doesn't help, and there are no other VMs on this host, I'd suggest you do a complete reinstall with overwriting the VMFS partition.

André

[Stendec75]

Found it! The Intel S2600CP motherboard of the server is broken. Why I know for sure? I placed the ARECA ARC-1212 RAID controller with the original SAS disk into another server, installed VM ESXi, installed the ARECA .vib driver, added the vmx to the inventory and started the VM. Everthing runs smoothly! I'll start an Intel RMA procedure tommorow.

I would like to thank everyone for his or her support! Case closed.