Virtual machine powers off just after Hyper Data Protector backup has started

[thuinformatik]

Hi all,

I've a crazy behaviour on a HP ML 110 host on firmware level SSP 09.2020 running with 6.7.0 Update 3 (Build 17167734) - HPE Customized Image ESXi 6.7.0 Update 3 version 670.U3.10.6.3. On this hardware i host a Windows 20k16 machine. I thought i try out the Qnap Hyper Data Protector solution to make backups.

I've configured a backup job by using the defaults of the HDP. If i launch the backup job, following happens:

- A snapshot would be created.
- Nearly at the same time the virtual machine is powered off.
- I need to wait until the virtual machine can be powered on again, up to 20min. The error is related to "failed to lock the snapshot file (-0000001.vmdk)".
- If i restart the ESXi host itself and after the host has been restarted, then the virtual machine starts immediately.
- I can remove the snapshot and everything is back to normal.
- After the first login at windows after the machine is up and running again, i got asked by Windows what happend and i have to select a reason for the unexptected shutdown - so i guess this is not a normal shutdown then.
- In the windows eventlogs there is nothing useful, as if the power cord would be plugged off on a physical host.

Here are the vmware.log, vmkernel.log & hostd.log files.
https://www.netcult.ch/vmlogs_clean.txt 

Does anybody having an idea what happens here?

Best regards
Tom

[jburen]

Could it be related to HA? So the backup of the VM has lead the host to think there is an HA event. So the VM is stopped to be started on another host... From hostd.log: 

Task Created : haTask-1-vim.VirtualMachine.powerOff-1618547495

[thuinformatik]

Hey J

Thank you for your reply.

I had a similar idea, but not sure if this make sense on a standalone free ESXi installation without vCenter. I think a free hypervisor version does not support HA and/or vCenter is required, or i‘m wrong?

Best regards

Tom

[E_Espinel]

Hello

The VMware vSphere hypervisor (ESXi Free version) is the same download as the full-blown ESXi but stays unlocked only for 60 days. It is the exact same ISO image. If you don't enter the serial number for the free version, the product works as full version during 60 days. When the trial period ends and the product locks out the advanced features and becomes ESXi free (you'll still have to apply a serial number, which is available with the free version when you download it.

The ESXi 6.7 free limitations you will certainly want to make note of include the following:

• No vCenter integration or clustering
• No backups from third-party backup solutions
• Maximum of 8 vCPUs for VMs running on ESXi 6.7 free
• Maximum of 2 CPUs in your physical ESXi host
• Maximum RAM 12TB
• No VMware Support

attach a link with more details

https://www.virtualizationhowto.com/2019/11/esxi-6-7-free-limitations-and-features/

 

[thuinformatik]

Hi Enrique

Thanks for reply and the details. The system is based on the free license, this is the reason i ask the community and not the commercial support.

I read something about, that the CBT is a functionality of the vSphere API + VDDK API (vSphere Disk Development Kit). HDP has CBT enabled by default. I read something about that CBT queries run through VADP (vSphere APIs for Data Protection) to request for transmission of all altered blocks since the last backup. I read something about that the vSphere APIs for Data Protection requires a payed license.

Questions:
- Do you know if CBT-based backups are possible with the Essentials license (and without vCenter on a standalone host) instead?
- What license options are excatly required to be able to perform CBT-based backup? Is it just "Storage-APIs" or really "vSphere APIs for Data Protection"? 
I ask, because i read somewhere that the "vSphere APIs for Data Protection"-option is only available in Enterprise Standard, Enterprise and Enterprise Plus Edition.

Probable conclusion
Maybe i accidentially tried to make a CBT-based backup of a VM running on a free licensed host. Is it possible that in such a case the virtual server will be hardly powered off, if a thrid party product tries to perform a backup this way? Maybe that should not be the case, just ignoring the QueryChangedDiskAreas("*") would probably be the better option. Is this behaviour intentionally? I read somewhere, in case someone faces unresolved locks, that only go away after an amount of time or not, is a sign, that that some piece of application hangs and does not release the files, i'm not sure if this is some sort of crash that happens in the background. Maybe this only happens if someone performs such a 'unforseen' action. Idk.. Any ideas on this?

Best regards
Tom

[E_Espinel]

Hello.

VMware vstorage APIs are included with all paid licensed vSphere editions

You can check the different types of licenses in the following link

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsphere/vmware-vsphere-pricing-whitepaper.pdf

VMware vSphere Essentials Kit and Essentials Plus Kit Datasheet

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjE9ujW8ZLwAhVkU98KHe6UB-MQFjADegQIDRAD&url=https%3A%2F%2Fwww.vmware.com%2Fcontent%2Fdam%2Fdigitalmarketing%2Fvmware%2Fen%2Fpdf%2Fproducts%2Fvsphere%2Fvmware-vsphere-essentials-datasheet.pdf&usg=AOvVaw3_c-aYuTRbsYhK23ggMk3e

 

[thuinformatik]

Hey

Thank you for the links, unfortunately those documents do not answer my questions.

Beste regards
Tom

[jburen]

I have read https://kb.vmware.com/s/article/1020128 but nothing about the free ESXi. In another article (from Nakivo) they actually mention ESXi free: https://helpcenter.nakivo.com/display/KB/Enabling+CBT+for+a+VM 

If you want to be sure that your issue is caused by CBT you can see if disabling CBT on the VM resolves your problem.

To disable CBT:

1. Power off the virtual machine.
2. Right-click the virtual machine and click Edit Settings.
3. Click the Options tab.
4. Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
5. Set the ctkEnabled parameter to false for the desired SCSI disk(s).
6. Power on the virtual machine.

 

[thuinformatik]

Hi J

As info: With a fresh trial license the ESXi host can successfully be backed up by HDP. It seems that this issue is linked in some way to the Free ESXi license.

If i think about what happens here, i think it can be related to the HDP backup application. Following idea:

- the HDP app tries to send an API call to the ESXi host to perform the snapshot.
- This command can be executed by the ESXi host.
- then the HDP app tries to send an API (storage) call to the ESXi host to make a CBT based backup by using QueryChangedDiskAreas("*") 
- ESXi is ignoring this request because the APIs are not licensed.
- then the HDP finds out that no data is arriving and creates a haTask to hardly power off the host (we saw the task was created).
- HDP maybe waits until the vm would be restarted on another host - but on Free ESXi license no other host is available to take over and restart the machine
- After 12h or so, the HDP backup job expires and throws a timeout or similar.

I raised ticket a Qnap explaining the situation (Ticket ID: Q-202104-16382)

Hint: Somebody told me, that CBT based backups should be possible if not initated via remote host via API. Not sure about that because in the FAQs of the GhettoVCB script i read:

29Q: Can ghettoVCB support CBT (Change Block Tracking)?
29A: No, that is a functionality of the vSphere API + VDDK API (vSphere Disk Development Kit). You will need to look at paid solutions such as VMware vDR, Veeam Backup & Recovery, PHD Virtual Backups, etc. to leverage that functionailty.

 

Have a good week,
Br, Tom

[erik1]

Hi, Does anyone has a solution for this problem? I still got the same problem. I enabled ctkEnabled = True etc. but does not matter. It still powers off the virtual machine for backing up