Cannot start VMs after upgrade to 7.0 - "The virtual machine feature mask incorrectly requires 'cpuid.stibp' = '1'."

[spoovy]

I upgraded my lab hosts from 6.7 to 7.0.  It all seemed to go smoothly, but I then noticed that none of them will power on.  The error message is:

• The virtual machine feature mask incorrectly requires 'cpuid.stibp' = '1'.
• The virtual machine feature mask incorrectly requires 'cpuid.fcmd' = '1'.
• The virtual machine feature mask incorrectly requires 'cpuid.ssbd' = '1'.
• The virtual machine feature mask incorrectly requires 'cpuid.ibrs' = '1'.
• The virtual machine feature mask incorrectly requires 'cpuid.ibpb' = '1'.
• Module 'FeatureCompatLate' power on failed.
• Failed to start the virtual machine.

After reading an article where someone had a similar problem trying to vmotion machines, I tried adding the following lines to the vmx files:

featMask.vm.cpuid.stibp = "Max:0"
featMask.vm.cpuid.ibrs = "Max:0"
featMask.vm.cpuid.ibpb = "Max:0"
featMask.vm.cpuid.fcmd = "Max:0"
featMask.vm.cpuid.ssbd = "Max:0"

Then reloading the VM (vim-cmd vmsvc/reload <id>) and trying again, but to no effect.

Any help gratefully accepted.

[daphnissov]

What did you do to these VMs or your lab to cause this? What type of CPUs are in your hosts? What OS type are these VMs?

[spoovy]

IWhat did you do to these VMs or your lab to cause this?

- As in the original post, I upgraded the hosts from 6.7 to 7.0.   No errors during upgrade. Both show they are now running 7.0.

What type of CPUs are in your hosts?

- both 7th gen intel core.  one is an i3, on is i5.

What OS type are these VMs?

A mix of various Linux OSes, Windows 2016, 2019. All fail.

[daphnissov]

Ok, so you are, in fact, running unsupported CPUs.

[IRIX201110141]

All of these extensions are related to Spectre/Meltdown remediation. CPU Microcode patches missing?

Regards,
Joerg

[spoovy]

Worked fine with 6.7.  I may have applied some ESXi patches at the time spectre appeared which updated the microcode, I can't remember, but if that were the case then presumably these patches would have been included in the base esxi 7.0 image no?

I'll see if the vendor has bios updates available. They're Intel NUCs, 7th gen I think.

[a_p_]

Do you have EVC enabled on the NUC cluster? If yes, it may be worth disabling (and later re-applying) it to see whether this helps.

André

[spoovy]

Unfortunately the vcenter box is one of the VMs that I can't boot.

[a_p_]

Does the VM's configuration (.vmx) file contain CPU feature settings?

André

[spoovy]

Apart from the ones I tried as set out in my first post, only these settings related to cpu:

# grep -i cpu DNS1.vmx
sched.cpu.units = "mhz"
sched.cpu.latencySensitivity = "normal"
numa.autosize.vcpu.maxPerVirtualNode = "1"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
featMask.vm.cpuid.stibp = "Max:0"

Regarding the flags mentioned in my first post I have tried with them, without them, but no difference either way.

I realise these are not supported hardware boxes but others seem to have 7.0 working on them (see https://www.virten.net/2020/04/homelab-will-esxi-7-0-run-on-intel-nuc/  for example) and they are a very common lab box to use so is still of interest to many users I would think.

[IRIX201110141]

Can you please remove "featMask.vm.cpuid.stibp = "Max:0" completely? You have to start the VM from commandline rather than from the GUI otherwise unregister and register it again.

vim-cmd vmsvc/getallvms | grep DNS1

vim-cmd vmsvc/power.in <ID>

Regards,
Joerg

[spoovy]

Flashing BIOS on the hosts fixed it.   None of those cpu feature flags are required now.