Running ESX under KVM with VM-x/EPT

[daybyday]

I've been experimenting with setting up a test lab on a linux host with KVM as the layer 1 hypervisor and ESX as the layer 2. With KVM configured for nested, ept and ignore_msrs, and ESX set to vmx.allowNested and hv.assumeEnabled things work fine, except that VMs under ESX don't use EPT.

With some help I'm now trying to determine which of these flags ESX requires for EPT to function, e.g. to get HWMMU working:

2016-02-11T21:17:00.157Z| vmx| I120: VPID and EPT Capabilities (0x00000d0106114041)
2016-02-11T21:17:00.157Z| vmx| I120:   R=0/W=0/X=1                      yes
2016-02-11T21:17:00.157Z| vmx| I120:   Page-walk length 3               yes
2016-02-11T21:17:00.157Z| vmx| I120:   EPT memory type WB               yes
2016-02-11T21:17:00.157Z| vmx| I120:   2MB super-page                   yes
2016-02-11T21:17:00.157Z| vmx| I120:   1GB super-page                    no
2016-02-11T21:17:00.157Z| vmx| I120:   INVEPT support                   yes
2016-02-11T21:17:00.157Z| vmx| I120:   Access & Dirty Bits               no
2016-02-11T21:17:00.157Z| vmx| I120:   Type 1 INVEPT                    yes
2016-02-11T21:17:00.157Z| vmx| I120:   Type 2 INVEPT                    yes
2016-02-11T21:17:00.157Z| vmx| I120:   INVVPID support                  yes
2016-02-11T21:17:00.157Z| vmx| I120:   Type 0 INVVPID                   yes
2016-02-11T21:17:00.157Z| vmx| I120:   Type 1 INVVPID                    no
2016-02-11T21:17:00.157Z| vmx| I120:   Type 2 INVVPID                   yes
2016-02-11T21:17:00.157Z| vmx| I120:   Type 3 INVVPID                   yes
...
2016-02-11T21:17:00.158Z| vmx| I120: MONITOR MODE: allowed modes          : BT32 HV HWMMU
2016-02-11T21:17:00.158Z| vmx| I120: MONITOR MODE: user requested modes   : BT32 HV HWMMU
2016-02-11T21:17:00.158Z| vmx| I120: MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
2016-02-11T21:17:00.158Z| vmx| I120: MONITOR MODE: filtered list          : HWMMU BT32 HV
2016-02-11T21:17:00.158Z| vmx| I120: HV Settings: virtual exec = 'hardware'; virtual mmu = 'hardware'

So far I've been able to get VMs to use HWMMU with the above flags active in a patched host kernel.

Could anyone tell me which flags ESX actually requires and which are optional or preferable for EPT/HWMMU to work?

Additionally, are more or different flags required with "vhv.enable = TRUE"? So far I have only tested with "vhv.enabled = FALSE"

It's a big performance jump to have this working so it would be very much appreciated. Thanks!

[admin]

ESXi does not require INVVPID support to use EPT.  Everything else marked 'yes' from your log file snippet is required.

To enable virtualized HV, you need all of the features marked 'yes,' including the INVVPID capabilities.  (There are constraints on the other VMX capability MSRs as well.)  If you support all Nehalem capabilities, you should be fine.

[daybyday]

jmattson, thank you for the detailed response. I think we will be able to get EPT upstreamed, will have to investigate whether if vhv can be supported as well.

On vhv support, I'm running in host-passthrough mode on a i7-2600k which should do it natively I believe; the entire feature set is as follows:

2016-02-12T14:25:28.777Z| vmx| I120: Host VT-x Capabilities:
2016-02-12T14:25:28.777Z| vmx| I120: Basic VMX Information (0x0098100011e57ed0)
2016-02-12T14:25:28.777Z| vmx| I120:   VMCS revision ID               300252880
2016-02-12T14:25:28.777Z| vmx| I120:   VMCS region length              4096
2016-02-12T14:25:28.777Z| vmx| I120:   VMX physical-address width   natural
2016-02-12T14:25:28.777Z| vmx| I120:   SMM dual-monitor mode             no
2016-02-12T14:25:28.777Z| vmx| I120:   Advanced INS/OUTS info            no
2016-02-12T14:25:28.777Z| vmx| I120:   True VMX MSRs                    yes
2016-02-12T14:25:28.777Z| vmx| I120:   VMCS memory type                  WB
2016-02-12T14:25:28.777Z| vmx| I120: True Pin-Based VM-Execution Controls (0x0000007f00000016)
2016-02-12T14:25:28.777Z| vmx| I120:   External-interrupt exiting     {0,1}
2016-02-12T14:25:28.777Z| vmx| I120:   NMI exiting                    {0,1}
2016-02-12T14:25:28.777Z| vmx| I120:   Virtual NMIs                   {0,1}
2016-02-12T14:25:28.777Z| vmx| I120:   Activate VMX-preemption timer  {0,1}
2016-02-12T14:25:28.777Z| vmx| I120:   Process posted interrupts      { 0 }
2016-02-12T14:25:28.777Z| vmx| I120: True Primary Processor-Based VM-Execution Controls (0xfff9fffe04006172)
2016-02-12T14:25:28.777Z| vmx| I120:   Interrupt-window exiting       {0,1}
2016-02-12T14:25:28.777Z| vmx| I120:   Use TSC offsetting             {0,1}
2016-02-12T14:25:28.777Z| vmx| I120:   HLT exiting                    {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   INVLPG exiting                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   MWAIT exiting                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   RDPMC exiting                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   RDTSC exiting                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   CR3-load exiting               {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   CR3-store exiting              {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   CR8-load exiting               {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   CR8-store exiting              {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Use TPR shadow                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   NMI-window exiting             {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   MOV-DR exiting                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Unconditional I/O exiting      {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Use I/O bitmaps                {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Monitor trap flag              {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Use MSR bitmaps                {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   MONITOR exiting                {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   PAUSE exiting                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Activate secondary controls    {0,1}
2016-02-12T14:25:28.778Z| vmx| I120: Secondary Processor-Based VM-Execution Controls (0x000000fb00000000)
2016-02-12T14:25:28.778Z| vmx| I120:   Virtualize APIC accesses       {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Enable EPT                     {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Descriptor-table exiting       { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Enable RDTSCP                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Virtualize x2APIC mode         {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Enable VPID                    {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   WBINVD exiting                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Unrestricted guest             {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   APIC-register virtualization   { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Virtual-interrupt delivery     { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   PAUSE-loop exiting             { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   RDRAND exiting                 { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Enable INVPCID                 { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Enable VM Functions            { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Use VMCS shadowing             { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   RDSEED exiting                 { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   EPT-violation #VE              { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Enable XSAVES/XRSTORS          { 0 }
2016-02-12T14:25:28.778Z| vmx| I120: True VM-Exit Controls (0x007fefff00036dfb)
2016-02-12T14:25:28.778Z| vmx| I120:   Save debug controls            {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Host address-space size        {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Load IA32_PERF_GLOBAL_CTRL     { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Acknowledge interrupt on exit  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Save IA32_PAT                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Load IA32_PAT                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Save IA32_EFER                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Load IA32_EFER                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Save VMX-preemption timer      {0,1}
2016-02-12T14:25:28.778Z| vmx| I120: True VM-Entry Controls (0x0000d3ff000011fb)
2016-02-12T14:25:28.778Z| vmx| I120:   Load debug controls            {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   IA-32e mode guest              {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Entry to SMM                   { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Deactivate dual-monitor mode   { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Load IA32_PERF_GLOBAL_CTRL     { 0 }
2016-02-12T14:25:28.778Z| vmx| I120:   Load IA32_PAT                  {0,1}
2016-02-12T14:25:28.778Z| vmx| I120:   Load IA32_EFER                 {0,1}
2016-02-12T14:25:28.778Z| vmx| I120: VPID and EPT Capabilities (0x00000d0106114041)
2016-02-12T14:25:28.778Z| vmx| I120:   R=0/W=0/X=1                      yes
2016-02-12T14:25:28.778Z| vmx| I120:   Page-walk length 3               yes
2016-02-12T14:25:28.778Z| vmx| I120:   EPT memory type WB               yes
2016-02-12T14:25:28.778Z| vmx| I120:   2MB super-page                   yes
2016-02-12T14:25:28.778Z| vmx| I120:   1GB super-page                    no
2016-02-12T14:25:28.778Z| vmx| I120:   INVEPT support                   yes
2016-02-12T14:25:28.778Z| vmx| I120:   Access & Dirty Bits               no
2016-02-12T14:25:28.778Z| vmx| I120:   Type 1 INVEPT                    yes
2016-02-12T14:25:28.778Z| vmx| I120:   Type 2 INVEPT                    yes
2016-02-12T14:25:28.778Z| vmx| I120:   INVVPID support                  yes
2016-02-12T14:25:28.778Z| vmx| I120:   Type 0 INVVPID                   yes
2016-02-12T14:25:28.778Z| vmx| I120:   Type 1 INVVPID                    no
2016-02-12T14:25:28.778Z| vmx| I120:   Type 2 INVVPID                   yes
2016-02-12T14:25:28.778Z| vmx| I120:   Type 3 INVVPID                   yes
2016-02-12T14:25:28.778Z| vmx| I120: Miscellaneous VMX Data (0x0000000000000065)
2016-02-12T14:25:28.778Z| vmx| I120:   TSC to preemption timer ratio      5
2016-02-12T14:25:28.778Z| vmx| I120:   VM-Exit saves EFER.LMA           yes
2016-02-12T14:25:28.778Z| vmx| I120:   Activity State HLT               yes
2016-02-12T14:25:28.778Z| vmx| I120:   Activity State shutdown           no
2016-02-12T14:25:28.778Z| vmx| I120:   Activity State wait-for-SIPI      no
2016-02-12T14:25:28.778Z| vmx| I120:   CR3 targets supported              0
2016-02-12T14:25:28.778Z| vmx| I120:   Maximum MSR list size            512
2016-02-12T14:25:28.778Z| vmx| I120:   Allow all VMWRITEs                no
2016-02-12T14:25:28.778Z| vmx| I120:   MSEG revision ID                   0
2016-02-12T14:25:28.778Z| vmx| I120: VMX-Fixed Bits in CR0 (0x0000000080000021/0xffffffffffffffff)
2016-02-12T14:25:28.778Z| vmx| I120:   Fixed to 0        0x0000000000000000
2016-02-12T14:25:28.778Z| vmx| I120:   Fixed to 1        0x0000000080000021
2016-02-12T14:25:28.778Z| vmx| I120:   Variable          0xffffffff7fffffde
2016-02-12T14:25:28.778Z| vmx| I120: VMX-Fixed Bits in CR4 (0x0000000000002000/0xffffffffffffffff)
2016-02-12T14:25:28.778Z| vmx| I120:   Fixed to 0        0x0000000000000000
2016-02-12T14:25:28.778Z| vmx| I120:   Fixed to 1        0x0000000000002000
2016-02-12T14:25:28.778Z| vmx| I120:   Variable          0xffffffffffffdfff
2016-02-12T14:25:28.778Z| vmx| I120: VMCS Enumeration (0x000000000000002e)
2016-02-12T14:25:28.778Z| vmx| I120:   Highest index                   0x17
2016-02-12T14:25:28.778Z| vmx| I120: VM Functions (0x0000000000000000)
2016-02-12T14:25:28.778Z| vmx| I120: hostCpuFeatures = 0x87d
2016-02-12T14:25:28.778Z| vmx| I120: hostNumGenPerfCounters = 4
2016-02-12T14:25:28.778Z| vmx| I120: hostNumFixedPerfCounters = 3
2016-02-12T14:25:28.778Z| vmx| I120: hostPerfCtrArch = 2
2016-02-12T14:25:28.778Z| vmx| I120: CPU0: PMC: IA32, Nehalem-C or later PMC: bad VMENTRY_CTL_LOAD_CPGC or VMEXIT_CTL_LOAD_CPGC
2016-02-12T14:25:28.778Z| vmx| I120: PMC: smmFrz (hw): (1)
2016-02-12T14:25:28.778Z| vmx| I120:  [c:0 f:1 e:0]
2016-02-12T14:25:28.778Z| vmx| I120: CPU1: PMC: IA32, Nehalem-C or later PMC: bad VMENTRY_CTL_LOAD_CPGC or VMEXIT_CTL_LOAD_CPGC
2016-02-12T14:25:28.778Z| vmx| I120: PMC: smmFrz (hw): (1)
2016-02-12T14:25:28.778Z| vmx| I120:  [c:0 f:1 e:0]
2016-02-12T14:25:28.778Z| vmx| I120: MONITOR MODE: allowed modes          : BT32 HV HWMMU
2016-02-12T14:25:28.778Z| vmx| I120: MONITOR MODE: user requested modes   : BT32 HV HWMMU
2016-02-12T14:25:28.778Z| vmx| I120: MONITOR MODE: guestOS preferred modes: HWMMU HV BT32
2016-02-12T14:25:28.778Z| vmx| I120: MONITOR MODE: filtered list          : HWMMU HV BT32
2016-02-12T14:25:28.778Z| vmx| I120: HV Settings: virtual exec = 'hardware'; virtual mmu = 'hardware'
...
2016-02-12T14:25:28.884Z| vmx| I120: [msg.cpuid.noVHVQuestion] Virtualized Intel VT-x/EPT is not supported on this platform.
2016-02-12T14:25:28.884Z| vmx| I120+ Continue without virtualized Intel VT-x/EPT?

Can you tell based on this whether if Type 1 INVVPID is the only missing feature for allowing vhv?

Thanks!

[admin]

It looks like you are missing the following features:

Advanced INS/OUTS info is required.

At least 4 CR3 targets must be supported.

All four INVVPID variants must be supported.

Descriptor-table exiting must be supported.

[knightxii]

Hi daybyday,

I'm also attempting the exact setup that you're doing. nested ESXI inside KVM (on Ubuntu host).  Just wondering if you were able to get the issue resolved yet?

My setup is with the latest Skylake i7-6700, on Ubuntu 16.04, qemu 2.5. BIOS has vt-x enabled. options kvm-intel ignore_msrs, ept, nested all set to "y" inside /etc/modprob.d/kvm_intel.conf.  I tried to do cpu host-passthrough via virt-manager but nothing seem to budge. Installation was flawless, no Hardware Virtualization error.

It's only when I make a VM that it keep saying  VT-x/EPT error everytime I try to make a VM thus, it proves that Nested ESXI inside that ESXi inside KVM is not possible.

Additionally when I check https://esxiIPAddress/mob/?moid=ha-host&doPath=capability, the nestedHVSupported = false. Is this the same case for you?

1) Were you able to resolve the EPT issue? What is your KVM CPU mode/feature/extension settings are like? I'm assuming you're using i7-2600k it must be very similar to the new i7-6700.

2) Which command did you use to list all that VT-x capabilities flag?

I've struggled with this issue for months with no solution insight. Any help from ESXi experts would be greatly appreciated.

[gshearer]

Hi All,

  I've been tinkering with this in my own virtual setup. Could use some help.   I'm using ESXi 6.5 and vCenter 6.5.  I'm attempting to build a full HCI lab inside my workstation which is:

Intel Core i7-6800K with 128gigs of ram and plenty of very fast raid-0 SSD flash to work with.

What's working: Basic vSphere functionality and vCenter and believe it or not vSan (though with a bunch of complaints about HCI of course)

What's not working: hardware-accelerated VM's -- Everytime I start a VM inside vSphere I get that annoying pop-up question about no support for hardware virtualization -- do you want to continue yes/no ?  -- This includes when vCenter starts.

I'm adding this to /etc/vmware/config on ALL of my virtualized ESXi hosts:

vhv.allow = "TRUE"

vhv.enable = "TRUE"

hv.assumeEnabled = "TRUE"

vmx.allowNested = "TRUE"

Here's my CURRENT QEMU/KVM command-line:

Where X is the host I'm booting:

I'd like to get hardware-accelerated VM's working and especially vMotion. Any suggestions as to what to use for KVM's machine/cpu type? I've tried most of them, but at the moment i'm simply using -cpu host (which exposes all of my Core i7-6800k to the guest)

Thanks in advance! :smileyhappy:

[gshearer]

Update: Removing vhv.enable = "TRUE" from my list of things added to /etc/vmware/config has fixed the VMX pop-up question issue. :smileyhappy:

Now to get EVC working.. If anyone has any suggestions for KVM / CPU model / Flags to add to the command line for this, I'd greatly appreciate it.

-G

[knightxii]

gshearer, are you able to verify that when you don't get anymore error that the VM you created are 64 bits and that It can be used as a Nested Virtualized Host? ie: can you spawn VM from that VM (in this case could be a nested vritualized ESXi 6 or 6.5). I'm just curious for my case as that's what I'm trying to test. This way you can have a virtualized cluster all under one box to test all the functionality.

KVM ---> virtual ESXi ---> nested virtual ESXi VM (64 bit with EPT) ---> VM

[gshearer]

I have not tested ALL functionality but this is working:

Core i7-6800K --> Linux kernel 4.9.8 x86_64 (ArchLinux) --> QEMU/KVM 2.8.0 --> ESXi 6.5 build 4564106 (guest) --->  ArchLinux Kernel 4.9.8 x86_64 (guest)

I'm still trying to find a stable QEMU emulator for storage backend, so far megasas2-gen2 has shown the most promise.

[gshearer]

More misery to report.

I have tried every single QEMU network device with ESXI 6.5 with the following results:

e1000-82544gc -- Works with MTU 9000 but I can only get one to work per PCI bus. I'm currently using this as my solution with a vmxnet3 as my management interface. I'd like to have 5 more for tinkering.

e1000-82545em -- Recognized but can't get any packets to flow even with MTU 1500

e1000e -- Pings work with MTU 1500 but random "Rx receiver hangs" in vmkernel.log and TCP doesn't work (packetloss)

all remaining intel based vnics (i825xxx) -- not recognized

rtl8139 -- no jumbo frames

vmxnet3 --- works great but only supports MTU's of 3058 -- yes I tested all the way up to the exact byte. Ping -s 3030 works.. -s 3031 does not. This explains why vSan complains about MTU.

ATTENTION GOOD PEOPLE AT VMWARE: Could you PLEASE add a support for KVM virtio-scsi-pci and virtio-net-pci ??  This would enable people to build labs to learn without buying expensive equipment -- also may prevent divorce from the noise they make.

In my case, I was hoping to build a completely virtual VMware HCI lab on my workstation -- using megasas-gen2, which (in theory) could translate SCSI UNMAP (overlay) to SATA-TRIM (underlay) --- allowing me to use cheap-o SSD's as well.

Note: This is as of QEMU-2.8.0 and Linux 4.9.8

[spigotadmin]

May I ask if you were ever able to successfully run ESXi nested on a KVM host? 

I've been beating my head against the wall for hours.  I cannot enable the option "Expose hardware assisted virtualization to the guest OS" on guest VMs in ESXi - I always get the "VT-x/EPT not available on this host" pop-up.  My CPUs support all needed extensions and they are enabled in kvm_intel.

Thanks for any info.

[hawken]

Hi,

I've been having problems running esxi under kvm too.

When it comes to exposing virtualization capability to the guest, you will need this in /etc/modprobe.d:

options kvm_intel nested=1

There are some more related options one may try, not that it has solved my problems:
options kvm_intel nested=1 enable_apicv=0
options kvm ignore_msrs=1

[spigotadmin]

I was never really able to get ESXi nested under KVM with 6.5 or 6.7.  I was able to get it functional under 6.0.  To clarify - all versions will install and run, but VMs under 6.5 or 6.7 are unusably slow and reasonably usable under 6.0.  As I mentioned, I was never able to get the option "Expose hardware assisted virtualization to the guest OS" on guest VMs in ESXi - I always get the "VT-x/EPT not available on this host" pop-up.

I spent many hours trying the various config changes you mentioned. 

[gshearer]

CPU FLAG HELP

==============

Bare metal: Intel i7-6800K (Broadwell)

CPU flags detected by linux (Kernel 4.9.7) from /proc/cpuinfo

fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts

ALL of those flags are exposed to QEMU/KVM guests with the "-cpu host" option with the following exceptions:

dts acpi tm pbe pebs bts nonstop_tsc aperfmperf dtes64 monitor ds_cpl est tm2 xtpr pdcm dca epb intel_pt cqm cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida pln pts

And I looked those up from arch/x86/include/asm/cpufeatures.h:

#define X86_FEATURE_DS          ( 0*32+21) /* "dts" Debug Store */

#define X86_FEATURE_ACPI        ( 0*32+22) /* ACPI via MSR */

#define X86_FEATURE_ACC         ( 0*32+29) /* "tm" Automatic clock control */

#define X86_FEATURE_PBE         ( 0*32+31) /* Pending Break Enable */

#define X86_FEATURE_PEBS        ( 3*32+12) /* Precise-Event Based Sampling */

#define X86_FEATURE_BTS         ( 3*32+13) /* Branch Trace Store */

#define X86_FEATURE_NONSTOP_TSC ( 3*32+24) /* TSC does not stop in C states */

#define X86_FEATURE_APERFMPERF  ( 3*32+28) /* APERFMPERF */

#define X86_FEATURE_DTES64      ( 4*32+ 2) /* 64-bit Debug Store */

#define X86_FEATURE_MWAIT       ( 4*32+ 3) /* "monitor" Monitor/Mwait support */

#define X86_FEATURE_DSCPL       ( 4*32+ 4) /* "ds_cpl" CPL Qual. Debug Store */

#define X86_FEATURE_EST         ( 4*32+ 7) /* Enhanced SpeedStep */

#define X86_FEATURE_TM2         ( 4*32+ 8) /* Thermal Monitor 2 */

#define X86_FEATURE_XTPR        ( 4*32+14) /* Send Task Priority Messages */

#define X86_FEATURE_PDCM        ( 4*32+15) /* Performance Capabilities */

#define X86_FEATURE_DCA         ( 4*32+18) /* Direct Cache Access */

#define X86_FEATURE_EPB         ( 7*32+ 3) /* IA32_ENERGY_PERF_BIAS support */

#define X86_FEATURE_INTEL_PT    ( 7*32+15) /* Intel Processor Trace */

#define X86_FEATURE_CQM         ( 9*32+12) /* Cache QoS Monitoring */

#define X86_FEATURE_CQM_LLC     (11*32+ 1) /* LLC QoS if 1 */

#define X86_FEATURE_CQM_OCCUP_LLC (12*32+ 0) /* LLC occupancy monitoring if 1 */

#define X86_FEATURE_CQM_MBM_TOTAL (12*32+ 1) /* LLC Total MBM monitoring */

#define X86_FEATURE_CQM_MBM_LOCAL (12*32+ 2) /* LLC Local MBM monitoring */

#define X86_FEATURE_DTHERM      (14*32+ 0) /* Digital Thermal Sensor */

#define X86_FEATURE_IDA         (14*32+ 1) /* Intel Dynamic Acceleration */

#define X86_FEATURE_PLN         (14*32+ 4) /* Intel Power Limit Notification */

#define X86_FEATURE_PTS         (14*32+ 6) /* Intel Package Thermal Status */

I'm hoping someone can tell me which of these missing flags vSphere 6.5 requires to enable EVC ?   Note that I have tried most all supported Intel EVC architectures, vCenter still complains with this message:

The host's CPU hardware should support the cluster's current Enhanced vMotion Compatibility mode, but some of the necessary CPU features are missing from the host.

Appreciate your time!!