unable to disable per-vm EVC

[xingchan]

Recently I upgraded a test vm the hardware version to 14. I noticed that EVC was enabled for this vm. When I shutdown the vm and looked at the per-vm EVC setting, EVC was disabled. When I started the VM, EVC was enabled again.

EVC is disabled on host and cluster level. I also looked at the vmx file: evcCompatibilityMode = "FALSE".

I'm running vCenter on build number 6.7.0.21000 and ESXi build 6.7.0, 10302608.

How can I disable EVC for this vm?

[RickVerstegen]

It is pretty easy to configure per-VM EVC from vSphere H5 client (flex client does not support).
Click on the VM >> Configure >> VMware EVC >>Edit >> Configure EVC mode.

VM must be powered off before configuring per-VM EVC

[xingchan]

Hi Rick,

that is the problem. When the vm is turned off the setting shows that EVC is disabled. When I turn the vm back on, EVC is enabled.

[RickVerstegen]

Can you check the EVC mode of the VM by using EVC Mode column?

In the vSphere Client, select a cluster or a host in the inventory. lick the VMs tab. Click the arrow in any column title, select Show/Hide Columns, and select the EVC Mode check box.

[xingchan]

It states Ivy Bridge Generation just like the other vm's. Other vm's EVC is not enabled.

[a_p_]

... that is the problem. When the vm is turned off the setting shows that EVC is disabled. When I turn the vm back on, EVC is enabled. ...

What's the physical CPU model that's installed in the host on which you start the VM?

Does the EVC mode that shows up after powering on the VM correspond to the physical CPU's generation.

André

[Frantisek Ferencik]

I have exactly the same behaviour. Any resolution for this?

[theclap]

Did anyone ever find a solution to this?

I'm hitting the same issue with a Win2k8R2 VM, HW version 15, on VMware ESXi, 6.7.0, 15160138

[cpetry7c1]

Was this ever resolved?  I'm having the same issue with 6.7g.  The cluster has EVC disabled, I power the VM off, it show's EVC disabled on the VM.  As soon as I power the VM back up, it shows EVC being enabled on the VM only.  I've tried going into the settings of the VM and resetting the CPUID under CPU/Advanced but that doesn't resolve the issue.

[a_p_]

If the VM's EVC level corresponds with the physical CPU on which the VM has been powered on, then I wouldn't consider this being an issue.

André

[holden777]

I have the same problem did you ever find a fix?

[wbishop_78]

Still waiting on a solution here.....

 

[Silvio Grotto]

I still have the same problem

[Nathan Prziborowski]

As per https://kb.vmware.com/s/article/59801?lang=en_US this is expected behavior.

The column for EVC mode (and also the per-VM EVC section, when the VM is powered on) is reporting the EVC mode the VM would need to be in (say if it migrated to an EVC cluster).

Although I think this information is very useful, the way it is represented in the UI definitely leads to confusion.

[Igubu]

This is unfortunately not true, or a bug exists in ESX 7 U2/Vcentre 7 U2. - I have a call open at the moment to troubleshoot - some information that might be useful for others:

In my case have the same issue, only problem is the detected EVC mode (per vm - only visible when the VM is powered on) is higher than physical supported EVC level of the host cpu's underneath! Our example: cluster with 4x hosts supporting Cascadelake (Xeon Platinum 8200 series) - no cluster EVC enabled, no per VM EVC either, and a LOT of vm's show " Icelake"  as the EVC supported mode when booted up.

Added 2x newer hosts that do support Icelake (Xeon Gold 6342) - created a new cluster for them, set Cluster EVC to Cascadelake, and unable to vmotion these "Icelake capable"  Vm's to it. So it seems the "supported"  EVC mode is identified incorrectly, and also the incorrect values are actually enforced and blocks cluster operations.

Hope this is resolved urgently, and more importantly without shutdowns of 1000+ production VM"s that are affected.

[lusos]

I have the same issue, did you find a solution ?

[Igubu]

Not yet nope - TBH have not experienced such poor service in 15+ years - not sure what's going on lately. Took 3 weeks of emails and waiting - no help - yesterday it was again transferred to a new engineer, (I was told previously a PR was done - 2 weeks ago!) - he's seen " something in the logs"  and busy with the PR request now - will update when I have something.

I did test this on completely new cluster with latest u3 patches hosts - same issue, so does seems to be an issue, not related to environments.

[mmasse]

I had (/have) what I assume is the same issue.

VM's powered off show "VMware EVC is Disabled"

Powering VM's on they change to EVC Mode "Intel® "Cascade Lake" Generation"

Was unable to migrate VM's between two dis-similar ESXi hosts, (higher feature CPU unable to migrate to lower feature CPU)

 

After a lot of troubleshooting I was able to get a workaround in place that allows my VM's to migrate between hosts.

 

System Versions

vSphere Server 7.0.3.00100

ESXi hosts - VMware ESXi, 7.0.2, 18538813

 

Workaround that worked for me, I am sure there may be a more optimal method, and some steps irrelevant.

Per VM from ESXi host (NOT from vCenter) 

State: VM Powered On

"Edit Settings" -> "VM Options" -> "Advanced" -> "Configuration Parameters" -> "Edit Configuration"

"Add Parameter"

      Key: evcCompatibilityMode

      Value: FALSE

Save/OK

(Tests at this point still show VM unable to migrate to lower featured ESXi CPU)

Power off VM.

(The next may be optional, it is possible a reboot is the only next step)

Unregister VM.

On lower ESXi host register VM

If using distributed virtual switch, you may need to log into the VCSA, edit the VM settings, and connect the network.

Power back on VM.

VM will still show "EVC Mode "Intel® "Cascade Lake" Generation"

But will now successfully migrate between ESXi hosts.

[Steven1973]

We have a environment with Intel Gold 6148 CPU's that has the same issue but we were not aware of it until we added two new hosts with newer Intel Silver 4316 CPUs.
We wanted to enable EVC on the 6148 cluster to be able to add the new hosts but enabling EVC was impossible.
After some investigation we found that about 50% of our VMs were running in 'EVC CPU Mode = Cascade Lake' on the old 6148 Skylake hosts.

 

This combination of physical hosts being Skylake and VMs running Cascade Lake, makes it impossible to enable EVC.

 

Enabling EVC in CPU Mode to Skylake comes with error "The host cannot be admitted to the cluster's current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode."

 

Enabling EVC in CPU Mode Cascade Lake comes with error "The host's CPU hardware does not support the cluster's current Enhanced vMotion Compatibility mode. The host CPU lacks features required by that mode."

 

Further investigation showed that vmx-version is causing this issue.
A new created VM with vmx-16 or earlier results in VMs running in 'EVC CPU Mode = Skylake' where new VMs with vmx version 17 or 18 results in VMs running in 'EVC CPU Mode = Cascade Lake'

 

[locco]

The problem is with the virtualHW.version.

per-VM EVC appears to be enabled from version 14 and after therefore you'll need to edit your vmx file and set the following in order to disable the EVC On power up.

virtualHW.version = "13"

 

[Igubu]

Hi All,

It's been a while - but good news is the very friendly and expert engineering teams from Vmware was looking at this over the last month or two in detail. 

I'm sure they will officially respond and explain it better,  and I believe there might be some documentation additions or changes to explain, as well as possibly some solutions to assist with resolution.

What is seen on the UX is confusing and might change in future to better explain, but the behavior is correct.  It's better that a more technical explanation is given by Vmware, but the short easy one is merely that the multitude of different instruction sets and features between CPU's are very close, and sometimes overlap  - so some features that might be visible on a baseline X is also seen on a slightly older cpu version - and that is what the UX shows.

The correct workaround to sort the behavior out for my use case is to either enable per-VM EVC to match the required level ("highest level supported on all hosts in clusters when migrations are required) - or have EVC enabled on the cluster BEFORE moving Vm''s to it.

It is not recommended to change HW versions to older/lower ones, and most other workarounds around config files etc are not ideal in my opinion. 

I'm sure there will be some more official answer soon, as well as indication of possible solutions to assist with changed/enabling per VM EVC mode in an automated way - but it will take some time as there are many complexities around this - but watch this space!

For those who have capacity, and need to resolve - a reboot is required either way - so one solution would be create a temp/new cluster with hosts - enabled the correct EVC mode, and shutdown/migrate the VM's to it (the ones with a too high EVC applied). Once done and all vm's are moved off the old cluster, enable the same EVC level on the old cluster and you can then migrate as needed with no more issues. Where this is not possible, a shutdown to enable per VM EVC mode is the other solution - at the moment though a manual process per vm, also requried a shutdown/reboot. This is of course not ideal for a large estate where hundreds or more of VM's are affected - in this case - lets hope some automation tools or processes are going to be released from Vmware to assist.

[nianliu]

experiencing the exact same issue and still no resolution. do we have any new update on this?

[Igubu]

Hi,

I have not received the confirmation of update of documentation yet. I can explain though that the behavior is " Expected " in some ways, because of the complexity of EVC, and especially when CPU generations are fairly close to each other.

For me I wanted to confirm it's not an issue/bug that will be fixed, before I migrated all SQL instances to it (which is why we added 2x dedicated hosts to a cluster - for all SQL) -but after this, it's confirmed for me that there isn't a way to "fix" this really. If you are planning to migrate VM"s to/from the new hosts and older ones, and if it can be a separate cluster, enable the EVC level on it to match the older hosts. That old saying " enable BEFORE moving". The odd one or to on the older hosts that might be detected as a generation newer - which will refuse to move as you have forced an older EVC on the new cluster - will have to be shut down and moved the first time.

I'll ping the engineer again for an update and comment here if he replies.  

[Mike Gerolami]

any news on this solution?

[Igubu]

Hi, 

I have reached out to engineering tonight - will post any updates receive - but the explanation after numerous testing by them is that it's not a "fault" as such, it's because of many complex things used and supplied by for example Intel which determines "broadly"  the different EVC levels. To me, even if it's not a solution as such, it's at least an explanation and confirmation that there isn't something inherently wrong, you just need to know about it.

As mentioned before, there is a workaround that solves the main issue here - how to migrate the vm's between older and newer hosts. The issue for me is the fact that older generation hosts incorrectly detects an EVC compatibility mode higher than what the physical cpu's detect and applies this on boot - even though EVC Is not enabled on the cluster or the individual VM. The fix is easy - force the correct EVC on the cluster or the VM itself, but both requires a reboot.

I'll post if there is any update tomorrow.

 

[GeoPerkins]

I am finding a similar problem and was very surprised at the result.

1. Used a live migration from legacy hardware (where cluster has EVC enabled, cascadelake), destination of live migration is a new cluster with EVC icelake. 

2. As expected VM arrives with cascadelake.

3. Powered off VM.

4. Powered on VM, expecting VM to adopt cluster's icelake EVC mode. Surprise! It does not.

This documentation https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-A1C095EF-1B0F-4C10-9190-CFDD07193B2B.html says "For powered on virtual machines with per-VM EVC disabled, the VMware EVC pane shows the EVC mode that the virutal machine inherits from its parent EVC cluster or host".

A. No EVC mode visible in the GUI. To what "EVC pane" is the above documentation referring to?!

B. The VM in my test did not inherit the EVC mode.

 

Here is the PowerCLI command I used to collect data before, while powered off, and after power on: 

Get-VM | Select-Object -Property Name,PowerState,@{Name='MinRequiredEVCModeKey';Expression={$_.ExtensionData.Runtime.MinRequiredEVCModeKey}},@{Name='Cluster';Expression={$_.VMHost.Parent}},@{Name='ClusterEVCMode';Expression={$_.VMHost.Parent.EVCMode}} | FT

 

 

[GeoPerkins]

I believe I have stumbled upon the solution.

The VM in question was still had virtual hardware level 13 (ESXi 6.5 compatible). I scheduled the compatibility upgrade and restarted, now the VM is at level 19 (ESXi 7.0U2 compatible) and the VM has adopted the icelake EVC mode.

In addition, there is now an "EVC pane" in the vCenter GUI on the Configure tab.

The PowerCLI also agrees, showing the VM having an attribute MinRequiredEVCModeKey=intel-icelake