AMD EPYC nested virtualization Hyper-V

[Pet Ko]

I have an issue with nested virtualization on ESX server with AMD EPYC CPU.

On ESX is running VM with Windows server 2019 with Hyper-V and VM created in Hyper-V gen.2 is not working. Only black screen is showing.

The same configuration on Intel CPU is working fine.

VM gen.1 created on Hyper-V with AMD EPYC CPU is working too, but this VM gen.1 has Legacy boot, while VM gen.2 has UEFI

I've tried also Windows server 2022 but with same result.

Do you have some hint what can I try?

[fk1h8theseStupi]

I have the exact same issue, Gen2 VMs don't work on the AMD EPYC CPU; I believe its been a long-standing issue, would be good to have a definitive response from somebody as to why and how we can overcome this issue.

 

[nballfr]

Did you find a solution to this issue?

[bunsenbunner]

I spent the better part of a day trying to get this working and gave up (shortly after finding this thread and realizing it wasn't just me).  Nested Server 2022 with Hyper-V role on ESXi 8.0b with EPYC Rome 7302 CPU.  Same issue - Gen 1 VMs power on fine, but Gen 2 go to black screen with the equivalent of 1 CPU core pegged (25% VM utilization on a quad core Server 2022 VM according to Hyper-V manager).  The odd thing is that the Server 2022 host VM doesn't show any CPU activity close to 25%.

I found some references to SEV-ES being a possible culprit, but in vCenter 8 did not see any option to disable that encryption feature.

[nballfr]

Maybe you should try disabling SEV-ES in the BIOS/EFI.
Please let us know if it did make the trick.
Thanks

[dp213]

RE the SEV-ES option:

 

a) doesn't exist

b) disabling security same issue

[William Lam]

JFYI - I've been able to reproduce the issue described here and I've reported this to Engineering. They are currently investigating the issue to understand what might be going on given G1 VM works fine but not G2. If we have more information to share, I'll be sure to reply back to this thread

[nballfr]

Thanks William for escalating this issue to Engineering as it is a major problem for us.
We are using lot of nested hypersvisors for testing purposes and now wish to use ESXi 8.0 on AMD.

Please check this thread:
https://communities.vmware.com/t5/VMware-Workstation-Player/Incompatible-AMD-V-on-Workstation-17-0-but-not-16-x/m-p/2949182#M40059

Thank you.

 

[stanj]

Same issue I am seeing on vSphere ESXi 7.0.3 21424296 Enterprise Plus

GEN1 Nested Hyper-V VMs boot and load

GEN2 goes to a black screen.

We are supporting a customer that needs GEN2 to work. 

If not, many passible delays ahead. 

 

[William Lam]

This will be fixed in a future update of ESXi 7.x/8.x. For now, only Gen1 will be functional when running an inner-VM inside of Hyper-V (running as VM)

JFYI - Nested Virtualization isn't officially supported by VMware

[stanj]

Thanks for the reply William.

Any idea what "future" may represent  - 1 month, 1 yr, 2 yrs?

It would be nice to have some idea in order to work with our customer on alternatives.

 

And yes, all of us know nested is a bad word at VMware, but with many labs now using it extensively for research (not necessarily production).

It would seem that it should be looked into more closely from an engineering and support stance?

 

thanks

[William Lam]

We don't comment on dates/versions. As I said, this is fixed and will be available in a future update. 

[stanj]

Thanks ..

We will monitor the next rerelease.

i assume this would be noted in the release notes ?

[William Lam]

I'm happy to share that latest vSphere 8.0 Update 1c resolves this issue! Even if you're not running a Hyper-V VM, if you are using Windows VBS, you could still see some performance degradations. For more details, please see https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-generation-2-nested-vm-running-on-amd-cpu.html 

[nballfr]

Using nested vSphere 8.0 Update 1c (ESXi build-22088125 and vCenter) on AMD EPYC 7763 processor:

Failed to power on virtual machine. This host supports AMD-V, but the AMD-V implementation is incompatible with VMware ESX.

It immediately fails when starting any VM, whether it is Windows or Linux.
Not sure if it is related to this issue that Update 1c was supposed to fix.

This is driving me nuts: any ideas or workaround please?
Thank you.

[William Lam]

Hi  - I assume you're running ESXi as a VM and you're attempting to power on a Windows or Linux VM? If so, can you provide the vmware.log for the outer (ESXi in VM) as well as the vmware.log for the Windows or Linux VM when you attempt to power those on? 

[stanj]

It would also be good to know if this is a nested Hyper-V VM or ESXi VM.

We see the issue with GEN2 Hyper-V VMs, nut on ESXi 7.

[nballfr]

That's right, running nested ESXi 8.0U2 attempting to power on a Windows or Linux VM on AMD processor.
Tried with EPYC 7763 and Ryzen 9 5950X, same issue on both processors!
Works just fine with nested ESXi 7.x on the same hardware/config

Please find the logs included as well as some screenshots.
Any help would be greatly appreciated.
Thank you.

VM Logs (vmware.log):
vmware.log 

ESXi Logs (hostd.log):
hostd.log

VM Config (w11-test.vmx):
w11-test.vmx

 

 

 

 

 

[stanj]

We are running ESXi 7.0 U2 on AMD EPCY 7452 processors

I have a VM running Windows Server 2019 with Hyper-V installed and have a GEN1 WIN10 VM and a GEN2 WIN10 VM.

I installed the WIN10 OS on the GEN1 nested VM without and issue and it boots up and runs

I tried installing the WIN10 OS on the GEN2 nested VM and it just boots to black screen..

 

[William Lam]

This issue is fixed in both vSphere 8.0U1c and vSphere 7.0U3o - https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-generation-2-nested-vm-running-on-amd-cpu.html 

[nballfr]

But how about the issue running VMs on top of nested ESXi 8.x with AMD processors?
Thank you

[William Lam]

Please see the blog post

[nballfr]

Could you provide me with a link for the blog post?

FYI, I'm using ESXi 8.0U2 which is still impacted.

[William Lam]

Sorry, I just realized what you posted isn't actually related to whats been described here, which is detailed in https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-generation-2-nested-vm-running-on-amd-cpu.html 

Also, please create a new thread rather than using this which is unrelated to issue you're having

From the looks of the screenshot, it seems your Hyper-V VM doesn't have VHV enabled? Can you please confirm that you're running ESXi on bare-metal? Typically this issue indicates that you're using a non-VMware Hypervisor on bare-metal and then attempting to do Nested Virtualization ... which may have issues as the bare-metal Hypervisor isn't ESXi (which we know woks fine in these scenarios)

 

[nballfr]

Please note bare-metal Hypervisor is ProxMox 7 or 8.
I would understand if VMs running on top of a nested ESXi 8.x be solely supported on a VMware bare-metal Hypervisor.
But as everything is working great with a nested ESXi 7.x on PM, I was hoping it would also be the case for ESXi 8.x.
Unless there is a technical limitation that could easily be overcame? Would you be able to help?
In which case, I'll create a new thread for that purpose for sure.
Thank you.

[William Lam]

I figured you were NOT using our Hypervisor and this was a 3rd party Hypervisor. Unfourntately, I can't speak to how they implement VHV and whether there's changes that needed on their solution, so I'm afraid I won't be able to help (especially as this is a VMware forum)

My personal recommendation is to use ESXi as the bare-metal Hypervisor, while you can say I'm bias, I have first hand data from thousands and thousands of customers who run our Hypervisor to virtualize other 3rd party Hypervisor because it just works and behaves exactly as they'd expect. I can't say this is the same experience when using other 3rd party Hypervisor, in fact this is how many do extensive testing  

You also reduce the need for another layer of virtualization for running your Hyper-V VM and you're already using ESXi, so might as well do it the proper way 

[nballfr]

Issue is finally fixed in Proxmox 8.1.x (kernel 6.5.x): VMs can now be created/ran on top of nested ESXi 8.x hypervisors.
Working using PVE Enterprise repository as well as PVE No-Subscription repository.
That's great news!

[lejeczeki]

I get that very message:

Task failed on server: This host supports AMD-V, but the AMD-V implementation is incompatible with VMware ESX.
Installer log files are located at /tmp/vcsaUiInstaller

While trying to deploy vCenter on ESXi (all 8.0U2) on AMD Ryzen Pro 3900 - process fails @ 80%

Virt platform in my case is Redhat's Qemu/KVM.

No VMware for me - I run a test/trial lab for the first time and if there is not a workaround for this that is then the end of the road - no chance I'll get more/different hardware only for trial/tests.

Thus, if there'll ever be a "fix" for it, I'll sure be grateful to learn of it.

many thanks, L.