ESXi

 View Only
Expand all | Collapse all

AMD EPYC nested virtualization Hyper-V

nballfr

nballfrFeb 20, 2023 07:43 PM

  • 1.  AMD EPYC nested virtualization Hyper-V

    Posted Dec 13, 2022 08:03 AM

    I have an issue with nested virtualization on ESX server with AMD EPYC CPU.

    On ESX is running VM with Windows server 2019 with Hyper-V and VM created in Hyper-V gen.2 is not working. Only black screen is showing.

    The same configuration on Intel CPU is working fine.

    VM gen.1 created on Hyper-V with AMD EPYC CPU is working too, but this VM gen.1 has Legacy boot, while VM gen.2 has UEFI

    I've tried also Windows server 2022 but with same result.

    Do you have some hint what can I try?



  • 2.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Jan 26, 2023 03:26 AM

    I have the exact same issue, Gen2 VMs don't work on the AMD EPYC CPU; I believe its been a long-standing issue, would be good to have a definitive response from somebody as to why and how we can overcome this issue.

     



  • 3.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Feb 20, 2023 07:43 PM

    Did you find a solution to this issue?



  • 4.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Mar 12, 2023 01:17 AM

    I spent the better part of a day trying to get this working and gave up (shortly after finding this thread and realizing it wasn't just me).  Nested Server 2022 with Hyper-V role on ESXi 8.0b with EPYC Rome 7302 CPU.  Same issue - Gen 1 VMs power on fine, but Gen 2 go to black screen with the equivalent of 1 CPU core pegged (25% VM utilization on a quad core Server 2022 VM according to Hyper-V manager).  The odd thing is that the Server 2022 host VM doesn't show any CPU activity close to 25%.

    I found some references to SEV-ES being a possible culprit, but in vCenter 8 did not see any option to disable that encryption feature.



  • 5.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Mar 20, 2023 02:55 PM

    Maybe you should try disabling SEV-ES in the BIOS/EFI.
    Please let us know if it did make the trick.
    Thanks



  • 6.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Mar 28, 2023 10:34 AM

    RE the SEV-ES option:

     

    a) doesn't exist

    b) disabling security same issue



  • 7.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Mar 28, 2023 02:31 PM

    JFYI - I've been able to reproduce the issue described here and I've reported this to Engineering. They are currently investigating the issue to understand what might be going on given G1 VM works fine but not G2. If we have more information to share, I'll be sure to reply back to this thread



  • 8.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Mar 31, 2023 02:42 PM

    Thanks William for escalating this issue to Engineering as it is a major problem for us.
    We are using lot of nested hypersvisors for testing purposes and now wish to use ESXi 8.0 on AMD.

    Please check this thread:
    https://communities.vmware.com/t5/VMware-Workstation-Player/Incompatible-AMD-V-on-Workstation-17-0-but-not-16-x/m-p/2949182#M40059

    Thank you.

     



  • 9.  RE: AMD EPYC nested virtualization Hyper-V

    Posted May 31, 2023 06:38 PM

    Same issue I am seeing on vSphere ESXi 7.0.3 21424296 Enterprise Plus

    GEN1 Nested Hyper-V VMs boot and load

    GEN2 goes to a black screen.

    We are supporting a customer that needs GEN2 to work. 

    If not, many passible delays ahead. 

     



  • 10.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted May 31, 2023 07:21 PM

    This will be fixed in a future update of ESXi 7.x/8.x. For now, only Gen1 will be functional when running an inner-VM inside of Hyper-V (running as VM)

    JFYI - Nested Virtualization isn't officially supported by VMware



  • 11.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Jun 01, 2023 01:40 PM

    Thanks for the reply William.

    Any idea what "future" may represent  - 1 month, 1 yr, 2 yrs?

    It would be nice to have some idea in order to work with our customer on alternatives.

     

    And yes, all of us know nested is a bad word at VMware, but with many labs now using it extensively for research (not necessarily production).

    It would seem that it should be looked into more closely from an engineering and support stance?

     

    thanks



  • 12.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Jun 01, 2023 02:30 PM

    We don't comment on dates/versions. As I said, this is fixed and will be available in a future update. 



  • 13.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Jun 05, 2023 02:45 PM

    Thanks ..


    We will monitor the next rerelease.


    i assume this would be noted in the release notes ?



  • 14.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Jul 28, 2023 12:05 PM

    I'm happy to share that latest vSphere 8.0 Update 1c resolves this issue! Even if you're not running a Hyper-V VM, if you are using Windows VBS, you could still see some performance degradations. For more details, please see https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-generation-2-nested-vm-running-on-amd-cpu.html 



  • 15.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Aug 14, 2023 11:00 AM

    Using nested vSphere 8.0 Update 1c (ESXi build-22088125 and vCenter) on AMD EPYC 7763 processor:

    Failed to power on virtual machine. This host supports AMD-V, but the AMD-V implementation is incompatible with VMware ESX.

    It immediately fails when starting any VM, whether it is Windows or Linux.
    Not sure if it is related to this issue that Update 1c was supposed to fix.

    This is driving me nuts: any ideas or workaround please?
    Thank you.



  • 16.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Aug 14, 2023 02:42 PM

    Hi  - I assume you're running ESXi as a VM and you're attempting to power on a Windows or Linux VM? If so, can you provide the vmware.log for the outer (ESXi in VM) as well as the vmware.log for the Windows or Linux VM when you attempt to power those on? 



  • 17.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Aug 16, 2023 11:33 AM

    It would also be good to know if this is a nested Hyper-V VM or ESXi VM.

    We see the issue with GEN2 Hyper-V VMs, nut on ESXi 7.



  • 18.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Oct 10, 2023 11:58 PM

    That's right, running nested ESXi 8.0U2 attempting to power on a Windows or Linux VM on AMD processor.
    Tried with EPYC 7763 and Ryzen 9 5950X, same issue on both processors!
    Works just fine with nested ESXi 7.x on the same hardware/config

    Please find the logs included as well as some screenshots.
    Any help would be greatly appreciated.
    Thank you.

    ESXi-8.0U2-Installer.png

    ESXi-8.0U2-Starting-VM-Error.pngESXi-8.0U2-Starting-VM-Error-Details.png

    VM Logs (vmware.log):
    vmware.log 

    ESXi Logs (hostd.log):
    hostd.log

    VM Config (w11-test.vmx):
    w11-test.vmx

     

     

     

     

     



  • 19.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Oct 11, 2023 11:17 AM

    We are running ESXi 7.0 U2 on AMD EPCY 7452 processors

    I have a VM running Windows Server 2019 with Hyper-V installed and have a GEN1 WIN10 VM and a GEN2 WIN10 VM.

    I installed the WIN10 OS on the GEN1 nested VM without and issue and it boots up and runs

    I tried installing the WIN10 OS on the GEN2 nested VM and it just boots to black screen..

     



  • 20.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Oct 11, 2023 04:35 PM


  • 21.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Oct 11, 2023 06:45 PM

    But how about the issue running VMs on top of nested ESXi 8.x with AMD processors?
    Thank you



  • 22.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Oct 11, 2023 07:37 PM

    Please see the blog post



  • 23.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Oct 11, 2023 07:40 PM

    Could you provide me with a link for the blog post?

    FYI, I'm using ESXi 8.0U2 which is still impacted.



  • 24.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Oct 11, 2023 07:47 PM

    Sorry, I just realized what you posted isn't actually related to whats been described here, which is detailed in https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-generation-2-nested-vm-running-on-amd-cpu.html 

    Also, please create a new thread rather than using this which is unrelated to issue you're having

    From the looks of the screenshot, it seems your Hyper-V VM doesn't have VHV enabled? Can you please confirm that you're running ESXi on bare-metal? Typically this issue indicates that you're using a non-VMware Hypervisor on bare-metal and then attempting to do Nested Virtualization ... which may have issues as the bare-metal Hypervisor isn't ESXi (which we know woks fine in these scenarios)

     



  • 25.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Oct 11, 2023 08:24 PM

    Please note bare-metal Hypervisor is ProxMox 7 or 8.
    I would understand if VMs running on top of a nested ESXi 8.x be solely supported on a VMware bare-metal Hypervisor.
    But as everything is working great with a nested ESXi 7.x on PM, I was hoping it would also be the case for ESXi 8.x.
    Unless there is a technical limitation that could easily be overcame? Would you be able to help?
    In which case, I'll create a new thread for that purpose for sure.
    Thank you.



  • 26.  RE: AMD EPYC nested virtualization Hyper-V

    Broadcom Employee
    Posted Oct 11, 2023 08:38 PM

    I figured you were NOT using our Hypervisor and this was a 3rd party Hypervisor. Unfourntately, I can't speak to how they implement VHV and whether there's changes that needed on their solution, so I'm afraid I won't be able to help (especially as this is a VMware forum)

    My personal recommendation is to use ESXi as the bare-metal Hypervisor, while you can say I'm bias, I have first hand data from thousands and thousands of customers who run our Hypervisor to virtualize other 3rd party Hypervisor because it just works and behaves exactly as they'd expect. I can't say this is the same experience when using other 3rd party Hypervisor, in fact this is how many do extensive testing  

    You also reduce the need for another layer of virtualization for running your Hyper-V VM and you're already using ESXi, so might as well do it the proper way 



  • 27.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Nov 30, 2023 05:04 PM

    Issue is finally fixed in Proxmox 8.1.x (kernel 6.5.x): VMs can now be created/ran on top of nested ESXi 8.x hypervisors.
    Working using PVE Enterprise repository as well as PVE No-Subscription repository.
    That's great news!



  • 28.  RE: AMD EPYC nested virtualization Hyper-V

    Posted Jan 08, 2024 06:00 PM

    I get that very message:

    Task failed on server: This host supports AMD-V, but the AMD-V implementation is incompatible with VMware ESX.
    Installer log files are located at /tmp/vcsaUiInstaller

    While trying to deploy vCenter on ESXi (all 8.0U2) on AMD Ryzen Pro 3900 - process fails @ 80%

    Virt platform in my case is Redhat's Qemu/KVM.

    No VMware for me - I run a test/trial lab for the first time and if there is not a workaround for this that is then the end of the road - no chance I'll get more/different hardware only for trial/tests.

    Thus, if there'll ever be a "fix" for it, I'll sure be grateful to learn of it.

    many thanks, L.