vSphere 7.0- Unable to perform PCI pass through with Nested Virtualization Enabled

[gskjelstad]

I have 2 HCL listed - VMWare servers that I recently upgraded to ESXi 7.0 from 6.5.

I'm attempting to get DDPIO working with some dedicated legacy SSL Accelerator cards (I know, I know, it's legacy support)

With ESXi 6.5, I was able to get around this problem by using the following syntax inside of the .vmx files and importing them.

vhv.enable = TRUE

hypervisor.cpuid.v0 = FALSE

However 7.0 doesn't seem to be respecting these flags and I continue to get the following:

PCI passthrough devices cannot be added when Nested Hardware-Assisted Virtualization is enabled. 

I'm not seeing anything in the release notes that would indicate that these customization would stop working..

I'm not expecting them to support the Guest or any of the passthrough configuration/functionality, but c'mon, I can't even power on the VM.

Before I jump in and open a ticket, does anyone have any direction on doing this on ESXi 7.0?

Thanks,

Garrett

Great Video on PCI Passthrough: PCI pass-through & Folding at Home on the VMware appliance (VMware vSphere ESXi 7) Jason Meers - YouTube

EDIT: 2020-06-09:

Added the

vhv.allowPassthru = TRUE

flag, and that didn't help either.

[Reza Ardaneh]

Hi

According to the error that you have mentioned, You can not pass through a PCI device when the hardware virtualization feature of CPU is enabled, so you should check your VM configuration to be sure that "Expose hardware-assisted virtualization to the guest OS" is disabled.

I hope this could be helpful

[gskjelstad]

I think I didn't make it clear that I WANT to do that behavior, and that I could do it in previous versions. (yes in 6.7, they removed the feature, and require a flag...)

I'm wanting to enable passthrough as well as nested virtualization the same time in the most recent version as well.

[Reza Ardaneh]

There is another tag in ESXi 6.7 u2, so I believe it can be used in vSphere7

If you didn't use this tag just give it a try:

vhv.allowPassthru = "TRUE"

I hope it could be helpful

Please share any solution if the problem was solved

[gskjelstad]

Adding in this advanced config flag didn't allow the VM to boot either. It continues to get the same error.

Any other ideas?

[bgwallace]

Team,

Does anyone know how we're supposed to enable VBS (which enables and then grays out the "Expose hardware assisted virtualization to the guest OS" setting) and then pass PCI devices like video cards to a VM which requires this setting to be disabled?  Am I missing something?

I tried adding the suggested vhv.allowPassthru and vhv.enable settings in 7.0 U3 but the VM won't register when using the UI (hangs indefinitely) and vim-cmd registers the VM but leaves it in an "invalid" status.

Did anyone find any solutions?  Any help much appreciated.  Seems like a major oversight on the VMWare Engineering team but maybe I'm missing something on how we're supposed to use both VBS and pass through PCI devices at the same time or if we're not why because it seems like we should be able to.

Thanks!

Brad

[Valentin Bondzio]

---

 wrote:

(...)
Am I missing something?

---

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-8361F64B-DE1D-4FF1-A9FC-4DDE7056DC23.html

Unsupported VMware Features on VBS

The following features are not supported in a virtual machine when VBS is enabled:

• Fault tolerance
• PCI passthrough
• Hot add of CPU or memory

 

This isn't an oversight, it is as a serious engineering challenge. VMware (lower case w) can't comment on potential future capabilities or when (if) they ship.

[christopherjef1]

I have this exact problem, had to rollback to 6.5 as that VM is critical

Here's a screenshot when I attempted to edit the VM config to remove and re-add while on 7 (which also didnt work)

[MarKol4]

Hi,

Did you find any solution?

Knowledge base article, sais "it is not recommended".

But some people used to run VMs with passthrough and nested virtualization succesfully in ESXi 6.7 and before. And now it is not recommended? Surely I can accept that it is not recommended (although it would be nice to know why). But not recommended does not mean forbidden or impossible, right?

Is there a way in latest ESXi 7.x to disable check described in kb article and and run VM with passthrough and nested virtualization together?

Thanks,

Marek.

 

[Khvastunov]

Hello.

For now we been able to bypass new checks for simltaneous use of nested virtualization and PCI passthrough with a little hack:

• power off VM in question
• remove VM from inventory
• edit .vmx file of this VM, you want to have both parameters there: vhv.allowPassthru = "true" and vhv.enable = "true"
• register VM

But be warned - we resulted in severe performance issues with this setup and now we consider downgrading to 6.5 

[SupportExperts]

hi gskjelstad

Did you find a solution? thanks

[stonic80]

For myself i was not able to do passthrough with k2200 video card until i put off Expose IOMMU to the guest OS in CPU option

[390843759]

Many thanks for your inspiration on that!

I had the same problem in vSphere 8, fixed doing that:

1) unregister vm esxi from vCenter

2) browse datastore containing vm esxi

3) In the vm .vmx file, add:

vhv.allowPassthru = "TRUE"

vhv.enable = "TRUE"

Good luck!

[phoenix3dfx225]

100% legend. I've been looking at this issue on and off for months.

The taking it out of the inventory, download the VMX file, amend then delete existing and upload amended VMX Now working.

 

[Tina White]

I was not able to get thist to work.  I did change it a little. 

A vendor wants to upgrade an application on a VM using Nvidia and wants to use nested virtualization.  The instructions they gave were using VMware 6.5.  We are on VMware 8.0.3. 

They wanted me to add:

vhv.enable = "TRUE"
hypervisor.cpuid.v0 = "FALSE"

This didn't work so I added:

vhv.allowPassthru = "TRUE"

vhv.enable = "TRUE"

hypervisor.cpuid.v0 = "FALSE"

It seemed to work... kept the Nvidia PCI device and had the box checked for "Expose hardware assisted virtualization to the guest OS" but I wasn't able to login. 

The box next to the network adapter for "connected" was unchecked.  When I tried to check it, I got the error "Nested Hardware-Assisted Virtualization is not supported on a virtual machine with a PCI passthrough device."

-------------------------------------------

[Tina White]

I was not able to get this to work.  I did change it a little.

A vendor wants to use nested virtualization to do an upgrade on a VM that is using Nvidia.  The instructions they sent were for VMware 6.5 and we are on VMware 8.0.3.

Their instructions say to edit the vmx and add:

vhv.enable = "TRUE"
hypervisor.cpuid.v0 = "FALSE"

When this didn't work, I found this thread and modified it a little to look like this:

vhv.allowPassthru = "TRUE"

vhv.enable = "TRUE"

hypervisor.cpuid.v0 = "FALSE"

Seemed to work, before I booted it up, I checked and the box next to "Expose hardware assisted virtualization to the guest OS" and the PCI device NVIDIA was still there.  

When I powered it on, I found the network adapter was not connected, the box was unchecked.  When I checked the box, I got the error "Nested Hardware-Assisted Virtualization is not supported on a virtual machine with a PCI passthrough device."

I'm wondering if there is a way to get this to work....

------------------------------
KY1121
------------------------------