Accelerate 3d Graphics causing VMWare to hang/freeze. When shutting down Host also getting BSOD VIDEO DXGKRNL_FATAL_ERROR 15.5.6 build-16341506

[TMWSales]

Win10 Host:

Dell Precision 7530

Win10 2004

Intel UHD 630 Driver: 27.20.100.8336

NVidia Quadro P3200 Driver: 451.77

VMWare Workstation Pro: 15.5.6 build-16341506

Win10 2004 Guest VM

Ever since updating to Win10 2004 I am getting the VM to hang/freeze.  If the VM doesn't hang/freeze and I am able to shut it down. Then when I shut down the host I get the BSOD.  Just using the host I don't have any problems.  I have tried different video driver versions and no change and same problem.

If I turn off Accelerate 3D Graphics in VMWare, I don't have a problem anymore and it doesn't freeze and I my host no longer gives me a BSOD when shutting down.

[bluefirestorm]

Try to set the Nvidia as the default graphics processor in both the Nvidia Control Panel and Windows Graphics settings in the Windows 10 2004 host.

This will prevent the Windows host OS trying to switch the graphics engine to Intel (or viceversa) when the VM is running with "3D Acceleration" on. That switching might be the cause of the freeze/hang that you experience and the subsequent crashing.

[TMWSales]

I adjusted Windows to have High Performance for VMWare Workstation in the graphics settings.  I also adjusted the manage 3D settings in NVidia control panel to set the program VMWare.exe to high performance NVidia Processor.  But I still have the same problem when I enable Accelerate 3D Graphics in VMWare.  It froze almost as soon as I got into windows and had to Close VMware through task manager.  Then I restarted my host and got the BSOD again for VIDEO_DXGKRNL_FATAL_ERROR.

This problem only started once I updated to Windows 10 2004. I never had this problem before updating.  Now it is past the two week mark to roll back the update of 2004.  But this appears to be a problem with VMWare and Windows 10 2004.  I am not using Hypervisor or it is not installed on Windows Features.  I reverted the settings and disabled Accelerate 3D Graphics and my Windows 10 VM is working without crashing on my host.  I also don't get any BSOD when shutting down the host.

My Windows 7 VM machine doesn't appear to have any problems with Accelerated 3D Graphics turned on.

[bluefirestorm]

The BSOD on the host appears to be related to DirectX Graphics. When 3D acceleration is enabled on the VM, by default, VMware Workstation will use DX11 of the Windows 10 host as the render engine for the VM. The Nvidia control panel default graphics processor setting is just to ensure the more powerful GPU (Nvidia) is used all the time. You could try setting it to Intel just to preclude whether it is an Nvidia only problem or not.

I haven't used much of Windows 10 2004 (upgraded from 1909) as a host OS. The short time that I have had to try with VMware Player is that when the "Reputation-based protection" was turned on in "App & browser control", it started blocking VMware-related services as a VM is powered on. If that is turned on, you could try turning it off.

How much graphics memory was allocated to the VM? Max is now 3GB, have you gone with the recommended 1GB? Is the VMware Tools of the VM up-to-date? Can you attach (at the lower right hand side of this reply box there is a link to attach files) the vmware.log of the VM?

Another alternative is make VMware Workstation use OpenGL instead of DX11 but it is unlikely to solve the problem if the cause lies somewhere else. On Linux hosts, OpenGL 4.x is used for 3D acceleration enabled VMs in Linux hosts.

This can be done by adding the following lines to the VM vmx configuration file.

mks.enableDX11Renderer = "FALSE"

mks.enableGLRenderer = "TRUE"

In the past, I have tried using the GL renderer with Intel HD graphics on Windows 10 host leads to rendering problems in a Windows 10 VM but seems OK with an Nvidia GPU.

[TMWSales]

I have also checked drivers and tried different versions.  I never had a problem before upgrading to Win10 2004.  My VM's would work without problems.  I will try some of these. 

My VMWare tools is up to date on all VM's.

I am using the recommended 1GB of video memory as default by VMWare.

My Host is Win10 2004 and the VM giving me trouble is also Win10 2004.

[bluefirestorm]

I run two Windows 10 2004 Professional VMs; one on an Ubuntu host with Workstation Pro 15.5.6 and the other on a MacBook Pro with Fusion Pro 11.5.6. They are not without their problems.

I suspect something got broken and/or is different with upgrade to Windows 10 2004 to a fresh install. I couldn't upgrade the VM from 1909 to 2004 when 4 or more vCPUs were assigned as it kept on failing with a BSOD in one of the reboots during the upgrade. The upgrade only got through after the vCPUs was reduced to 2. The VMs power up also fail with a BSOD when 4 or more vCPUs are assigned to the VM. The workaround was to install the Hyper-V options inside the VMs. But yet when a new VM with 4 vCPUs is created with a install of Windows 10 2004 from Windows Media Creator, the install proceeds and power-ups succeeds and doesn't require the Hyper-V option workaround.

Another problem is the Windows Defender offline scan in the VM with 4vCPUs also fails with a BSOD just after it reboots to start the offline scan. To be able to proceed, the number of vCPUs had to be reduced to 2. I haven't tried if this problem exists with a fresh install of 2004.

How many vCPUs have you assigned? Perhaps the freezing (and subsequent host BSOD) is a different manifestation of the problem.

[TMWSales]

I have 6 vCPU assigned and haven't had any problems with that.  If I disable the 3D accelerated graphics. I can run Prime95 to a blend test and it performs without failure until I stop it.  So I know I can stress the system and it handles it without problems.  But the moment I enable 3D graphics in the VM only for Windows10 2004.  Even if I just start the VM and let it sit at the desktop, it will freeze randomly.  Especially if I let the system sit idle.  Then as soon as the VM freezes.  I have to end task it.  Then I will reboot my host and I get the BSOD.  If the VM doesn't freeze and I shut down the VM and then close out of VMWare,  Everything seems fine.  I shut down the host and I get the BSOD. 

I had no problems performing the upgrade.  I initially did the upgrade back in May, but I kept having instalbility problems with VMWare.  So I rolled the update back for the VM and the host.  Everything ran fine without problems.  I performed the update again in June/July.  This time my VM worked better, but then I started to notice these random freezes and the BSOD when the freeze would happen.  I figured I would wait it out and see if any updates come out, but nothing has.  I have checked Hyper-V and it should not be enabled at all.  I tried newer drivers, older drivers. But same problem.  So I am guessing some compatibility with VMWare and Windows10.  RIght now I am leaving 3D graphics off in this particular VM as I don't require it for my software.

I am going to try to create a new Win10 2004 VM and see if that has any problems this weekend.

[TMWSales]

I had some time to play with this and so far a new Windows 10 VM doesn't have problems with just base windows. I am not pushing any graphics, etc.  But I was working on one of my existing Windows 7 VM and started to have problems there.  I disable Accelerated 3D graphics and now that VM is not having any problems.  This appears to be a problem with VMWare and the host when running Windows 10 2004.  I am currently running the newest graphics drivers from NVidia and Intel and newest updates to Windows 10 as of September 6.  Still the same problems.

[Fezzik]

Had issues as well with 2004 install - turning off 3D acceleration was the workaround until I found other post stating "change the hardware compatibility of my VM's from Workstation 10.x or 11.X to 15.X.".  This fixed it for me. 

[UnitUniverse2]

I think there is something wrong within those Win10 200x versions. I've installed several versions as guests and when running an NVIDIA demo

https://www.guru3d.com/files-details/nvidia-island-direct3d-11-tech-demo.html 

the ver.200x will show nothing but a ghost app button on its task bar. In comparison the ver.190x and earlier will show this:

There is another issue within those 200x versions:
https://developercommunity.visualstudio.com/content/problem/1208774/fpu-exception-in-fmod0-x-in-windows-10-version-200.html?childToView=1216371#comment-1216371
or
https://developercommunity.visualstudio.com/content/problem/1207405/fmod-after-an-update-to-windows-2004-is-causing-a.html
BTW the 'fmod' is a low-level(user mode still) runtime function which is called by many softwares and librarys invoked by many softwares here and there.