SUMMARY
On a Windows host, a Linux guest presented via the DX12 present backend turns permanently black whenever the host does a secure-desktop switch — a UAC elevation prompt or a Win+L lock/unlock. The VM keeps running (guest, apps, SSH all alive); only the displayed image is lost. It does NOT recover on its own, and NOT on a guest reboot or a VM reset — only on Suspend/Resume or a full power-cycle. A Windows guest on the identical host does NOT hit this. I've traced it to a host-side DX12 "device-removed partial recovery" bug in vmware-vmx.exe. I'm looking to (a) confirm others see it, and (b) get a VMware engineer to confirm it's tracked / fixable — because both known workarounds are unacceptable.
IMPORTANT — not every VM is affected, and it is intermittent: it hits LINUX guests presented via DX12 (VKRenderer); a WINDOWS guest on the same host does not hit it at all. And even on an affected Linux guest, a given secure-desktop switch is hit-or-miss. So "it works fine here" does not mean the bug is absent — see the note under STEPS TO REPRODUCE.
ENVIRONMENT
• VMware Workstation 26.0.0 build-25388281 ("26h1"); the same behavior exists in 25.x (build-25219725).
• Host: Windows 11, UAC enabled. NVIDIA (Optimus) laptop — also reproduces in discrete-only BIOS mode, so it is not an Optimus GPU-switch artifact.
• Guest: Kubuntu, KDE Plasma Wayland/KWin, stock in-tree vmwgfx. Reproduces fully stock.
• Present config: mks.enableDX12Presentation="TRUE", remoteMKSHardwareAccel="TRUE"; renderer = ISBRenderer (VKRenderer). Host log at power-on: "MKS-HWinMux: Started DX12 presentation backend."
STEPS TO REPRODUCE
1. Power on the Linux guest with the DX12 present backend active.
2. On the host, trigger a UAC elevation (any admin prompt = secure-desktop switch) — or lock/unlock the host.
3. Repeat about 10–20 times. Intermittently the guest window goes black and stays black.
4. Confirm it clears only via Suspend/Resume or power-on — not by closing/resizing the VMware window, switching monitors, changing guest resolution, guest reboot, or VM reset.
WHY NOT EVERYONE SEES IT — two things determine whether you reproduce it:
• Which guest (VM-dependent): this affects LINUX guests presented via the DX12 backend (VKRenderer). A WINDOWS guest on the same host does NOT hit it — its WDDM present path never sees the device removal. So the single biggest factor is what is running in the VM. (Among the Linux guests I've tested the behavior is consistent — two Kubuntu VMs behaved identically and a fully-stock guest fails too — so I have not seen one Linux VM be intrinsically immune; the remaining variation is the per-switch lottery below.)
• Per switch (intermittent): on an affected Linux guest, some secure-desktop switches cause a genuine DXGI_ERROR_DEVICE_REMOVED (black); others only transient DXGI_ERROR_INVALID_CALL / E_ACCESSDENIED that do not trigger the recovery (survive). So you may need several attempts to hit it, and surviving a few does not mean you are immune.
Caveat on scope: I've only tested this on ONE physical host, so I can't say whether the host GPU/driver changes how OFTEN a switch escalates to a genuine device removal — that's plausible but unverified, which is exactly why I'm asking others to report their setup below. The underlying vmware-vmx recovery defect is host-side and would apply to any host that does experience a genuine DEVICE_REMOVED with a Linux/DX12 guest.
CURRENT BEST MITIGATION (for anyone else hitting this)
• UAC case — fixed by disabling the UAC secure desktop: UAC slider to "Notify me... (do not dim my desktop)" (HKLM\...\Policies\System\PromptOnSecureDesktop = 0). No secure-desktop switch means no device removal means no black, and DX12 present is retained. Caveats: modest security reduction, and it does not cover lock/unlock.
• Lock/unlock case — no config fix. Practical options: disable idle auto-lock (so a lock never surprises you mid-call), and use Suspend/Resume to clear it if it happens.
WHAT I'VE ALREADY RULED OUT (all tested, no effect)
• mks.enableDX12Presentation="FALSE" — avoids the bug but disables the DX12 BlitToSharedSurface GPU present path, falling back to software compositing (large present-latency regression vs a Windows guest). Not acceptable.
• enableDXGIPresentation="TRUE" (DX12 off) — fails to start here: "MKS-HWinMux: Failed to start a presentation backend" (only GDI/software starts).
• Disable MPO (Dwm\OverlayTestMode=5), NoLockScreen=1, TdrDelay — no effect (and hr is DEVICE_REMOVED 0x887A0005, not DEVICE_HUNG — a genuine removal, not a TDR/timeout; VRAM is not exhausted).
• mks.dx12.gpuPreference / mks.dx12.vendorId, NVIDIA high-performance — no effect.
• HAGS off, PCIe Link State Power Management off — no effect (expected: the removal is a desktop/session switch, not a power or scheduling event).
• USB passthrough, concurrency, guest kernel/compositor/driver (fully stock fails identically).
TECHNICAL ROOT CAUSE (host-side reverse engineering — optional detail)
Included so it's clear this is a real, localized defect, not a config problem. Function addresses are 26h1, image base 0x140000000; I can provide logs and decompilations.
The Linux-guest present is a cross-process shared-surface blit: the sandboxed VKRenderer creates a D3D shared texture plus an NT shared handle (DX12Renderer::CreateSharedSurface, FUN_14021c1c0), the guest frame is blitted into it (FUN_1402278f0), and the present backend opens that surface and blits it into the swapchain back buffer behind an IDXGIKeyedMutex (DXGIBlitToBackbuffer, FUN_1401b0850 / FUN_140240e50) on an HWND FLIP_DISCARD swapchain.
On the secure-desktop switch the host DX12 present device is removed (DX12Window::Present returns hr = DXGI_ERROR_DEVICE_REMOVED 0x887A0005). The recovery (FUN_140264730 -> FUN_140175b20 -> reset-worker FUN_1401753c0 -> StartBackend FUN_1401790e0) rebuilds the renderer + present backend + swapchain successfully — Present() even returns S_OK — but it never re-establishes the shared-surface content binding. Three steps that run only at MKS ThreadPowerOn (FUN_14014aec0) or on the renderer's surface-publish event are skipped:
1. the shared-surface re-import handshake HWinMux_SharedSurfaceReady (FUN_140179b80) that re-opens the source into the NEW device and re-acquires its keyed mutex;
2. the SWBScreenComp per-screen source-handle table (DAT_1415d6870), whose only writer is the cold-init FUN_14015fa70;
3. a refresh of the per-window cached surface id (DX12Window+0x28) — the texture builders have a same-size early-out, so an unchanged-resolution recovery reuses the stale surface.
Because the guest keeps rendering, DX12EndFrame (FUN_140242690) re-registers the cached (dead) surface id as the present source every frame, so the keyed-mutex AcquireSync returns 0x80 and the binary logs "DXGIPresentation: Shared surface in an inconsistant state!" — blit skipped — black, forever. This is the classic DX12 "partial device-lost recovery" failure that Microsoft's own device-removed guidance / DirectXTK12 DeviceResources warn about: on device removal you must recreate the device + swapchain AND all device-dependent + content bindings, not just device + swapchain.
WHAT I'M ASKING VMWARE (either resolves it)
1. Complete the recovery: on DEVICE_REMOVED, after restarting the renderer/backend, also re-import the guest shared surface into the new present device + re-acquire its keyed mutex, invalidate the per-window cached texture so it re-creates/re-DEFINEs on the new device, and re-init the SWBScreenComp source table — i.e. re-establish the content binding, not just the swapchain.
2. Or prevent the removal: figure out why the Windows/WDDM guest present path never sees DEVICE_REMOVED across the same secure-desktop switch, and apply the same handling to the Linux/VKRenderer path.
QUESTIONS FOR THE COMMUNITY / VMWARE
• Anyone else running a Linux guest with DX12 presentation seeing black-outs on UAC prompts or lock/unlock? Please reply so this gets weighted — whether you DO or DON'T reproduce it (reports either way help). Please include: guest OS/distro and whether DX12 presentation is on; and host GPU + driver version, laptop or desktop, and whether it's an Optimus/hybrid-graphics system. Comparing setups that hit it vs. those that don't is how we pin down which configurations actually escalate a desktop switch to a genuine device removal.
• Can a VMware engineer confirm whether this is already tracked internally, and whether a fix is planned for a Workstation update?
I have vmware.log excerpts (the DEVICE_REMOVED recovery vs a healthy Suspend/Resume bring-up), Frida instrumentation traces, and per-function decompilations for all addresses above — glad to attach whatever helps. Thanks!