VMware Workstation 17.6.4 - input lag with 3D acceleration on Linux guest: which `.vmx` keys actually take effect

[Isto Nikula]

Posting an empirical investigation of the input-lag-with-3D-acceleration regression on VMware Workstation 17.6.x for Linux guests, in case it's useful for triage. Several threads here and elsewhere mention .vmx workarounds, but most appear to be silently ignored on current builds. Sharing what does and doesn't change selector behavior, with log excerpts.

REPRODUCER
----------

Hypervisor: VMware Workstation Pro 17.6.4 build-24832109
Host:       Windows 11, NVIDIA Quadro, driver 573.91. No Optimus (BIOS set
            to discrete-only).
Guest:      Arch Linux, kernel 6.19.x, Mesa 26.0.x, X11 session, vmwgfx
            kernel module loaded, xf86-video-vmware NOT installed.
Symptom:    With mks.enable3d = "TRUE", typing/cursor input is visibly
            laggy across the desktop. Disabling 3D restores responsiveness.
            Matches user reports going back to mid-2024 17.6.x updates.

WHAT vmware.log SHOWS
---------------------

With 3D enabled, the renderer selector lands on the sandboxed Vulkan path:

    MKSRenderMain: PowerOn allowed BasicOps ISB DX12 DX11 DX11Basic VK
    MKSRenderMain: ISB enabled by config
    ISBRendererComm: Sandbox Renderer: VKRenderer
    MKSRenderMain: Found Full Renderer: ISBRenderer (VKRenderer)

The IPC boundary to mksSandbox.exe is the apparent source of the input
lag - disabling renderers until ISB cannot start makes the lag go away
(at the cost of falling out of full 3D, see below).

.VMX KEYS - EMPIRICALLY TESTED ON 17.6.4
----------------------------------------

For each key, the VM was fully shut down, the key added to the .vmx,
and vmware.log inspected after restart to confirm whether selector
behavior actually changed.

Keys that work (MKSRenderMain selection actually changes):

  * mks.enable3d = "TRUE" / "FALSE"
        Master switch for 3D acceleration.

  * mks.enableVulkanRenderer = "FALSE"
        Removes VK from the "PowerOn allowed" list.

  * mks.enableDX12Renderer = "FALSE"
        Removes DX12 from the "PowerOn allowed" list.

  * mks.enableDX11Renderer = "TRUE"
        Allows DX11 (already allowed by default).

Keys that are silently ignored (parsed into DICT, no runtime effect):

  * mks.gl.renderer = "dx11renderer"
        DICT entry logged, selector keeps Vulkan.

  * mks.isb.enable = "FALSE"
        DICT entry logged, ISB still enabled.

  * mks.sandbox.enable = "FALSE"
        DICT entry logged, ISB still enabled.

  * mks.enableBasicRenderer = "FALSE"
        DICT entry logged, DX11Basic still selected.

The "DICT logged" rows show up as

    In(05) vmx DICT mks.X = "..."

in vmware.log - VMware reads them - but MKSRenderMain behavior is
unchanged. From the outside these read as no-ops on this build; whether
they were removed, never existed, or are reserved for a different code
path, I can't tell.

REACHABLE END STATES
--------------------

Combining the keys that do work, only two states are reachable on
17.6.4:

1) 3D enabled (any combination that allows ISB to start):

   ISB sandbox runs ->
       Found Full Renderer: ISBRenderer (VKRenderer)
   or  Found Full Renderer: ISBRenderer (DX12Renderer)

   Guest sees: SVGA3D, shader model SM_5_1X, full hardware acceleration.
   Cost: input lag present.

2) 3D disabled, OR both VK and DX12 disabled:

   ISB fails to start -> selector retries "without 3D requirements" ->
       Found Full Renderer: DX11BasicRenderer

   Guest's Mesa drops to llvmpipe, no real 3D in the guest.
   Cost: no GPU acceleration, but no input lag.

The selector skips full DX11 in favor of DX11Basic when ISB fails -
even with mks.enableDX11Renderer = "TRUE" and
mks.enableBasicRenderer = "FALSE" set together. There appears to be no
.vmx-controllable path to "low-lag full DX11 in-process" on 17.6.4.
That older path (which, per user reports, gave smooth interactivity in
VMware 15.x) seems to no longer exist, or is gated by something not
exposed in .vmx config.

QUESTIONS FOR VMWARE
--------------------

1) Are the silently-ignored keys above deprecated, removed, or were
   they never supported in this form? Documentation pointers would
   be useful.

2) Is there a supported way on 17.6.x to get hardware-accelerated 3D
   in a Linux guest *without* the ISB sandbox in the input path - i.e.,
   to recover the responsiveness of pre-16.x in-process rendering?
   If not by design, this is the regression being reported across
   multiple threads.

3) Would the team be interested in vmware.log / mksSandbox.log from
   a reproducer? Happy to provide.

CAVEATS
-------

- One build (17.6.4 build-24832109), one host driver vintage (573.91),
  one GPU class (Quadro). Older host GPUs may behave differently - the
  host's D3D feature level reportedly gates which renderers are
  reachable.

- Multi-vCPU was not isolated as a variable; there's a separate
  documented bug about Linux multi-vCPU keyboard lag on Win11 hosts
  that may overlap.

- Investigation was AI-assisted (drafting and synthesis); the log
  excerpts and observed selector behavior are real.

FULL WRITEUP
------------

Mirrored as a Gist with the same content for permanent reference and
easier linking from elsewhere:

https://gist.github.com/istonikula/3c84a651a45509bd38dd79303f4a26d4

-------------------------------------------

[Michael Banack]

There are several separate issues with very similar symptoms, usually triggered by very specific Host/Guest configurations which has made it hard for us to get a handle on this.

It sounds like you have rather thoroughly tried various combinations of the graphics renderer, which is responsible for drawing the 3D graphics off-screen.  We also have a separate module we call the presentation backend which actually draws that off-screen content into the on-screen window.  You can control the presentation backend using these configuration options:

mks.enableGDIPresentation = TRUE

mks.enableDXGIPresentation = FALSE

mks.enableVulkanPresentation = FALSE

mks.enableDX12Presentation = FALSE

mks.enableGLPresentation = FALSE

That configuration will cause us use GDI to draw the actual window contents, which is the most reliable but least performant path.  In theory that might let you keep 3D graphics for the VM at the cost of some performance loss.

You can also try selecting other presentation options to see which of those still show the issue for you.  There should be a log message something like "Started GDI presentation backend." to indicate that it worked.

-------------------------------------------

[richard612]

My own workaround for the input lag bug:

mks.svgaThread.forcePolling = "TRUE"
mks.svgaThread.pollUS = "250"

I've used this only with VMware Workstation 25H2 on Linux -- no idea if effective on 17.6.4 and/or Windows hosts.

For me the bug is unpredictable as to which VMs will be impacted.  Every impacted VM has 3D enabled except one.  That one exception required enabling 3D and applying the above workaround.

Two other VMs (both Windows 11) were essentially identical; one started life as a clone of the other!  Yet one was impacted and one not.  I've diff'd their .vmx files line-by-line to ensure identical configurations.

My host is dual GPU.  Using mks.vk.ForceDevice to select which GPU made no difference.  The usual advice to disable Vulkan presentation and enable SyncPerEndFrame never worked for me.

-------------------------------------------