Have same issue too only with Linux (Debian 11,x x64) guest, Windows VMs start OK.
The "funny" part is that Linux has no UI at all, getting the error before not just UI starts but I think even before kernel starts, seems like VMWare BIOS tries to do something and crashes.
As for "Edit --> Preferences --> Display" it was unchecked to start with .
Also some people suggested to change mks.enable3d = "TRUE" to "FALSE" in .vmx file. But I did not had it there at all, adding mks.enable3d = "FALSE" did not helped.
One observation although, the problem happened twice with VMware 17.5 (that was working fine before) , that's was actually the reason for me to upgrade to 17.6 , that's why remember.
My suspicion that it started not due 17.6 update , but due some Windows 11 23H2 update on my host . Kind of suspect KB5041585 as it was installed about the time I started to get issues , but can't be sure, it can be NVIDIA driver (I have GeForce RTX4070) or something else .
One more thing - I am not running Windows Hyper-V on host (disabled) as it prevents running docker inside my guest VMs , maybe related.
From log:
---------------------------
2024-09-12T22:10:47.144Z In(05) mks MKSControlMgr: connected
2024-09-12T22:10:47.157Z In(05) svga SWBScreen: Screen 0 Defined: xywh(0, 0, 640, 480) flags=0x3
2024-09-12T22:10:47.160Z In(05) mks MKS-VMDB: VMDB requested a screenshot
2024-09-12T22:10:47.163Z In(05) svga MKSScreenShotMgr: Taking a screenshot
2024-09-12T22:10:47.183Z In(05) mks SWBWindow: Number of MKSWindows changed: 0 rendering MKSWindow(s) of total 1.
2024-09-12T22:10:47.183Z In(05) mks SWBWindow: Window 1 Defined: src screenId=-1, src xywh(0, 0, 640, 480) dest xywh(0, 0, 640, 480) pixelScale=1, flags=0x8
2024-09-12T22:10:47.183Z In(05) mks MKS-HWinMux: Backend Switch Queued
2024-09-12T22:10:47.184Z In(05) mks DXGIPresentation: Enumerating adapter 0
2024-09-12T22:10:47.184Z In(05) mks DXGIPresentation: `NVIDIA GeForce RTX 4070` vendor=0x10de device=0x2786 revision=161
2024-09-12T22:10:47.184Z In(05) mks DXGIPresentation: video=12011MB system=0MB shared=32722MB
2024-09-12T22:10:47.184Z In(05) mks DXGIPresentation: LOCAL budget 11243M usage 9M avail 5749M res 0M
2024-09-12T22:10:47.184Z In(05) mks DXGIPresentation: NON-LOCAL budget 31954M usage 3M avail 16105M res 0M
2024-09-12T22:10:47.213Z In(05) mks DXGIPresentation: Using device unknown; adapter `NVIDIA GeForce RTX 4070`
2024-09-12T22:10:47.213Z In(05) mks DXGIPresentation: Enumerating adapter 1
2024-09-12T22:10:47.213Z In(05) mks DXGIPresentation: `Microsoft Basic Render Driver` vendor=0x1414 device=0x008c revision=0
2024-09-12T22:10:47.213Z In(05) mks DXGIPresentation: video=0MB system=0MB shared=32722MB
2024-09-12T22:10:47.213Z In(05) mks DXGIPresentation: LOCAL budget 31954M usage 0M avail 16105M res 0M
2024-09-12T22:10:47.214Z In(05) mks DXGIPresentation: NON-LOCAL budget 0M usage 0M avail 0M res 0M
2024-09-12T22:10:47.216Z In(05) mks DXGIPresentation: NVIDIA driver version 561.09 (branch r560_88)
2024-09-12T22:10:47.217Z In(05) mks MKS-HWinMux: Started DXGI presentation backend.
2024-09-12T22:10:47.219Z In(05) mks SWBWindow: Number of MKSWindows changed: 1 rendering MKSWindow(s) of total 2.
2024-09-12T22:10:47.219Z In(05) mks SWBWindow: Window 0 Defined: src screenId=-1, src xywh(0, 0, 640, 480) dest xywh(0, 0, 640, 480) pixelScale=1, flags=0x6
2024-09-12T22:10:47.223Z In(05) mks DX11BasicResourceSharedAllocateResources: LOCAL budget 11243M usage 16M avail 5749M res 0M
2024-09-12T22:10:47.223Z In(05) mks DX11BasicResourceSharedAllocateResources: NON-LOCAL budget 31954M usage 4M avail 16105M res 0M
2024-09-12T22:10:47.223Z Wa(03) mks DX11BasicResourceSharedAllocateResources: hr=E_INVALIDARG
2024-09-12T22:10:47.227Z Wa(03) svga ----Win32 exception detected, exceptionCode 0xc0000005 (access violation)----
2024-09-12T22:10:47.227Z Wa(03) svga ExceptionAddress 0x7ff706999865 eflags 0x00010206
2024-09-12T22:10:47.227Z Wa(03) svga rwFlags 0 badAddr 0
2024-09-12T22:10:47.227Z Wa(03) svga rax 0x15df248ba0d8 rbx 0x15f811b3370 rcx 0
2024-09-12T22:10:47.227Z Wa(03) svga rdx 0x15df248ba0d8 rsi 0x15f810c7760 rdi 0x15fbf5e3f30
2024-09-12T22:10:47.227Z Wa(03) svga r8 0x30a4dff840 r9 0x15f811b01c0 r10 0xffee0d1d32e
2024-09-12T22:10:47.227Z Wa(03) svga r11 0x404010000400 r12 0 r13 0x1
2024-09-12T22:10:47.227Z Wa(03) svga r14 0 r15 0x3
2024-09-12T22:10:47.227Z Wa(03) svga rip 0x7ff706999865 rsp 0x30a4dff7f0 rbp 0x1
2024-09-12T22:10:47.227Z Wa(03) svga LastBranchToRip 0 LastBranchFromRip 0
2024-09-12T22:10:47.227Z Wa(03) svga LastExceptionToRip 0 LastExceptionFromRip 0
2024-09-12T22:10:47.227Z Wa(03) svga The following data was delivered with the exception:
2024-09-12T22:10:47.227Z Wa(03) svga -- 0
2024-09-12T22:10:47.227Z Wa(03) svga -- 0
2024-09-12T22:10:47.227Z In(05) svga CoreDump: Minidump file Q:\Virtual Machines\Debian 11.x 64-bit\vmware-vmx.dmp exists. Rotating ...
2024-09-12T22:10:47.227Z In(05) svga CoreDump: Current max dump count is 5.
2024-09-12T22:10:47.228Z Wa(03) svga CoreDump: Writing minidump to Q:\Virtual Machines\Debian 11.x 64-bit\vmware-vmx.dmp
2024-09-12T22:10:47.231Z In(05) vcpu-0 SVGA: Registering MemSpace at 0xe8000000(0x0) and 0xfe000000(0x0)
2024-09-12T22:10:47.231Z In(05) vcpu-0 SVGA: FIFO is already mapped
2024-09-12T22:10:47.239Z In(05) vcpu-0 SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)
2024-09-12T22:10:47.239Z In(05) vcpu-0 SVGA: FIFO is already mapped
2024-09-12T22:10:47.336Z In(05) svga CoreDump: including module base 0x0x7ff706740000 size 0x0x01c0a000