Check if the 3D acceleration is successfully enabled on the VM. On the dxdiag in the VM, it should show Feature Level 11_0 support. Assuming that the host is Windows 10/11; if the 3D acceleration is not enabled this could be due to the change in host GPU requirement. My best guess is version 17.5 requires Feature level 12_0 or newer on the host GPU as 16.x already required Feature Level 11_1.
Nvidia Maxwell Gen 2 or newer supports Feature Level 12_1. Maxwell Gen 1 or earlier supports Feature Level 11_0 only.
A possible workaround to try is to add these lines to the VM vmx file (if host GPU upgrade is not possible at the moment)
mks.enableDX12Renderer = "FALSE"
mks.enableDX11Renderer = "FALSE"
mks.enableGLRenderer = "TRUE"
If it works add it to the %PROGRAMDATA%\VMware Workstation\config.ini so that this workaround does not need to be applied on individual VM vmx files.
Note this workaround is NOT recommended for Intel integrated GPUs as rendering issues will be quite obvious even with the Windows 10/11 VM bootup. I don't know what the effect is on AMD GPUs.
If the issue is not the host GPU/3D acceleration in the VM, the issue could be Hyper-V is enabled on the Windows 10/11 host; or the host CPU has e-cores and only e-cores are being used. These issues have different solutions/workarounds. You could attach the vmware.log and mksSandbox.log files of a VM in a subsequent reply; or you could search this forum for issues regarding disabling Hyper-V on the host, e-cores, etc.