This had me crazy for a few days, finally tracked it down, sharing for everyone's benefit. If you have a recent linux distro with kernels 6.9 or 6.10, you will need to modify the vmx file like this
mks.enableDX12Renderer = "FALSE"
mks.enableGLRenderer = "TRUE"
Add one line space at the end and save the file. Next go into VM settings and tick the box for 3d acceleration. Now you should be able to boot