This is a bug in VMware, in fact, it is several bugs in VMware. If you create a macOS guest on ESXi 6, 6.5, 6.7 you can replicate one or more display related bugs (some of which are interactions with VMware fusion).
1. If you set a custom resolution for a macOS guest and disconnect the virtual console while it is running, then disconnect (for example using VMware fusion as the console, quit fusion then start it again and reconnect,) the console will physically come back as an 800x600 window but macOS will believe it is rendering at 1024x768 regardless of tools version, fusion version, video ram size for the VM. If you instead resize the VM and shut it down, then when you boot it back up after quitting and restarting fusion (or VMRC), it will remember the console size and things will render 1:1 Instead of being scaled and blurry.
1b. This will also prevent console resizing above 1024x768 if you try to do a Remote Desktop or VNC connection to the running VM, so if you connect that way to a running VM and drag the window to be larger, you may be scaling the 1024x768 display window to an arbitrary size, which can cause a blurry appearance.
2. If you look at the VMware tools preferences file in ~/Library/Preferences you’ll see a high DPI true/false that defaults to true. This feature is completely broken and will not behave right in session or across sessions, causing weird issues with console resizing, effective display resolution, etc. It also puts titanic CPU load on the machine when enabled because you end up doing a scale up and a scale down in CPU instead of GPU. Change the setting to false to have predictable behavior.
3. Fusion does not properly remember window locations across sessions along with its inability to remember console sizes for running macOS guests. Depending on whether they are hosted on ESXi or locally, you’ll see slightly different behavior with whether the windows always re-open to the center of the screen or to their last location, but if you have a console window size that does not fit when it is relocated to the center of the screen, you can also get weird interactions with display resolution and console size that cause bad rendering of the guest GUI.