To change resolution for your remote desktop session, I think you should select it before starting the session, i.e. before you start the remote session, expand Options | Display | Display configuration.
To change the VMs resolution, connect to ESXi host, Open console of the VM from there and change the resolution the same way as above. Hope this helps.