VMware workstation is a Type 2 hypervisor (runs on an OS).
ESXi is a Type 1 hypervisor (runs directly on bare metal).
You cannot install a Type 1 hypervisor on a Type 2 hypervisor. The error you are getting is due to how ESXi needs to communicate directly with bare metal resources.
VMware workstation is meant to allow you to run VMs on your client workstation without having to install a Type 1 hypervisor.
If you are wanting to experiment with a ESXi, you will want to install that on a bare metal build (a server for example).
Same logic applies to Hyper-V - it is already a Type 1 hypervisor, you cannot build a virtual host (example bare metal) inside that environment - only virtual machines