To run VMs inside an ESXi VM, check the "Expose hardware virtualization to the guest OS" in the CPU setting of the VM. This adds the line
vhv.enable = "TRUE"
to the vmx configuration file.
That is the ESXi VM equivalent of enabling Intel VT-x on a physical server.
For the Hardware Enforced Data Execution Prevention", this is the XD (Execute Disable) in the system BIOS of the physical server. It might be called DEP or XD or something else so check the manual of the physical server.