I would suggest starting with my post, as it is the least disruptive to the VM's and the ESXi host. Also, if you perhaps could share a screenshot of your configuration -> networking via the vSphere client, we might get a better understanding of your enviroment.
A few things to check / clarify:
1) Does the ESXi host lose connectivity as well when the VM's lose connectivity?
2) Does only one VM lose connectivity at a time? So, if one loses, the other 4 still work?
3) I'm not really sure what you mean by X'ed out. Do you mean that in Windows, the NIC is in a disconnected state? If so, check on the settings of the NIC if the NIC is still connected:
4) When you say "reboot the server", do you mean the VM or the entire ESXi host?
5) Unlikely, but do the physical switchport(s) the ESXi host is attached to, report any errors?
Most of the festivities have already ended here, so back to work now :smileyhappy:.