I always try to stay under a peak usage of 90% in order to avoid too much ballooning or swapping to disk. you can monitor the statistics as you go to make sure you aren't swapping too much. In most cases as long as you have vmware tools installed with the balloon driver, this will help the situation out.
Be aware that this 90% threshold I go by is for larger ESXi hosts with 256GB of RAM or more. When you start dealing with lower RAM capacities in a host, you have to be aware of the reserved space for vmkernel, as well as overheard for the virtual machines as you migrate them.