VMware vSphere

 View Only
  • 1.  VMware Tools Heartbeat failure

    Posted Apr 29, 2011 03:40 PM

    I have a VM that rebooted itself this morning. It had the message below in the events.

    This virtual machine reset by HA Reason: VMware tolls heartbeat failure. A screenshot is saved.

    I looked at that screenshot, but don't know what to do with that. Is there any logs I can look at to determine what happened?

    Thanks,

    Scott



  • 2.  RE: VMware Tools Heartbeat failure

    Posted Apr 29, 2011 04:22 PM

    You must investigate on VM guest OS log to see way VMware Tools has stopped to respond.

    Or disable VMware HA VM restart, if you are sure that the VM is running correct.

    Andre



  • 3.  RE: VMware Tools Heartbeat failure

    Posted Apr 29, 2011 04:44 PM

    Can you tell me what logs I can look at specifically?

    I've looked at the vmware.log file on the VM, /var/log/messages and hostd.log messages on the host. I clearly shows that HA caused a reboot of the VM, but I can't find any information on why it occurred. Are there any VMware Tools logs in the OS? Or something in vCenter logs?

    Thanks,

    Scott



  • 4.  RE: VMware Tools Heartbeat failure

    Posted Apr 29, 2011 05:19 PM

    I have had similar issues when there was a great deal of network activity. You can cut back on sensitivity for virtual machine HA if that is the case.

    Try rvtools http://robware.net I will show a current VM heartbeat status. If you refresh you may see heartbeats change from green to some other color.



  • 5.  RE: VMware Tools Heartbeat failure

    Posted Apr 29, 2011 06:16 PM

    RVTools does show that the heartbeat is alive. Isa there a way to determine what caused the heartbeat to  fail? IO cant find this in any logs. Thanks.



  • 6.  RE: VMware Tools Heartbeat failure

    Posted Apr 29, 2011 06:34 PM

    This will be a standard troubleshooting exercize. Eliminate the things that it isn't. You need to dig through any and all logs, event entries, process trails etc. Correlate times remembering that ESXi uses UTC time. There isn't a specific list. The restart happens because of a loss of heartbeat. There isn't a cause associated with it other than loss of heartbeat. At the time of the restart was anything going on inside the guest. Some usual or unusual process or event. If you have access to physical switch logs anything there.

    Perhaps there is a script (check the PowerCLI or vCLI forums) that can gather heartbeat statistics over a period of time. If at some period of the day or night you have the heartbeat go from green to yellow on a regular basis. Perhaps this was an accumulation of things where at the time of the yellow some other unusual event coincided and caused an extended loss of heartbeat. ??????



  • 7.  RE: VMware Tools Heartbeat failure

    Posted Apr 30, 2011 12:58 PM

    Thanks for the suggestion. I've scoured the logs (vCenter, vmware.log, hostd.log, messages, vpxd.log, vmkernel) but there's no indication of anything right before it decides to restart. Our network was not overloaded at the time.

    Is there a local VMware Tools log in the OS? I definitely need to find out why this occurred, but I don't know what else I can look at?

    Thanks,

    Scott



  • 8.  RE: VMware Tools Heartbeat failure

    Posted Apr 30, 2011 04:52 PM

    Try the windows event logs. Make sure when you go through the ESXi logs that you take into account the time difference between UTC and your local timezone.

    If you have enable debug logging http://kb.vmware.com/kb/1007873 to locate the log file location.



  • 9.  RE: VMware Tools Heartbeat failure

    Posted Apr 30, 2011 05:11 PM

    You may not find a smoking gun. Some temporary spike in app activity on one VM or several. If you have VM HA at the highest level a loss of heartbeat for 30 seconds will cause a restart. Look at resources available to this VM. Is there something about that VM. Resource allocation inbalance across all VMs. Lack of total resources on the host. ????? If you don't find anything specific then I would adjust the VM HA setting to be a little more relaxed.



  • 10.  RE: VMware Tools Heartbeat failure

    Posted Apr 29, 2011 04:24 PM

    Hi Scott,

    You should have a look at vCenter, VM and host logs. You should also check guest's internal log file. Have a look at the following kb for some more details:

    Good luck.

    Regards

    Franck