ESXi

 View Only
  • 1.  VMware ESXi 5 host stops sending syslogs to remote server (Splunk)

    Posted May 08, 2013 07:54 PM

    We've recently installed a Splunk syslog server and are pointing our devices to it.   I noticed that when we stop/start the server (or even service) the logs from all my ESXi 5 hosts stop coming in.

    It seems to be a known issue

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003127

    Following step 5 does restart the log flowing.  But there is no way that I want to log on to the console and run that every time someone does something in Splunk that needs a restart or that Windows box is rebooted for patches.

    I started writing a bash script (below) that I could CRON on the hosts, but found out that ESXi doesn't really have a cron area like ESX4 (not i) had.  If I brute force create it on the hosts, it will get removed with patches.

    # this checks to see if the syslog server is writting to the Third party syslog server "SPLUNK"

    # it will restart the syslog service if it sees that it has stopped

    const='failed to write log'

    if [ -e /var/log/.vmsyslogd.err ]; then

    tail /var/log/.vmsyslogd.err -n 1 |grep "failed to write log"

      if [ $? = 0 ]; then

         echo "$const  ; Found at the LAST line, restarting syslog server."

      fi

    fi

    I was going to cron this to run every 15 minutes and if it saw the last line in the log was that it stopped "failed to write log" then I would add a " esxcli system syslog reload " inplace of the echo line.

    I have vCenter on a Windows box and would like to have it run a scheduled task against all my hosts (maybe a csv file) and then issue "esxcli system syslog reload " if that is found.  I can't figure out how to do this, can anymore help me out?

    I'd like to use what I have, I don't have a vMA or splunks VM either.




  • 2.  RE: VMware ESXi 5 host stops sending syslogs to remote server (Splunk)
    Best Answer

    Broadcom Employee
    Posted May 08, 2013 08:20 PM

    William Lam posted a script on how do this on his blog site.  He's uses a vCenter alarm to alert in the event connectivity to the loghost is lost.

    virtuallyGhetto: Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm



  • 3.  RE: VMware ESXi 5 host stops sending syslogs to remote server (Splunk)

    Posted May 09, 2013 06:51 PM

    Thanks Gleed.   That link helped me setup an alert. 

    Afterwards, I figured it out.

    I was using TCP, if I change to UDP, the patch that was supposed to fix this (5.0 patch 3) actually fixes UDP and not TCP... just UDP.

    So I changed to UDP and it auto restarts after I test restarting my syslog server (splunk).

    Giving you credit since its a good post you linked me to that showed me some other good stuff.