VMware ESXi 5 host stops sending syslogs to remote server (Splunk)

[coafark]

We've recently installed a Splunk syslog server and are pointing our devices to it.   I noticed that when we stop/start the server (or even service) the logs from all my ESXi 5 hosts stop coming in.

It seems to be a known issue

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003127

Following step 5 does restart the log flowing.  But there is no way that I want to log on to the console and run that every time someone does something in Splunk that needs a restart or that Windows box is rebooted for patches.

I started writing a bash script (below) that I could CRON on the hosts, but found out that ESXi doesn't really have a cron area like ESX4 (not i) had.  If I brute force create it on the hosts, it will get removed with patches.

# this checks to see if the syslog server is writting to the Third party syslog server "SPLUNK"

# it will restart the syslog service if it sees that it has stopped

const='failed to write log'

if [ -e /var/log/.vmsyslogd.err ]; then

tail /var/log/.vmsyslogd.err -n 1 |grep "failed to write log"

  if [ $? = 0 ]; then

     echo "$const  ; Found at the LAST line, restarting syslog server."

  fi

fi

I was going to cron this to run every 15 minutes and if it saw the last line in the log was that it stopped "failed to write log" then I would add a " esxcli system syslog reload " inplace of the echo line.

I have vCenter on a Windows box and would like to have it run a scheduled task against all my hosts (maybe a csv file) and then issue "esxcli system syslog reload " if that is found.  I can't figure out how to do this, can anymore help me out?

I'd like to use what I have, I don't have a vMA or splunks VM either.





[Kyle Gleed]

William Lam posted a script on how do this on his blog site.  He's uses a vCenter alarm to alert in the event connectivity to the loghost is lost.

virtuallyGhetto: Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm

[coafark]

Thanks Gleed.   That link helped me setup an alert. 

Afterwards, I figured it out.

I was using TCP, if I change to UDP, the patch that was supposed to fix this (5.0 patch 3) actually fixes UDP and not TCP... just UDP.

So I changed to UDP and it auto restarts after I test restarting my syslog server (splunk).

Giving you credit since its a good post you linked me to that showed me some other good stuff.