Making sense of ESXi syslog? too many syslog messages!

[markvm2]

I am sending all of my ESXi host syslog messages to Splunk with the VMware Syslog collector. Everything is working great for collecting the syslog messages and sending them to Splunk via the Syslog Collector, but now I am trying to fiigure out how to sort through this mess. Each ESXi host produces a rediculous amount of syslog messages, most of which I don't think I would ever need. Basically we are wanting to audit and keep track of changes made to the vSphere environment, not so much for troubleshooting purposes. Lets say a change is made to a vSwitch or a VM is restarted, how can we easily search through syslog and see this?

In the past week since i started sending the ESXi syslog to Splunk, some of the hosts have over 1,000,000+ events logged in Splunk. I am trying to figure out how to easily search through all of this and make sense of it all. Does anyone have any tips on how to do this?

[dtracey]

Hi Mark,

I think on ESXi 5 you can do it via the vSphere client on the configuration tab of the host:

Under Software | Advanced Settings | Config | HostAgent | log

Decrease the config.HostAgent.log.level to 'panic', 'error', 'warning or 'info' rather than 'verbose.

in 4.1 you used to have to do it by changing the value in the /etc/vmware/hostd/config.xml file - which was probably unsupported!

Hope this helps.

Dan

Edit:  You might still have to edit an .xml file to reduce the vpxa logging - /etc/opt/vmware/vpxa/vpxa.cfg

[markvm2]

The info level does not seem too log much, it looks like a bunch of garbage to me? I guess I will have to index everything and master search strings in Splunk.

What is the difference between the ESXi host logs and the vCenter logs? Does everything in the vCenter logs show up in the ESXi host logs from the vpxa user?

[dtracey]

Mark,

There is some overlap in the vpxa logs of the hosts and vCenter logs (for example when vCenter initiates a task relating to that host such as a vMotion or VM power on).

The vCenter vpxa logs will additionally include stuff that occurs only at the vCenter level, such as vApp actions, permissions changes etc.

Dan 

[Tsjo]

I would look at syslog gathering in another way because, as you said, it's a lot of logs and it all looks like a mess.

Collect the syslog messages from the ESXi hosts, your AD, and so on but only filter on the essential AD events (user logins, service start/stop, ... ) and keep the ESXi logs as backup on the server in case of forensic work in the future.

When that is done, install a VMWare dedicated monitoring application, for example Veeam ONE (http://www.veeam.com/virtual-server-management-one-free.html) or vOPS (http://www.vkernel.com/products/server-explorer/overview).