SPLUNK for VMware

[JDLangdon]

Has anyone had any luck at getting SPLUNK for VMware setup? I'm trying to setup the VMware sourcetypes but I'm obviously doing something wrong configured correctly.

________________________________

Jason D. Langdon

[travesty]

I'm also trying to setup the VMware application for Splunk but can't get it to work. Can anybody provide some problems they ran into and the resolutions to these problems?

[vanberg]

I was able to get it up and running for our setup in about 45 min's (forgot to enable the firewall settings)

Works great

Eric

[JDLangdon]

I was able to get it up and running for our setup in about 45 min's (forgot to enable the firewall settings)

I can get the syslog portion working fine but I cannot get the VMware api's to work correctly. I have two guys from SPLUNK look at my log files and config files and neither of them could offer any workable suggestions either.

________________________________

Jason D. Langdon

[travesty]

Where should I be looking for firewall settings? I believe the problem I've run into is that VCenter is expecting a certificate from the Splunk server but for whatever reason the Splunk server isn't sending one. I'm stuck.

[vanberg]

Depends....where are you in the setup? Are you installing on Windows? What part are you stuck at?

Assuming its windows:

goto a command prompt and type

echo%JAVAHOME%
echo %SPLUNK_HOME%

And paste the results in here...we can start from there.

Also, please paste in the url from your vmware.conf file (C:\Program Files\Splunk\etc\apps\vmware\default)

I will try and help you get it up and running.

Eric

[travesty]

I have Splunk installed on a Linux distribution, Fedora 10. Both of those variables are set, so when I ran the command a blank line was followed. The url from the vmware.conf file is, address of VCenter)/sdk

[vanberg]

I dont have a fedora box setup, but I should still be able to help get it going, plus i might build one up.

More questions:

1. Can you see the main splunk page when you go through a web browser?

2. Do you have the VMWare Application already installed?

3. From the splunk page, restart the splunk service.

3. Run the command cd $SPLUNK_HOME/etc/apps/vmware

4. Run the command java -jar lib/splunk.jar

5. Paste the first 20 lines from step 4.

Eric

[travesty]

1. yes

2. yes, it was installed from splunkbase through our splunk interface

Here is what I grabbed from the test.

Started

Caught Exception : Exception : org.apache.axis.AxisFault Message : ; nested exception is:

gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL StackTrace :

AxisFault

faultCode: {[http://schemas.xmlsoap.org/soap/envelope/}Server.userException|http://schemas.xmlsoap.org/soap/envelope/%7DServer.userException]

faultSubcode:

faultString: gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

faultActor:

faultNode:

faultDetail:

{[http://xml.apache.org/axis/}stackTrace:gnu.javax.net.ssl.provider.AlertException|http://xml.apache.org/axis/%7DstackTrace:gnu.javax.net.ssl.provider.AlertException]: UNEXPECTED_MESSAGE: remotely generated; FATAL

at gnu.javax.net.ssl.provider.SSLEngineImpl.unwrap(libgcj.so.9)

at javax.net.ssl.SSLEngine.unwrap(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl.doHandshake(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl$SocketOutputStream.write(libgcj.so.9)

at java.io.BufferedOutputStream.flush(libgcj.so.9)

at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:516)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.retrieveServiceContent(VimBindingStub.java:23449)

at com.vmware.apputils.vim.ServiceConnection.connect(ServiceConnection.java:54)

at com.vmware.apputils.vim.ServiceUtil.clientConnect(ServiceUtil.java:36)

at com.vmware.apputils.AppUtil.connect(AppUtil.java:389)

at com.splunk.VMWareHostConnection.init(Splunk4VMI.java:275)

at com.splunk.Splunk4VMI.init(Splunk4VMI.java:393)

at com.splunk.Splunk4VMI.main(Splunk4VMI.java:573)

{[http://xml.apache.org/axis/}hostname:APP-07-SPLUNK.gripa.local|http://xml.apache.org/axis/%7Dhostname:APP-07-SPLUNK.gripa.local]

gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.retrieveServiceContent(VimBindingStub.java:23449)

at com.vmware.apputils.vim.ServiceConnection.connect(ServiceConnection.java:54)

at com.vmware.apputils.vim.ServiceUtil.clientConnect(ServiceUtil.java:36)

at com.vmware.apputils.AppUtil.connect(AppUtil.java:389)

at com.splunk.VMWareHostConnection.init(Splunk4VMI.java:275)

at com.splunk.Splunk4VMI.init(Splunk4VMI.java:393)

at com.splunk.Splunk4VMI.main(Splunk4VMI.java:573)

Caused by: gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

at gnu.javax.net.ssl.provider.SSLEngineImpl.unwrap(libgcj.so.9)

at javax.net.ssl.SSLEngine.unwrap(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl.doHandshake(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl$SocketOutputStream.write(libgcj.so.9)

at java.io.BufferedOutputStream.flush(libgcj.so.9)

at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:516)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

...16 more

Exception running : Splunk4VMI

Caught Exception : Exception : org.apache.axis.AxisFault Message : ; nested exception is:

gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL StackTrace :

AxisFault

faultCode: {[http://schemas.xmlsoap.org/soap/envelope/}Server.userException|http://schemas.xmlsoap.org/soap/envelope/%7DServer.userException]

faultSubcode:

faultString: gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

faultActor:

faultNode:

faultDetail:

{[http://xml.apache.org/axis/}stackTrace:gnu.javax.net.ssl.provider.AlertException|http://xml.apache.org/axis/%7DstackTrace:gnu.javax.net.ssl.provider.AlertException]: UNEXPECTED_MESSAGE: remotely generated; FATAL

at gnu.javax.net.ssl.provider.SSLEngineImpl.unwrap(libgcj.so.9)

at javax.net.ssl.SSLEngine.unwrap(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl.doHandshake(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl$SocketOutputStream.write(libgcj.so.9)

at java.io.BufferedOutputStream.flush(libgcj.so.9)

at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:516)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.retrieveServiceContent(VimBindingStub.java:23449)

at com.vmware.apputils.vim.ServiceConnection.connect(ServiceConnection.java:54)

at com.vmware.apputils.vim.ServiceUtil.clientConnect(ServiceUtil.java:36)

at com.vmware.apputils.AppUtil.connect(AppUtil.java:389)

at com.splunk.VMWareHostConnection.init(Splunk4VMI.java:275)

at com.splunk.Splunk4VMI.init(Splunk4VMI.java:393)

at com.splunk.Splunk4VMI.main(Splunk4VMI.java:573)

{[http://xml.apache.org/axis/}hostname:APP-07-SPLUNK.gripa.local|http://xml.apache.org/axis/%7Dhostname:APP-07-SPLUNK.gripa.local]

[JDLangdon]

Looks like you're using the wrong version of java.

________________________________

Jason D. Langdon

[travesty]

I'm using version 1.5, which is supposed to be compatible.

[JDLangdon]

which did you install, java or jdk? I had to download and install jdk1.6.0.13 before it would work.

________________________________

Jason D. Langdon

[travesty]

We upgraded our Java version and installed the matching JDK version. It appears to have fixed the problem. How long does it take for Splunk to index all the data in order for me to see results in the VMware dashboards?

[JDLangdon]

I never did get the VMware dashboards working.

________________________________

Jason D. Langdon

[travesty]

It ran over night and it collected a good deal of data but none of the data was populated for the VMware dashboards. I'll install some ESX updates to see if that solves the problem.

[SimonSplunk]

So you're seeing data when you search: sourcetype=vmware_api ?

If so, whats the latest timestamp you see? Can you click on "Report on Results"? Do you see fields on the left?

Regarding the dashboards, do you even see them in the pulldown? If not, are you using LDAP or not logging in as "admin"?

[travesty]

All of the saved searches for the VMware app were disabled, so none of the information was being populated in the dashboards. Is this by default? I've enabled all of the saved searches and my dashboards are now being populated.

[SimonSplunk]

Hi there - Simon from Splunk here. I would try upgrading to Java 1.6 if you can.

You can also contact me directly: simon at splunk dot com