Agent child processes hangs

[davidk_hyperic]

Hi,

I am running several shell script monitors and they seem to be affecting the agents, like this:

[the agent stops reporting]

23259 /home/hyperic/agent-4.1.2/wrapper/sbin/../../wrapper/sbin/wrapper /home/hyperic
23276 /usr/jdk/latest/bin/java -Djava.compiler=NONE -Djava.security.auth.login.config
14118 /usr/jdk/latest/bin/java -Djava.compiler=NONE -Djava.security.auth.login.config
16360 /usr/jdk/latest/bin/java -Djava.compiler=NONE -Djava.security.auth.login.config

$ ps -ef | grep java
root 16360 23276 0 13:39:55 ? 0:00 /usr/jdk/latest/bin/java -Djava.compiler=NONE -Djava.security.auth.login.config
root 23276 23259 0 Aug 19 ? 13:12 /usr/jdk/latest/bin/java -Djava.compiler=NONE -Djava.security.auth.login.config
root 14118 23276 0 Aug 19 ? 0:00 /usr/jdk/latest/bin/java -Djava.compiler=NONE -Djava.security.auth.login.config

I attached the jstack for the last 2 child procs, 16360 & 14118. Please let me know what other info is helpful.


Thank you,

David

[jvalkeal_hyperi]

Are you using solaris???

Then check this thread:
http://communities.vmware.com/message/1934522#1934522

If this is the case, it's known issue. Actual bug is somewhere in jvm and(grin*) sun is not fixing it on 1.5 series. Modifying java.security is a workaround what I've used successfully.

[davidk_hyperic]

Thank you!

Indeed, this is solaris, opensolaris snv_89, I guess I'll have to play with java.security.

My java version:


$ java -version
java version "1.6.0_06"
Java(TM) Platform, Standard Edition for Business (build 1.6.0_06-b02)
Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)

$ pfexec /usr/java/bin/jinfo 1289
Attaching to process ID 1289, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 10.0-b22
Java System Properties:

java.runtime.name = Java(TM) Platform, Standard Edition for Business
snmp.mibs.dir = ../../bundles/agent-4.1.2-1053/pdk/mibs
sun.boot.library.path = /usr/jdk/instances/jdk1.6.0/jre/lib/i386
java.vm.version = 10.0-b22
java.vm.name = Java HotSpot(TM) Server VM
file.encoding.pkg = sun.io
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
wrapper.key = 6hJbUG_AqE9B1Ipu
sun.net.inetaddr.ttl = 60
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /home/hyperic/agent-4.1.2/wrapper/sbin
wrapper.version = 3.3.3
java.runtime.version = 1.6.0_06-b02

[davidk_hyperic]

Sun Bug ID (or CR :) 6276483 claims that the problem has been fixed in Solaris Nevada build snv_24 java 1.5

I am running snv_89, java 1.6

Unfortunately, I am running in a non-global zone, so /usr space is read-only to me so there is nothing I can do with java.security file. This is virtual container environment hosted at Joyent.

I added the latest jdk to my /opt directory but now java can't find its core:

/opt/usr/jdk/instances/jdk1.6.0/bin/amd64/java -version
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object

Will dig some more.

[jvalkeal_hyperi]

Ah, I didn't know that there is some fixes. I thought sun is not gonna back port the fix to 1.5. Need to see if this is the fix which will resolve jvm issues.

I don't know if 1.6 is supported by hyperic. Plain java stuff 'should' be compatible, but there is also some JNI stuff(sigar) which may raise issues with java6.

This error message may become of some missing stuff. If I remember jvm installation for amd64 is based on two different installation packages. 64 stuff is just add-on. If you downloaded it from sun, use both - basic installation package for x86 and amd64 bit add-on package.

Do you know that it's also possible to package jre withing agent installation. If there is a 'jre' directory directly under agent root(where you have data, conf, etc directories) startup scripts will find java from there. I've always used this approach to create package which can be extracted to all similar platforms. This way I always know that my hq installation is similar throughout my system. Different solaris installation always has different jvm versions.

[davidk_hyperic]

Thank you for your very helpful suggestions, I am now running a local instance of java 6 with hyperic - will see if/when the problem repeats itself. Packaging jre inside hyperic agent seems very sensible, I'll try it on another instance of my agent - I am running about 70 of them, in prod.

Cheers,

David