View Only
  • 1.  Agent child processes hangs

    Posted Aug 20, 2009 07:22 PM

    I am running several shell script monitors and they seem to be affecting the agents, like this:

    [the agent stops reporting]

    23259 /home/hyperic/agent-4.1.2/wrapper/sbin/../../wrapper/sbin/wrapper /home/hyperic
    23276 /usr/jdk/latest/bin/java -Djava.compiler=NONE
    14118 /usr/jdk/latest/bin/java -Djava.compiler=NONE
    16360 /usr/jdk/latest/bin/java -Djava.compiler=NONE

    $ ps -ef | grep java
    root 16360 23276 0 13:39:55 ? 0:00 /usr/jdk/latest/bin/java -Djava.compiler=NONE
    root 23276 23259 0 Aug 19 ? 13:12 /usr/jdk/latest/bin/java -Djava.compiler=NONE
    root 14118 23276 0 Aug 19 ? 0:00 /usr/jdk/latest/bin/java -Djava.compiler=NONE

    I attached the jstack for the last 2 child procs, 16360 & 14118. Please let me know what other info is helpful.

    Thank you,


  • 2.  RE: Agent child processes hangs

    Posted Aug 20, 2009 08:05 PM
    Are you using solaris???

    Then check this thread:

    If this is the case, it's known issue. Actual bug is somewhere in jvm and(grin*) sun is not fixing it on 1.5 series. Modifying is a workaround what I've used successfully.

  • 3.  RE: Agent child processes hangs

    Posted Aug 20, 2009 08:12 PM
    Thank you!

    Indeed, this is solaris, opensolaris snv_89, I guess I'll have to play with

    My java version:

    $ java -version
    java version "1.6.0_06"
    Java(TM) Platform, Standard Edition for Business (build 1.6.0_06-b02)
    Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)

    $ pfexec /usr/java/bin/jinfo 1289
    Attaching to process ID 1289, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 10.0-b22
    Java System Properties: = Java(TM) Platform, Standard Edition for Business
    snmp.mibs.dir = ../../bundles/agent-4.1.2-1053/pdk/mibs
    sun.boot.library.path = /usr/jdk/instances/jdk1.6.0/jre/lib/i386
    java.vm.version = 10.0-b22 = Java HotSpot(TM) Server VM
    file.encoding.pkg = = SUN_STANDARD
    sun.os.patch.level = unknown
    wrapper.key = 6hJbUG_AqE9B1Ipu = 60 = Java Virtual Machine Specification
    user.dir = /home/hyperic/agent-4.1.2/wrapper/sbin
    wrapper.version = 3.3.3
    java.runtime.version = 1.6.0_06-b02

  • 4.  RE: Agent child processes hangs

    Posted Aug 20, 2009 08:44 PM
    Sun Bug ID (or CR :) 6276483 claims that the problem has been fixed in Solaris Nevada build snv_24 java 1.5

    I am running snv_89, java 1.6

    Unfortunately, I am running in a non-global zone, so /usr space is read-only to me so there is nothing I can do with file. This is virtual container environment hosted at Joyent.

    I added the latest jdk to my /opt directory but now java can't find its core:

    /opt/usr/jdk/instances/jdk1.6.0/bin/amd64/java -version
    Error occurred during initialization of VM
    java/lang/NoClassDefFoundError: java/lang/Object

    Will dig some more.

  • 5.  RE: Agent child processes hangs
    Best Answer

    Posted Aug 20, 2009 09:16 PM
    Ah, I didn't know that there is some fixes. I thought sun is not gonna back port the fix to 1.5. Need to see if this is the fix which will resolve jvm issues.

    I don't know if 1.6 is supported by hyperic. Plain java stuff 'should' be compatible, but there is also some JNI stuff(sigar) which may raise issues with java6.

    This error message may become of some missing stuff. If I remember jvm installation for amd64 is based on two different installation packages. 64 stuff is just add-on. If you downloaded it from sun, use both - basic installation package for x86 and amd64 bit add-on package.

    Do you know that it's also possible to package jre withing agent installation. If there is a 'jre' directory directly under agent root(where you have data, conf, etc directories) startup scripts will find java from there. I've always used this approach to create package which can be extracted to all similar platforms. This way I always know that my hq installation is similar throughout my system. Different solaris installation always has different jvm versions.

  • 6.  RE: Agent child processes hangs

    Posted Aug 20, 2009 09:46 PM
    Thank you for your very helpful suggestions, I am now running a local instance of java 6 with hyperic - will see if/when the problem repeats itself. Packaging jre inside hyperic agent seems very sensible, I'll try it on another instance of my agent - I am running about 70 of them, in prod.