DX Application Performance Management

 View Only
Expand all | Collapse all

Introscope enterprise manager service stopped

Haruhiko Davis

Haruhiko DavisSep 05, 2014 10:37 AM

  • 1.  Introscope enterprise manager service stopped

    Posted Sep 03, 2014 10:14 AM

    One one of our introscope Enterprise manager server the EM service stopped , and when  we saw the IntroscopeEnterpriseManager.log , we saw the below error during the same time:

     

    9/03/14 03:17:37.844 AM CDT [ERROR] [TT Optimization] [Manager] Uncaught Exception in Enterprise Manager:  In thread TT Optimization and the message is java.lang.OutOfMemoryError: PermGen space

     

    Does anybody know what this error means?



  • 2.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 10:23 AM
    What version of APM?

     


    What OS and version?

    Running 32- or 64- bit?

    Min/Max heap size?

    Total physical memory?


  • 3.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 10:49 AM

    Hi Davis,

     

    Sorry to ask, But where do i see all this information?



  • 4.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 10:59 AM
    On Windows, run 'msinfo32'. On *nix, 'uname -a'.

    Your JVM settings should be listed in the EM LAX file in the root directory where it is installed. If you're running as a Windows service, look at <em_home>\bin\EMService.conf.

    Remember to set the min/max heap the same. Check to see if you have 'MaxPermSize' property set and what its value is. Please post your settings here along with the EM log. Block out any confidential info such as IP addresses and host names.


  • 5.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 11:26 AM
      |   view attached

    Its a 64 bit OS. Windows 2008 R2.

     

    # Java Additional Parameters

    #wrapper.java.additional.1=

    wrapper.java.additional.1=-Djava.awt.headless=false

    wrapper.java.additional.2=-XX:MaxPermSize=512m

     

    # Initial Java Heap Size (in MB)
    wrapper.java.initmemory=6144


    # Maximum Java Heap Size (in MB)
    wrapper.java.maxmemory=6144

     

    Attached the EMService.log

    Attachment(s)



  • 6.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 01:51 PM

    HelloRohit123

    Your Java Additional parameters should look like this:

     

    # Java Additional Parameters

    #wrapper.java.additional.1=

    wrapper.java.additional.1=-Djava.awt.headless=false

    wrapper.java.additional.2=-XX:MaxPermSize=256m

    wrapper.java.additional.3=-Dmail.mime.charset=UTF-8

    wrapper.java.additional.4=-Dorg.owasp.esapi.resources=./config/esapi

    wrapper.java.additional.5=-Xss512k

     

    I'll read through your logs shortly.



  • 7.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 01:53 PM

    Hi Davis,

     

    Is there any recommendation from CA to have those values for the parameters listed above in the EM config? I need to justify this to my team.

     

    Thanks



  • 8.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 02:07 PM

    Those are the defaults and they shouldn't be changed.

     

    The only thing you should change is the min/max heap. Allow for 2GB RAM free for OS operations. Make sure the min/max heap are the same. For details why, please refer the Sizing and Performance Guide in the bookshelf.



  • 9.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 03:40 PM

    We have all the above parameters set with the above values, and also the Min and Max heap size are set around 6 GB aand they both are same.



  • 10.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 03:41 PM

    # Java Additional Parameters

    #wrapper.java.additional.1=

    wrapper.java.additional.1=-Djava.awt.headless=false

    wrapper.java.additional.2=-XX:MaxPermSize=512m

    wrapper.java.additional.3=-Dmail.mime.charset=UTF-8

    wrapper.java.additional.4=-showversion

    wrapper.java.additional.5=-XX:+UseConcMarkSweepGC

    wrapper.java.additional.6=-XX:+UseParNewGC

    wrapper.java.additional.7=-XX:CMSInitiatingOccupancyFraction=50

    wrapper.java.additional.8=-Xss256k

    wrapper.java.additional.9=-server



  • 11.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 04:45 PM
    Property '-Xss' should be set to '512k' for 64-bit OS.


  • 12.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 05:24 PM

    Hi Davis,

     

    Thanks a lot for the reply. Can you let me know what this property does?

     

    Thanks



  • 13.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 05:30 PM

    .\java -X
        -Xmixed           mixed mode execution (default)
        -Xint             interpreted mode execution only
        -Xbootclasspath:<directories and zip/jar files separated by ;>
                          set search path for bootstrap classes and resources
        -Xbootclasspath/a:<directories and zip/jar files separated by ;>
                          append to end of bootstrap class path
        -Xbootclasspath/p:<directories and zip/jar files separated by ;>
                          prepend in front of bootstrap class path
        -Xdiag            show additional diagnostic messages
        -Xnoclassgc       disable class garbage collection
        -Xincgc           enable incremental garbage collection
        -Xloggc:<file>    log GC status to a file with time stamps
        -Xbatch           disable background compilation
        -Xms<size>        set initial Java heap size
        -Xmx<size>        set maximum Java heap size
        -Xss<size>        set java thread stack size
        -Xprof            output cpu profiling data
        -Xfuture          enable strictest checks, anticipating future default
        -Xrs              reduce use of OS signals by Java/VM (see documentation)
        -Xcheck:jni       perform additional checks for JNI functions
        -Xshare:off       do not attempt to use shared class data
        -Xshare:auto      use shared class data if possible (default)
        -Xshare:on        require using shared class data, otherwise fail.
        -XshowSettings    show all settings and continue
        -XshowSettings:all
                          show all settings and continue
        -XshowSettings:vm show all vm related settings and continue
        -XshowSettings:properties
                          show all property settings and continue
        -XshowSettings:locale
                          show all locale related settings and continue

    The -X options are non-standard and subject to change without notice.



  • 14.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 03, 2014 05:33 PM

    I would also recommend changing your conf file to match the one i posted as those are the recommended values for running a 64-bit JVM.



  • 15.  Re: Introscope enterprise manager service stopped

    Posted Sep 03, 2014 05:57 PM

    Hi Davis,

     

    We have these values set correct in our Test boxes:

     

    # Java Additional Parameters

    #wrapper.java.additional.1=

    wrapper.java.additional.1=-Djava.awt.headless=false

    wrapper.java.additional.2=-XX:MaxPermSize=256m

    wrapper.java.additional.3=-Dmail.mime.charset=UTF-8

    wrapper.java.additional.4=-Dorg.owasp.esapi.resources=./config/esapi

    wrapper.java.additional.5=-showversion

    wrapper.java.additional.6=-XX:+UseConcMarkSweepGC

    wrapper.java.additional.7=-XX:+UseParNewGC

    wrapper.java.additional.8=-XX:CMSInitiatingOccupancyFraction=50

    wrapper.java.additional.9=-Xss512k

    wrapper.java.additional.10=-Dsearch.typeviewer.clamp.size=500

     

    I will try to replicate these in other systems. As a workaround we are renaming the log directory and traces directory and restart the EM service, then we get rid of the issue for at least 3 months. Not sure how good is this workaround.



  • 16.  Re: Introscope enterprise manager service stopped
    Best Answer

    Broadcom Employee
    Posted Sep 03, 2014 06:01 PM

    I recommend removing the ones that don't match the ones I have because those are the recommended settings for a 32-bit JVM.

     

    You can see this if you look at Introscope_Enterprise_Manager.lax.



  • 17.  Re: Introscope enterprise manager service stopped

    Posted Sep 04, 2014 10:10 AM

    Hi Davis,

     

    Can you let me know where do i find Introscope_Enterprise_Manager.lax?



  • 18.  Re: Introscope enterprise manager service stopped

    Posted Sep 04, 2014 10:13 AM

    I am able to find the lax file. So is this the file where the recommended values of those properties are mentioned?



  • 19.  Re: Introscope enterprise manager service stopped

    Posted Sep 04, 2014 10:16 AM

    Found the recommended values as well.

     

    #   LAX.NL.JAVA.OPTION.ADDITIONAL

    #   -----------------------------

    #   place any optional Java arguments (such as heap size) here. The following switches are recommended for 32-bit Sun JVMs:

    #   -showversion -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=50 -XX:MaxPermSize=256m -Xss256k

    #   Note that extra switches(-showversion -XX:+UseConcMarkSweepGC -XX:+UseParNewGC) to default EM JVM args are added for Sun 32-bit JVMs only

    #   If you are using both Hot Failover and the CEM UI,

    #   uncomment this line when starting the Secondary Introscope Enterprise Manager (or second Primary),

    #   and comment out the other one.

    #   Note that this is similar to the active line except for the last switch (-Dosgi.clean=false) and extra switches for Sun 32-bit JVMs only

    #   Note update Xss value to 512k for 64-bit platforms

    #   Set the lax java option additional values like: -Xms512m -Xmx1024m -Djava.awt.headless=false -XX:MaxPermSize=256m -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi  -Xss512k



  • 20.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 04, 2014 10:55 AM
    That is exactly what I wanted you to see.

    Change conf file to match the settings in the lax file and restart your EM. Post any errors you see; ignore warnings.


  • 21.  Re: Introscope enterprise manager service stopped

    Posted Sep 05, 2014 10:35 AM

    Hi Davis,

     

    Do we need to change these values in the MOM and Collector both?



  • 22.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 05, 2014 10:37 AM
    Both if they are 64-bit OS and JRE


  • 23.  Re: Introscope enterprise manager service stopped

    Posted Sep 05, 2014 11:48 AM

    Hi Davis,

     

    Do we need to change the numbering also if we are deleting any of the property. If you notice each property is appended with a number, So if number 8 is deleted, Then the properties have to be adjusted accordingly?



  • 24.  Re: Introscope enterprise manager service stopped

    Broadcom Employee
    Posted Sep 05, 2014 01:00 PM
    Yes change the numbering so it is in sequence just to make it easier to read.