Layer7 Access Management

Expand all | Collapse all

SPS Crash RHEL

Jump to Best Answer
  • 1.  SPS Crash RHEL

    Posted 04-13-2015 04:19 PM

    Hi All,

     

       We are experiencing JAVA crash when we are trying to start the SPS services on RHEL server.

     

    SPS version

    [13/Apr/2015:11:55:23-990] [INFO] - CA Secure Proxy Server

    [13/Apr/2015:11:55:23-993] [INFO] - Version 12.52 , Update 0101 , Label 640

    [13/Apr/2015:11:55:23-993] [INFO] - File Version: 12.52 .0101 .640

     

     

    RHEL release.

     

    Red Hat Enterprise Linux Server release 6.4 (Santiago)

    Linux hostname 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

     

    # A fatal error has been detected by the Java Runtime Environment:

    #

    #  SIGSEGV (0xb) at pc=0x09222441, pid=22232, tid=2687167344

    #

    # JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)

    # Java VM: Java HotSpot(TM) Server VM (24.75-b04 mixed mode linux-x86 )

    # Problematic frame:

    # C  [libstdc++.so.6+0x8d441]  std::string::assign(std::string const&)+0x21

    #

    # Core dump written. Default location: /opt/CA/secure-proxy/proxy-engine/core or core.22232

    #

    # If you would like to submit a bug report, please visit:

    #   http://bugreport.sun.com/bugreport/crash.jsp

    # The crash happened outside the Java Virtual Machine in native code.

    # See problematic frame for where to report the bug.

    #

     

     

    ---------------  T H R E A D  ---------------

     

    Any suggestions what we can do so that we will be able to start the SPS services?

    Any one faced this problem? Let me know if needs more information.

     

    Thank you



  • 2.  Re: SPS Crash RHEL

    Posted 04-14-2015 12:32 AM

    Hi,

     

    It seems odd that SPS crash during startup. The SPS version R12.52SP1CR1 is the latest version so it should contains latest fix. I suggest to open CA support ticket to follow up as analysis need to be done to core dump file, hs_err_*** file and SPS logs. There is a script that CA Support can provide to collect all libraries that related to dump file and packaged in pkgapp. Best to engage CA Support for analysis

    Thanks.

    Kar Meng



  • 3.  Re: SPS Crash RHEL

    Posted 04-14-2015 08:01 AM

    additionally i would  collect core dumps and at least one strace output using follow child processes to provide to CA when i open the case.



  • 4.  Re: SPS Crash RHEL

    Posted 04-14-2015 10:10 AM

    Thanks Karmeng/Josh, I was also thinking to open the case with Support, but I was in impression that this could be something I could be missing and someone might have seen this happening can suggest what could be happening wrong, if they have faced same kind of crash.



  • 5.  Re: SPS Crash RHEL

    Posted 04-14-2015 10:32 AM

    Richard Richard.Leto

     

    I see that OS is a 64bit OS. However note SPS is a 32bit Application and hence would be capped by limits on 32bit Java. It is bit annoying that 32bits having further restrictive caps. Since Java has crashed, here my recommendation to have this information handy to avoid too many to and fro communications.

     

    • Java Hot Spot file i.e. hs_err*
    • Core Dump (make sure this is a full dump and not an application level dump)
      • Make sure ulimit -c is set to unlimited.
    • Make sure you are running a trace on Java Memory Utilization using TOP command and output that to a file every 5mins.
      • Generate a Graph to see Java Memory Utilization.

     

    Additional input

    • Make sure you provide a copy of C:\CA\secure-proxy\proxy-engine\conf\SmSpsProxyEngine.properties
      • Making sure JVM settings is tuned appropriately for SPS.
      • There are documents / best practices suggested for configuring 32bit Java / JVM options after considering your RAM Capacity on Server.

     

    Regards

     

    Hubert



  • 6.  Re: SPS Crash RHEL

    Posted 04-14-2015 12:49 PM

    Thanks Hubert, one thing I am noticing on my systems where I am trying to run SPS services (I got this issue on 3 Linux Servers I am guessing this something needs to be Infrastructure but not sure what needs to be done specifically) I upgraded the JDK from 1.7 to JDK 1.8 (I hope that's supported, but still problem persists) now the services are starting but when trying to browse the proxyui URL JAVA crashes. But with JDK1.7 services were failing to start and generating core.

     

    # JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26)

    # Java VM: Java HotSpot(TM) Server VM (25.40-b25 mixed mode linux-x86 )

    # Problematic frame:

    # V  [libjvm.so+0x4fc018]  java_lang_Throwable::fill_in_stack_trace(Handle, methodHandle, Thread*)+0x388

    #

    # Core dump written. Default location: /opt/CA/secure-proxy/proxy-engine/core or core.7793

     

    I am generating system level core on the systems.

     

    just noticed SmSpsproxyEngine.properties file shows:

     

    // *****************************************************************************

    //                      Section 1: Environment Variables (Windows platforms only)

    // *****************************************************************************

    NETE_SPS_ROOT=/opt/CA/secure-proxy

    NETE_SPS_JAVA_HOME=%NETE_SPS_ROOT%\JDK\1.5.0_10

    NETE_SPS_TOMCAT_HOME=%NETE_SPS_ROOT%\Tomcat

    Path=%NETE_SPS_JAVA_HOME%\jre\bin;%SYSTEMROOT%\system32;%SYSTEMROOT%;%NETE_SPS_ROOT%\agentframework\bin

     

     

    //set path for agent logging config file

    //Uncomment this path when multiple instances of the STS are deployed

     

     

    //STS_AGENT_LOG_CONFIG_FILE=%NETE_SPS_ROOT%\proxy-engine\conf\sts-config\globalconfig\agent-multiinstance-log4j.xml

    STS_AGENT_LOG_CONFIG_FILE=

     

     

    // *****************************************************************************

    //                      Section 2: JVM Startup Options (Windows platforms only)

    // *****************************************************************************

    NETE_SPS_PROXYENGINE_CMD="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Xms512m -Xmx1024m -XX:MaxPermSize=256M -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -Dlogger.properties="%NETE_SPS_TOMCAT_HOME%/properties/logger.properties" -DSM_AGENT_LOG_CONFIG="%STS_AGENT_LOG_CONFIG_FILE%" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources;%NETE_SPS_ROOT%\agentframework\java\cryptoj.jar" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf"

    NETE_SPS_PROXYENGINE_CMD_STOP="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf" -stop

     

    **************************************************************************************************

     

    Will it possible for you to provide documents/best practices for tuning JAVA for SPS if you have them handy I can take a reference.

     

    Thank You



  • 7.  Re: SPS Crash RHEL
    Best Answer

    Posted 04-14-2015 12:53 PM

    Richard Richard.Leto

     

    I think you jumped too fast. JDK 1.8 aint supported yet. Always check the Support Matrix. I think JDK1.8 suppports is getting out in next release.



  • 8.  Re: SPS Crash RHEL

    Posted 04-14-2015 01:54 PM

    I know I jumped on JDK1.8 and its not supported yet, but I was curious to find if that works some how, and i saw progress, anyways I will be downgrading the JDK and I would expect that SPS will not start at all.



  • 9.  Re: SPS Crash RHEL

    Posted 05-19-2015 07:03 PM

    Hi Richard,

     

    I just found your note on the the JVM core dump when starting SPS 12.52 SP1 CR1 on RHEL 6.4. I'm just facing the exact same issue and was wondering if you've already come to a conclusion about what the issue was. I'm preparing a support ticket, but thought that I's check with you in parallel too.

     

    Best Regards

    Jonas



  • 10.  Re: SPS Crash RHEL

    Posted 05-20-2015 12:08 AM

    Hi Jonas,

     

    Not sure if Richard solved the issue but I think submit a ticket to report the SPS crash is right path to go as the product shouldn't crash.

    One of the suggestion that I can make is try disable advance authentication (AA) and check if the issue still persist. This approach can help to isolate the problem (at least we isolate the AA). I remember some other customer has some issue when AA enable but cannot remember if it contribute to crash.

     

    To disable AA:

    /proxy-engine/server.conf

            <Context name="AALoginService">

        docBase="aaloginservice"

        path="aaloginservice"

        enable="no"

       </Context>

       <Context name="Advacned Auth Application">

        docBase="authapp"

        path="authapp"

        enable="no"

       </Context>

       <Context name="UI Application">

        docBase="uiapp"

        path="uiapp"

        enable="no"

       </Context>

        </Contexts>

     

    If it does not help, open ticket with CA Support to review the hs_err log, core file.

     

    Regards,

    Kar Meng



  • 11.  Re: SPS Crash RHEL

    Posted 05-21-2015 07:14 AM

    Thank you Kar Meng,

     

    Actually Session Assurance and Advanced Authentication were not properly configured / not used and as such the context associated would need to be disabled in the SPS server.conf otherwise SPS will not start properly.

     

    After disabling them, SPS started properly in Jonas's case.

     

    Julien.



  • 12.  Re: SPS Crash RHEL

    Posted 05-21-2015 11:42 AM

    Thanks Julian/KerMeng,

     

        After disabling the Advanced Authentication the SPS services started well, One thing I noted I hope it will help to others also, if Advanced Auth Services don't started properly in Policy server then SPS AA services will create problem, this was happening to me when I used PS on Linux(Service for AA was not started), when I pointed SPS to windows SPS started well.

     

    Thanks for your suggestions.