SPS Crash RHEL

[Richard Leto]

Hi All,

 

   We are experiencing JAVA crash when we are trying to start the SPS services on RHEL server.

 

SPS version

[13/Apr/2015:11:55:23-990] [INFO] - CA Secure Proxy Server

[13/Apr/2015:11:55:23-993] [INFO] - Version 12.52 , Update 0101 , Label 640

[13/Apr/2015:11:55:23-993] [INFO] - File Version: 12.52 .0101 .640

 

 

RHEL release.

 

Red Hat Enterprise Linux Server release 6.4 (Santiago)

Linux hostname 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

 

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x09222441, pid=22232, tid=2687167344

#

# JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)

# Java VM: Java HotSpot(TM) Server VM (24.75-b04 mixed mode linux-x86 )

# Problematic frame:

# C  [libstdc++.so.6+0x8d441]  std::string::assign(std::string const&)+0x21

#

# Core dump written. Default location: /opt/CA/secure-proxy/proxy-engine/core or core.22232

#

# If you would like to submit a bug report, please visit:

#   http://bugreport.sun.com/bugreport/crash.jsp

# The crash happened outside the Java Virtual Machine in native code.

# See problematic frame for where to report the bug.

#

 

 

---------------  T H R E A D  ---------------

 

Any suggestions what we can do so that we will be able to start the SPS services?

Any one faced this problem? Let me know if needs more information.

 

Thank you

[Karmeng]

Hi,

 

It seems odd that SPS crash during startup. The SPS version R12.52SP1CR1 is the latest version so it should contains latest fix. I suggest to open CA support ticket to follow up as analysis need to be done to core dump file, hs_err_*** file and SPS logs. There is a script that CA Support can provide to collect all libraries that related to dump file and packaged in pkgapp. Best to engage CA Support for analysis

Thanks.

Kar Meng

[Legacy User]

additionally i would  collect core dumps and at least one strace output using follow child processes to provide to CA when i open the case.

[Richard Leto]

Thanks Karmeng/Josh, I was also thinking to open the case with Support, but I was in impression that this could be something I could be missing and someone might have seen this happening can suggest what could be happening wrong, if they have faced same kind of crash.

[Legacy User]

Richard Richard.Leto

 

I see that OS is a 64bit OS. However note SPS is a 32bit Application and hence would be capped by limits on 32bit Java. It is bit annoying that 32bits having further restrictive caps. Since Java has crashed, here my recommendation to have this information handy to avoid too many to and fro communications.

 

• Java Hot Spot file i.e. hs_err*
• Core Dump (make sure this is a full dump and not an application level dump)
  • Make sure ulimit -c is set to unlimited.
• Make sure you are running a trace on Java Memory Utilization using TOP command and output that to a file every 5mins.
  • Generate a Graph to see Java Memory Utilization.

 

Additional input

• Make sure you provide a copy of C:\CA\secure-proxy\proxy-engine\conf\SmSpsProxyEngine.properties
  • Making sure JVM settings is tuned appropriately for SPS.
  • There are documents / best practices suggested for configuring 32bit Java / JVM options after considering your RAM Capacity on Server.

 

Regards

 

Hubert

[Richard Leto]

Thanks Hubert, one thing I am noticing on my systems where I am trying to run SPS services (I got this issue on 3 Linux Servers I am guessing this something needs to be Infrastructure but not sure what needs to be done specifically) I upgraded the JDK from 1.7 to JDK 1.8 (I hope that's supported, but still problem persists) now the services are starting but when trying to browse the proxyui URL JAVA crashes. But with JDK1.7 services were failing to start and generating core.

 

# JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26)

# Java VM: Java HotSpot(TM) Server VM (25.40-b25 mixed mode linux-x86 )

# Problematic frame:

# V  [libjvm.so+0x4fc018]  java_lang_Throwable::fill_in_stack_trace(Handle, methodHandle, Thread*)+0x388

#

# Core dump written. Default location: /opt/CA/secure-proxy/proxy-engine/core or core.7793

 

I am generating system level core on the systems.

 

just noticed SmSpsproxyEngine.properties file shows:

 

// *****************************************************************************

//                      Section 1: Environment Variables (Windows platforms only)

// *****************************************************************************

NETE_SPS_ROOT=/opt/CA/secure-proxy

NETE_SPS_JAVA_HOME=%NETE_SPS_ROOT%\JDK\1.5.0_10

NETE_SPS_TOMCAT_HOME=%NETE_SPS_ROOT%\Tomcat

Path=%NETE_SPS_JAVA_HOME%\jre\bin;%SYSTEMROOT%\system32;%SYSTEMROOT%;%NETE_SPS_ROOT%\agentframework\bin

 

 

//set path for agent logging config file

//Uncomment this path when multiple instances of the STS are deployed

 

 

//STS_AGENT_LOG_CONFIG_FILE=%NETE_SPS_ROOT%\proxy-engine\conf\sts-config\globalconfig\agent-multiinstance-log4j.xml

STS_AGENT_LOG_CONFIG_FILE=

 

 

// *****************************************************************************

//                      Section 2: JVM Startup Options (Windows platforms only)

// *****************************************************************************

NETE_SPS_PROXYENGINE_CMD="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Xms512m -Xmx1024m -XX:MaxPermSize=256M -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -Dlogger.properties="%NETE_SPS_TOMCAT_HOME%/properties/logger.properties" -DSM_AGENT_LOG_CONFIG="%STS_AGENT_LOG_CONFIG_FILE%" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources;%NETE_SPS_ROOT%\agentframework\java\cryptoj.jar" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf"

NETE_SPS_PROXYENGINE_CMD_STOP="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf" -stop

 

**************************************************************************************************

 

Will it possible for you to provide documents/best practices for tuning JAVA for SPS if you have them handy I can take a reference.

 

Thank You

[Legacy User]

Richard Richard.Leto

 

I think you jumped too fast. JDK 1.8 aint supported yet. Always check the Support Matrix. I think JDK1.8 suppports is getting out in next release.

[Richard Leto]

I know I jumped on JDK1.8 and its not supported yet, but I was curious to find if that works some how, and i saw progress, anyways I will be downgrading the JDK and I would expect that SPS will not start at all.

[Anon Anon]

Hi Richard,

 

I just found your note on the the JVM core dump when starting SPS 12.52 SP1 CR1 on RHEL 6.4. I'm just facing the exact same issue and was wondering if you've already come to a conclusion about what the issue was. I'm preparing a support ticket, but thought that I's check with you in parallel too.

 

Best Regards

Jonas

[Karmeng]

Hi Jonas,

 

Not sure if Richard solved the issue but I think submit a ticket to report the SPS crash is right path to go as the product shouldn't crash.

One of the suggestion that I can make is try disable advance authentication (AA) and check if the issue still persist. This approach can help to isolate the problem (at least we isolate the AA). I remember some other customer has some issue when AA enable but cannot remember if it contribute to crash.

 

To disable AA:

/proxy-engine/server.conf

        <Context name="AALoginService">

    docBase="aaloginservice"

    path="aaloginservice"

    enable="no"

   </Context>

   <Context name="Advacned Auth Application">

    docBase="authapp"

    path="authapp"

    enable="no"

   </Context>

   <Context name="UI Application">

    docBase="uiapp"

    path="uiapp"

    enable="no"

   </Context>

    </Contexts>

 

If it does not help, open ticket with CA Support to review the hs_err log, core file.

 

Regards,

Kar Meng

[Julien_Nitot]

Thank you Kar Meng,

 

Actually Session Assurance and Advanced Authentication were not properly configured / not used and as such the context associated would need to be disabled in the SPS server.conf otherwise SPS will not start properly.

 

After disabling them, SPS started properly in Jonas's case.

 

Julien.

[Richard Leto]

Thanks Julian/KerMeng,

 

    After disabling the Advanced Authentication the SPS services started well, One thing I noted I hope it will help to others also, if Advanced Auth Services don't started properly in Policy server then SPS AA services will create problem, this was happening to me when I used PS on Linux(Service for AA was not started), when I pointed SPS to windows SPS started well.

 

Thanks for your suggestions.

[Satish Singh]

Hi Leto,

I am also facing similar issue , Can you please let me know what was the solution shared by Broadcom.
The issue is in production , will be great if you share soon.
Regards,
Satish

[Patrick Dussault]

Hi Satish,

As per the last message from Richard Leto, the issue was that the SPS
wasn't able to start.

And following the indications from Julien, disabling AA in server.conf
solved the issue, and the SPS was able to start.

What is the issue you face ? Are you able to start SPS or is it
refusing to start ?

I hope this helps,

Best Regards,
Patrick

[Satish Singh]

Hi Patrick,
I am not facing SPS restart issue infcat the proxy engine is getting stopped as JVM crashes and hence Tomcat is stopped.Post the crash we restart the SPS and it start working again.
We have already disabled the AA in server.conf .
Stop/Start/Restart of SPS is working fine. The issue Tomcat is getting stopped by itself . I have already raised a case. 
We are using java 1.6.45 with SPS 12.52, though its not compatible but it is working fine except the crash issue.

What is meant by 2nd point , 1st point already taken care.
To get around this issue you have two options:
1- Disable the app deployment for the Advanced Auth Application and the UI application (If you are not going to use Device DNA)
2- Enable the RiskMinder Service on the policy server

Regards,
Satish

[Patrick Dussault]

Hi Satish,

1- Disable the app deployment for the Advanced Auth Application and
the UI application (If you are not going to use Device DNA)

2- Enable the RiskMinder Service on the policy server

I guess Richard meant :

1- Remove the Session Assurance end point you configured in the AdminUI;

or if you need session assurance,

2- Make sure that Session Assurance is configured with the Policy
Server and it is up and running;

Reading the comments on this thread, the problem doesn't seem to be a
crash while running, but a problem that prevent the SPS to start, as
Richard mentioned opening the thread :

"We are experiencing JAVA crash when we are trying to start the
SPS services on RHEL server."

I hope this helps,

Best Regards,
Patrick