We are experiencing JAVA crash when we are trying to start the SPS services on RHEL server.
[13/Apr/2015:11:55:23-990] [INFO] - CA Secure Proxy Server
[13/Apr/2015:11:55:23-993] [INFO] - Version 12.52 , Update 0101 , Label 640
[13/Apr/2015:11:55:23-993] [INFO] - File Version: 12.52 .0101 .640
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Linux hostname 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb) at pc=0x09222441, pid=22232, tid=2687167344
# JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)
# Java VM: Java HotSpot(TM) Server VM (24.75-b04 mixed mode linux-x86 )
# Problematic frame:
# C [libstdc++.so.6+0x8d441] std::string::assign(std::string const&)+0x21
# Core dump written. Default location: /opt/CA/secure-proxy/proxy-engine/core or core.22232
# If you would like to submit a bug report, please visit:
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
--------------- T H R E A D ---------------
Any suggestions what we can do so that we will be able to start the SPS services?
Any one faced this problem? Let me know if needs more information.
It seems odd that SPS crash during startup. The SPS version R12.52SP1CR1 is the latest version so it should contains latest fix. I suggest to open CA support ticket to follow up as analysis need to be done to core dump file, hs_err_*** file and SPS logs. There is a script that CA Support can provide to collect all libraries that related to dump file and packaged in pkgapp. Best to engage CA Support for analysis
additionally i would collect core dumps and at least one strace output using follow child processes to provide to CA when i open the case.
Thanks Karmeng/Josh, I was also thinking to open the case with Support, but I was in impression that this could be something I could be missing and someone might have seen this happening can suggest what could be happening wrong, if they have faced same kind of crash.
I see that OS is a 64bit OS. However note SPS is a 32bit Application and hence would be capped by limits on 32bit Java. It is bit annoying that 32bits having further restrictive caps. Since Java has crashed, here my recommendation to have this information handy to avoid too many to and fro communications.
Thanks Hubert, one thing I am noticing on my systems where I am trying to run SPS services (I got this issue on 3 Linux Servers I am guessing this something needs to be Infrastructure but not sure what needs to be done specifically) I upgraded the JDK from 1.7 to JDK 1.8 (I hope that's supported, but still problem persists) now the services are starting but when trying to browse the proxyui URL JAVA crashes. But with JDK1.7 services were failing to start and generating core.
# JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26)
# Java VM: Java HotSpot(TM) Server VM (25.40-b25 mixed mode linux-x86 )
# V [libjvm.so+0x4fc018] java_lang_Throwable::fill_in_stack_trace(Handle, methodHandle, Thread*)+0x388
# Core dump written. Default location: /opt/CA/secure-proxy/proxy-engine/core or core.7793
I am generating system level core on the systems.
just noticed SmSpsproxyEngine.properties file shows:
// Section 1: Environment Variables (Windows platforms only)
//set path for agent logging config file
//Uncomment this path when multiple instances of the STS are deployed
// Section 2: JVM Startup Options (Windows platforms only)
NETE_SPS_PROXYENGINE_CMD="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Xms512m -Xmx1024m -XX:MaxPermSize=256M -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -Dlogger.properties="%NETE_SPS_TOMCAT_HOME%/properties/logger.properties" -DSM_AGENT_LOG_CONFIG="%STS_AGENT_LOG_CONFIG_FILE%" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources;%NETE_SPS_ROOT%\agentframework\java\cryptoj.jar" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf"
NETE_SPS_PROXYENGINE_CMD_STOP="%NETE_SPS_JAVA_HOME%\bin\java.exe" -Dcatalina.base="%NETE_SPS_TOMCAT_HOME%" -Dcatalina.home="%NETE_SPS_TOMCAT_HOME%" -Djava.endorsed.dirs="%NETE_SPS_TOMCAT_HOME%\endorsed" -Djava.io.tmpdir="%NETE_SPS_TOMCAT_HOME%\temp" -DHTTPClient.log.mask=0 -DHTTPClient.Modules="HTTPClient.RetryModule|org.tigris.noodle.NoodleCookieModule|HTTPClient.DefaultModule" -classpath "%NETE_SPS_TOMCAT_HOME%\bin\proxybootstrap.jar;%NETE_SPS_TOMCAT_HOME%\properties;%NETE_SPS_JAVA_HOME%\lib\tools.jar;%NETE_SPS_TOMCAT_HOME%\bin\bootstrap.jar;%NETE_SPS_ROOT%\resources" com.netegrity.proxy.ProxyBootstrap -config "%NETE_SPS_ROOT%/proxy-engine/conf/server.conf" -stop
Will it possible for you to provide documents/best practices for tuning JAVA for SPS if you have them handy I can take a reference.
I think you jumped too fast. JDK 1.8 aint supported yet. Always check the Support Matrix. I think JDK1.8 suppports is getting out in next release.
I know I jumped on JDK1.8 and its not supported yet, but I was curious to find if that works some how, and i saw progress, anyways I will be downgrading the JDK and I would expect that SPS will not start at all.
I just found your note on the the JVM core dump when starting SPS 12.52 SP1 CR1 on RHEL 6.4. I'm just facing the exact same issue and was wondering if you've already come to a conclusion about what the issue was. I'm preparing a support ticket, but thought that I's check with you in parallel too.
Not sure if Richard solved the issue but I think submit a ticket to report the SPS crash is right path to go as the product shouldn't crash.
One of the suggestion that I can make is try disable advance authentication (AA) and check if the issue still persist. This approach can help to isolate the problem (at least we isolate the AA). I remember some other customer has some issue when AA enable but cannot remember if it contribute to crash.
To disable AA:
<Context name="Advacned Auth Application">
<Context name="UI Application">
If it does not help, open ticket with CA Support to review the hs_err log, core file.
Thank you Kar Meng,
Actually Session Assurance and Advanced Authentication were not properly configured / not used and as such the context associated would need to be disabled in the SPS server.conf otherwise SPS will not start properly.
After disabling them, SPS started properly in Jonas's case.
After disabling the Advanced Authentication the SPS services started well, One thing I noted I hope it will help to others also, if Advanced Auth Services don't started properly in Policy server then SPS AA services will create problem, this was happening to me when I used PS on Linux(Service for AA was not started), when I pointed SPS to windows SPS started well.
Thanks for your suggestions.