Layer 7 Access Management

Helping to debug SSO Policy Server java processes - redirecting stdout/stderr to timestamped file. 

06-05-2017 03:46 AM

Why do we want to see java stdout: 

Debugging java federation and custom components running in the JVM in the smpolicysrv policy server can be a painful business.    The policy server runs as a service, and redirects and writes to stdout/stderr to null device, effectively throwing them away.

    

The main use for stdout is to see the Stack trace written by the JVM when a process fails (for example OutOfMemoryException, or failed to find class file, or class file is not an acceptable compiled version).  Also a lot of java library components written by third parties and even in java itself, write trace information to stdout (for example the java –verbose, or –Djavax.net.debug=all). So easy access to stdout/stderr often assist with general debugging and monitoring custom components even when there is not a specific problem.

 

Now JVM stdout only gets written when the JVM is started, it is not enough the start the policy server, you also have to call some java component.   The below trace logs were triggered using the javaauthapi-delay.zip example which is also attached. 

 

The method here adds a  "premain" java class to setup java to redirect stdio to a timestamped file.  For completeness, and as an alternative method, we also show how to run the policy server from the cmd line. 

 

 

Solution 1 - Deploy open source RedirectStdio.jar library:  

Attached is a java utility that allows redirecting of the java stdout / stderror of the JVM - that may help capture the stacktrace of the java errors & Exceptions .   Redirection of java stdout/stderr is fairly simple it is done by calling the following java functions:  System.setOut(..); System.setErr(..);  There is also a  "-javaagent" directive that can direct a JVM to run a "premain" function prior to starting the routine.   That is useful as we do not have to change any of the deployed custom auth/az/assertiongenerator code to get access to the java stdout. 

   

Fortunately it is fairly easy to install :

 

a) Deploy the java-redirectstdio.zip

Unzip the attached java-redirectstdio.zip file.   The deployment includes stc code, you may need to rebuild from src, depending on the java version your using (I suspect the build one will work for you, since it is build with java .1.6). But compile is simple, edit build.bat file to point to JDK on policy server, and run the build.bat

 

 

b1-a) Change JVMOptions.txt to add :

There are two methods for setting the parameters to RedirectStdio.jar, the first uses a properties file, which is the more convenient, and allows dynamically enable/disable of the redirect via enable= setting in the .properties file.  The second puts all the parameters on the entry line in the JVMOptions.txt and requires a restart of the policy server for the new parameters to take effect. 

 

This section b1-a) and b1-b) describes the .properties method, section b2) describes the older cmd line option method. 

 

For the SSO Policy server install, we need to change the startup JVM runtime options. These are stored in the JVMOptions.txt file.   We need to add one line, with -javaagent , which contains both the .jar file containing the premain() class file, and then a parameters pointing to the default properties file: 

-javaagent:C:/work/java-redirectstdio/RedirectStdio.jar=C:/work/java-redirectstdio/dist/redirectstdio.properties

Note: If you have spaced in the distribution path, you will need to add %20 as the URL encoded space.   And you may want to change the parameters, for the location of the RedirectStdio.jar file and path to where it writes the properties files are : :  Here is screenshot of my JVMOptions.txt file: 

 

 

 

b1-b) Change redirectstdio.properties file:

The redirectstdio.properties is where parameters to the redirectstdio can be set, the most important ones are "enable=true" which enabled the redirect of java stdout/stderr and prefix=  which determines where the logs are written. 

 

The properties file is checked every 30sec, and if there are any changes it is re-read.  The main use for this is to dynamically be able to turn the logging on & off (via the enable= setting).

 

 

 

b2) Change JVMOptions.txt to add :

This section describes how to put the parameters on the JVMOption.txt cmd line, you probably want to use the previosu method, and skip this setup. 

 

For the SSO Policy server install, we need to change the startup JVM runtime options. These are stored in the JVMOptions.txt file.   We need to add one line, with -javaagent , which contains both the .jar file containing the premain() class file, and then also a list of parameters as per : 

-javaagent:C:/work/java-redirectstdio/RedirectStdio.jar=prefix=C:\\Program%20Files%20(x86)\\CA\\siteminder\\log\\redirect_,rolloverSize=10000,numLogsToRetain=20

Note the %2 0 is used where space is required in the file name.  And you may want to change the parameters, for the logation of the RedirectStdio.jar file and path to where it writes the log files are to be generated. :  Here is screenshot of my JVMOptions.txt file: 

 

c) Restart the smpolicysrv

As soon as some java activity is started log files should appear in the C:\\Program%20Files%20(x86)\\CA\\siteminder\\log\\ directory.

 

And the logs should contain all the stdout log data: 

 

 

 

Solution 2 - Run smpolicysrv from the cmd line: 

One workaround is to run the policy server from the command line.   You can stop the service on windows, or run the stop-all command on linux : 

Then run the smpolicysrv directly from the cmd line: as per: 

As you can see anything written to java stdout (in this case it is from our custom delay auth scheme) is printed to the DOS cmd: 

 

But often that is not convenient, since it: will interrupt current users; can have additional complications when running a service from the command line (such as win2008); often for managed environments a change requests or special approval process is needed, before this can be done, which further delays resolving the problem; and finally a restart may actually clear the exception condition that you are trying to isolate.

 

Clearly it is best if you to run it in your QA environment on one machine before running it on production.  And often for production deployment, if all the PS machines are acting the same, then installing on only one should be sufficient to give some idea of what is being written to the java stdout/stderror log. 

 

(Added -2.zip version, which has a bit more debugging and catch Exception for the bg thread monitoring for changes in the .properties file)

Statistics
0 Favorited
0 Views
2 Files
0 Shares
0 Downloads
Attachment(s)
zip file
javaauthapi-delay.zip   81K   1 version
Uploaded - 05-29-2019
zip file
java-redirectstdio-2.zip   49K   1 version
Uploaded - 05-29-2019

Tags and Keywords

Comments

10-12-2017 08:24 PM

Hi Vaibhav, I copied this reply from the internal support case you raised, since it would be relevant here too.

 

 

Hi Vaibhav, 

I've updated the attached zip file, it has a bit more debug logging around the background thread that checks for changes in the  .properties file, and also now has a catch for (Throwable) in that background thread - maybe an exception was occurring and stopping the bg thread. 

If you run the smpolicysrv from the command line (unix or windows) like so: 
 


Then you should be able to see both the non-redirected output as well as that redirected to the file:
 


And then toggle the enable=*** value in the properties file and see what happens - hopefully it does toggle and hopefully you can see the output being sent to the console or to the log file.  And if not - then hopefully it shows us why it is failing to detect the new enable value. 

Best of luck !

Cheers - Mark

----
Mark O'Donohue
Snr Principal Support Engineer

10-11-2017 07:19 AM

Thank you for sharing this tip ! 

 

enable=false doesn't seems to work i.e. dynamic turning on/off logs with this flags doesn't seems to work. Please suggest if any other alternative.

06-06-2017 11:19 AM

Related Entries and Links

No Related Resource entered.