Layer7 API Management

Enabling CA Single Sign On (CA SSO) tracing in API Gateway 

05-28-2018 12:27 AM

1. Background: 

If you are familiar with CA SSO gateway java agent, then you probably know it has a fairly detailed trace logging that logs to STDOUT when the java runtime setting :" is added. 


But when you try this on API Gateway, by setting that value in the  /opt/SecureSpan/Gateway/node/default/etc/conf/

This does not generate any output. 


2. Note: 

For the following, I have used the instructions below to setup the stdout/stderr logging in API Gateway. 

Enable Logging of STDOUT and STDERR streams. 

And I have used the CA SSO policy setup as described (and attached) here: 

Integrating APIM Gateway with CA Single Sign-On - adding a grace time for updating SMSESSION cookie. 



3. CA SSO Usage in API Gateway:

The java SSO agent comes in two forms: a JNI version form where the java is a thin layer over the C++ agent; and the pure java form where the whole agent is written in java.  From a CA SSO perspective, both the C++ agent and the pure-java agent are fairly regularly used, but for most java applications the pure-java one is usually the most convenient to deploy.   


But the API Gateway uses the JNI version of the CA SSO agent.


4. Debugging the CA SSO Agent

One other difference between the two CA SSO agent forms is that the pure java CA SSO agent has a very detailed logging capability, whereas the JNI CA SSO agent does not (unless you hack into making some C++ native calls).


So from our perspective, where we are on the API Gateway trying to debug what the CA SSO agent is doing - it would be useful if we can enable the pure-java agent so we can determine what is happening - even if after that we return to using the JNI form of the agent.  


Fortunately the pure-java and JNI versions of the CA SSO agent can be switched by using different .jar file - plus a few settings.  


5. API Gateway use of CA SSO Agent 


The CA SSO (formally siteminder) assertions are in this module in : /opt/SecureSpan/Gateway/runtime/lib directory


It calls the CA SSO java sdk components in the same directory: 

note, one of the .jar files starts with upper case Sm and the other two start with lower case sm.


These should match the CA SSO sdk components in the /opt/CA/sdk/java64 directory: 

There may be slight differences, between the file sizes, two directory files due CR deployment.  


Note1: The smjavaagentapi.jar is the JNI form of the CA SSO Agent.  The smagentapi.jar is the pure-java form of the CA SSO Agent.


Note2: The xxxx-12.5.jar are from Gateway 9.2, but Gateway 9.3 uses CA SSO R12.52 so the version extension for gw 9.3 are ***-12.52.jar.  You'll need to keep that in mind when doing the following steps.


6. Updating API Gateway to use the pure java CA SSO Agent 


Basically we are going to swap out usage of the smjavaagentapi.jar and use the smagentapi.jar.  


Normally we would move the smjavaagentapi.jar from the directory and drop in the smagentapi.jar.   However there is a dependency from  layer7-siteminder-9.2.00.jar on smjavaagentapi-12.5.jar, so in this case when we put in smagentapi.jar we will need to give it the "smjavaagentapi-12.5.jar" name. 


Also, if you are lucky and the file sizes for the three SSO .jar files in /opt/SecureSpan/Gateway/runtime/lib match those in /opt/CA/sdk/java64  then you probably only need to update the smjavaagentapi-12.5.jar file.  Here I am updating usage of all three SSO .jar files to those in /opt/CA/sdk/java64 just to be sure. 


Here is what we do - we rename the three .jar files so they won't be picked up by the classloader : 


mv smjavaagentapi-12.5.jar smjavaagentapi-12.5-jar-orig
mv SmJavaApi-12.5.jar SmJavaApi-12.5-jar-orig
mv smjavasdk2-12.5.jar smjavasdk2-12.5-jar-orig



Then link them to the current versions in /opt/CA/sdk/java64 - with special care for the smjavaagentapi-12.jar so that we link that to the pure-java implementation. eg: 

ln -s /opt/CA/sdk/java64/smagentapi.jar smjavaagentapi-12.5.jar
ln -s /opt/CA/sdk/java64/SmJavaApi.jar SmJavaApi-12.5.jar
ln -s /opt/CA/sdk/java64/smjavasdk2.jar smjavasdk2-12.5.jar

(note the link of smagentapi.jar to smjavaagentapi...jar)




7.  Check to see if API Gateway is logging CA SSO trace to STDIO log file - first attempt.


As mentioned previously we've used the CA SSO policy  setup as described here :  

Integrating APIM Gateway with CA Single Sign-On - adding a grace time for updating SMSESSION cookie. 


Using our CA SSO setup we can click the "Test" button at the end. 


It comes back saying that the validation passed : 


The using the stdout/stderr logging as dewscribed here :Enable Logging of STDOUT and STDERR streams. 


We check the stdout_0_0.log that we have setup then we (should) be able to see the normal CA SSO pure-java logging : 

So we have put in the pure-java module, and enabled the logging 


But we are not quite out of the woodwork yet, even though it reported a validation "success" we can also see in the logs that we have some error occurring (and also if we go to actually using a CA SSO assertion it will also fail). 


We can see in the stdout.log the following SecurityException is raised : 

Fortunately we know the cause of this one - and how to fix it.   


CA SSO (at least prior to R12.8) use RSA BSAFE cryptoj.jar as the security provider.  RSA provide a few different jar files, previously for example, cryptoj.jar, would, by default, allow all crypto methods and cryptojFIPS.jar would, again by default, only allow FIPS compliant algorithms.  Clearly the RSA cryptoj provided in API Gateway is the one that defaults to FIPS only mode. It is in the same /opt/SecureSpan/Gateway/runtime/lib directory as the other jar files we have been looking at.   


The error  java.lang.SecurityException: "Algorithm not allowable in FIPS140 mode: RC2/CBC/PKCS5Pad", tells us that RSA cryptoj is only allowing FIPS ciphers. But in the CA SSO setup we picked (in the screenshot earlier) we chose COMPAT mode, so it is using some older ciphers.   


So we need to allow RSA cryptoj to use non-FIPS ciphers. 



8.  Configure RSA cryptoj to allow use non FIPS140 algorithms


To allow NON FIPS methods in RSA cryptoj you edit the /opt/SecureSpan/JDK/jre/lib/security/ file : 


And append the following line : 


to the file : 

Then on startup, allows the provider jar file to provide NON_FIPS140 algorithm.  Other startup modes are documented and explained here:… 


And then we need to restart the gateway service.

service ssg restart


9.  Check (again) to see if API Gateway is logging CA SSO trace to STDIO log file - success. 


If we go back again to our CA SSO setup, and click the "Test" button : 


We still get the "Validation passed" result, but this time when we check the stdout_0_0.log in 

 /opt/SecureSpan/Gateway/default/var/logs then we don't see an exception trace, and we see successful conneciton.




10.  Checking the whole  (again) to see if API Gateway is logging CA SSO trace to STDIO log file - success. 


Also we can check an actual CA SSO operation, via API Policy, (this step would have failed without the prior step NON_FIPS mode being set, although I did not show that earlier).


My case I have setup of /mainREST policy 


Integrating APIM Gateway with CA Single Sign-On - adding a grace time for updating SMSESSION cookie. 






11. Reverting back after debugging

To revert back to the JNI CA Single Sign On agent


The main step is to remove the links created in step (here I put then in a directory sso-debug, in case I need them latter on).


Then restore the original .jar files : 


You can also comment out the RSA 

vi /opt/SecureSpan/JDK/jre/lib/security/



And comment out the siteminder debug setting in the file : 

vi /opt/SecureSpan/Gateway/node/default/etc/conf/



Hope hat helps you find your integration problems with API Gateway and CA Single Sign On.


Cheers - Mark

0 Favorited
0 Files

Tags and Keywords

Related Entries and Links

No Related Resource entered.