Layer7 Access Management

Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server)

By Mark ODonohue posted 03-06-2017 09:38 PM

  

Introduction 

The purpose of this blog entry is to show how to enable all the different types of trace logs that are available in CA SSO Access Gateway (formerly known as Secure Proxy Server).   I will also be referring to the Access Gateway product as "Ag" in the article -however some of the slides predate the name change so will show up as SPS.  

 

Ag can be used a few different ways, and depending upon what you are using it for will determine what logs you want to enable.   I've split this up into different themes:

  • Ag Logging when used as Reverse Proxy Server
  • Ag Logging when used as Federation Gateway
  • Ag Logging for ProxyUI 
  • Ag Logging for WebServices

 

This log covers the Logginfg when used Ag is used as a Reverse Proxy Server the other logging profiles will be added as a separate documents at a later date. 

 

 

 

Ag Logging when used as a Reverse Proxy Server: 

 

The following gives an overview of the major components of Ag and also shows the name of (all) the logs that can be enabled and where they get their data from:

 

 

 

When used as a reverse proxy server, requests come in from the client, to Apache httpd, get passed to Apache/tomcat and then get forwarded to a backend server for processing.  The backend then completes the request and the data is then passed back to tomcat, to httpd and back to the client.    Note: I wont be discussing the use of fiddler and wireshark, but the diagram indicates where they would be used.

 

In summary we have: 

  • Apache Logs
  • Mod_Jk logs
  • Proxy Engine Logs
  • Web Agent Logs
  • Httpclient Logs

Each of which is covered in the sections below. 

 

 

Apache Logs 

The two major logs for apache httpd are access_log and error_log, these log the interaction with the user <-> httpd process. The httpd.conf entries are :

 

- Access_log  - settings in httpd.conf :  

The formats are defined here : 

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

 

And the rotating logs are set here : 

CustomLog "|'C:/Program Files/CA/secure-proxy/httpd/bin/rotatelogs.exe' 'C:/Program Files/CA/secure-proxy/httpd/logs/access_log' 10M" common

 

Occasionally it is good to supplement what is in the access_log and to get some insight to a problem.  The example above it shows the %{User-Agent}i header, but you can also use that to capture cookies for example:  \"%{SMSESSION}C\" .   The option %T is also useful, since it logs the total time it took Ag to process the request and return the response to the user. 

 

A complete list of LogFormat parameters is available here: 

Tech Note : Enable httpclient logging in Access Gateway 12.7 

 

Note: And one final point to remember, the the access_log entry is written at the END of processing the request.  So if apache httpd crashed then the requests that are currently in flight when the crash happened are NOT logged. 

 

- Error_log  - settings in httpd.conf :  

The formats are defined here : 

# LogLevel: values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn

 

And the rotating logs are set here : 

ErrorLog "|'C:/Program Files/CA/secure-proxy/httpd/bin/rotatelogs.exe' 'C:/Program Files/CA/secure-proxy/httpd/logs/error_log' 10M"


For debugging, you can raise the LogLevel to debug.   Apache 2.4 also has extra levels trace1 ... trace8, there are needed when you want to trace the raw data packets and SSL handshaking problems between the front end client and the httpd process. So for debugging often we can recommend : 

LogLevel trace8

 

The apache error_log is also good place to find the exact httpd and mod_jk version numbers: 

 

 

Mod_Jk Logs

Mod_jk is the Apache httpd module that forwards requests onto tomcat.  The log settings for it are in httpd.conf : 

 

JkLogFile "|'C:/Program Files/CA/secure-proxy/httpd/bin/rotatelogs.exe' 'C:/Program Files/CA/secure-proxy/httpd/logs/mod_jk.log' 10M"

JkLogLevel error

The log-level parameter describes what detail of logging should occur.   Possible values are : debug  info  error

 

For debug level logging, it is best to also set the JkRequestLogFormat, to display more detail of the transaction:

JkLogLevel debug

JkRequestLogFormat "%w %V %T %m %H %p %U %s"

That will show most of the raw byte data of what is send from httpd -> tomcat and what is returned.  The settings are explained here :  https://tomcat.apache.org/connectors-doc/reference/apache.html 

 

Sample mod_jk.log : 

 

 

Proxy Engine Logs

The proxy engine has two main logs :  

    server.log

    nohup*.out  

These are in secure-proxy/proxy-engine/logs directory by default.   The server.log is the log4j logging for the proxy-engine, and the nohup_<pid>.out log is the redirect of stdout and stderror logs 

 

server.log

Logging level for server.log is set in Tomcat/properties/logger.properties 

log4j.rootCategory=INFO,SvrFileAppender

log4j.rootCategory.ResourceBundle=root

The log level can bec changed to OFF, FATAL, ERROR, WARN, INFO, DEBUG, ALL

 

nohup_<date>_<time>.out log : 

We generally don't change the logging in this one as it logs the stdout/stderr logs from the proxy-engine.  Although one useful tip is adding "-verbose" to the java startup, and then you get the exact .jar file that each class is loaded from in this log.   A new timestamped log is started each time the proxy-engine is started.  The nohup log is good at capturing the stacktrace when Exceptions are thrown in proxy-engine eg: 

 

 

 

Web Agent Logs

 

Ag comes witn the standard WebAgent logs.  These are enabled via the ACO settings as per the normal agent eg: 

Must be enabled, and setup as normal agent ACO parameter: 

 

WebAgent.log 

LogAppend="NO"
LogFile=“YES"
LogFileName=“c:\ca\proxy-engine\logs\WebAgent.log"
LogFileSize="100"

 

 

WebAgentTrace.log 

TraceAppend="NO"
TraceConfigFile=“c:\ca\proxy-engine\conf\defaultagent\SecureProxyTrace.conf"
TraceFile=“YES"
TraceFileName=“c:\ca\proxy-engine\logs\WebAgentTrace.log"
TraceFileSize="100"

 

 

WebAgentTrace.log SecureProxyTrace.conf settings :

The SecureProxyTrace.conf is slightly different to the WebAgentTrace.conf.  It has ProxyAgent as default. 

I also tend to add Agent_Con_Manager, and AgentFunc as components. 

And add data items :  PreciseTime, Function,  and SrcFile as shown below: 

 

WebAgentTrace.log with proxy-rule messages : 

Additionally for the webagenttrace to log the proxy rule evaluation you need to add debug=“yes” to proxy-rules.xml to get additional error messages specific to SPS :

 

 

WebAgentTrace.log examples: 

After setting the above then we endup with normal trace log like: 

 

 

And with Ag specific messages for proxy-rules such as:

 

 

HttpClient Logs

Http client  logs the raw data GET/POST that is sent to the backend and and reply that is received.   So it is good for debugging the interaction with the backend server. 

 

To enable httpclient logging in server.conf set :
           httpclientlog = “yes”

and restart the proxy-engine service. 

 

Note: For Ag R12.7 there is extra setting needed to enable httpclient logging: 

https://communities.ca.com/community/ca-security/ca-single-sign-on/blog/2017/09/01/tech-note-enable-httpclient-logging-in-agent-gateway-127

 

HttpClient / Java SSL Logging

Java has the ability to log the SSL handshake and transfer of data.  This is done by adding  -Djavax.net.debug=all 
to the java runtime startup.  The file this needs to be applied to differs per platform :

For Windows - proxy-engine/conf/SpsProxyEngine.properties
For Unix - proxy-engine/proxyserver.sh

As show below: 

Enable SSL tracing for java:

 

SSL Tracing in the nohup and server.log files : 

These logs then show the SSL handshake, and decrypt/hash of each packet send and received when proxy-engine communicats to the SSL backend: 

 

Have a nice time enjoying your logging. 

 

Cheers - Mark

----
Mark O'Donohue
Snr Principal Support Engineer - Global Customer Success

 

This document is part of a series on Logging in SSO components: 

Tech Tip:How to enable trace logging in SSO (aka Siteminder) Webagent 

Tech Tip : Policy Server Loggings 

Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server) 

5 comments
14 views

Permalink

Comments

07-03-2018 05:54 AM

Just note if you do get error in SSL layer - then that can be :

 

a) JCE patch not installed - that one is covered a fair bit

b) Attempt to use a disabled (usually non FIPS) cipher or hash

 

Problem a) used to be the main issue, but recently b) is becoming more of an issue, since by default cryptojFIPS.jar only allows FIPS methods, and occasionally a backend or client wants to use them.   RSA BASFE (cryptoj.jar) is the crypto provider for SPS/Ag up till version 12.8 -

 

Below is how to allow usage of less secure HASH and CIPHER algorithms in SPS/Ag if you encounter this issue. 

 

 

To allow NON FIPS methods in RSA cryptoj you edit the /opt/SecureSpan/JDK/jre/lib/security/java.security file : 

 

And append the following line : 

com.rsa.cryptoj.jce.fips140initialmode=NON_FIPS140_MODE

to the file : 

Then on startup, allows the provider jar file to provide NON_FIPS140 algorithm.  Other startup modes are documented and explained here: 

https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp1502.… 

 

 

 

 

This is copied from point 8 on :

https://communities.ca.com/docs/DOC-231181940-enabling-ca-sso-tracing-in-api-gateway 

04-05-2018 08:13 PM

Just to cover the -verbose in a bit more detail. 

 

Add -verbose option to SPS can be useful since it shows what .jar file it loads each class from, and also when it loaded the class.  So you can determine (sometimes) :

  • If various crypto classes are not from the provider you expect, or
  • If a class is being loaded from an unexpected .jar file
    (maybe a .jar file installed in JDK ext directory is being used, rather than one loaded wihtin sps for example) 

 

To add -verbose to the java runtime startup.  The file this needs to be applied to differs per platform :

For Windows - proxy-engine/conf/SpsProxyEngine.properties
For Unix - proxy-engine/proxyserver.sh

 

Here is screenshot of the Windows SpsProxyEngine.properties file : 

 


 

The unix one is similar. 

 

Cheers - Mark

12-10-2017 05:13 PM

Reminder :

Module : mod_log_forensic is also available in SPS, and can be useful, should add update into doco for that : 

mod_log_forensic - Apache HTTP Server Version 2.4 

 

Cheers - Mark

 

10-12-2017 11:15 PM

For httpclient.log logging : 

 

This link is also applicable - when enabling httpclient logging, the following log4j settings allows you to separate the requests per thread, so you can determine which backend response belongs to which sent query. 

 

Tech Tip: How to change log4j format to show threadid (for Ag/SPS httpclient.log) 

 

Cheers - Mark

----
Mark O'Donohue
Snr Principal Support Engineer - Global Customer Success

03-10-2017 03:53 PM

Thanks for sharing this great info with the community Mark!

Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server)