Introduction:
Web Agent logging function to monitor the performance of the Web Agent and its communication with the Policy Server. The following gives an overview of the logging that can be performed on the websever/webagent and where they get their data from:
In Summary the Log file groups we have are :
- Web Server Logs (IIS and Apache httpd)
- Web Agent Logs (WebAgent.log, WebAgentTrace.log)
- SM_TLI_LOG_FILE
And a number of trace tools that are useful.
- fiddler
- wireshark
- netstat
- strace (Unix)
- Sysinternals Process Monitor (Windows)
- IIS Failed Request tracing (Windows)
Below we cover each of the log file groups in some more detail.
1. Web Server Logs (IIS and Apache httpd)
1.1 Microsoft IIS web server Logs
<need something on normal IIS log>
1.2 Apache httpd web server Logs
The two major logs for apache httpd are access_log and error_log, these log the interaction with the user <-> httpd process. The httpd.conf entries are :
- Access_log - settings in httpd.conf :
The formats are defined here :
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
Occasionally it is good to supplement what is in the access_log and to get some insight to a problem. The example above it shows the %{User-Agent}i header, but you can also use that to capture cookies for example: \"%{SMSESSION}C\" . The option %T is also useful, since it logs the total time it took Ag to process the request and return the response to the user.
A complete list of LogFormat parameters is available here:
https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#formats
Note: And one final point to remember, the the access_log entry is written at the END of processing the request. So if apache httpd crashed then the requests that are currently in flight when the crash happened are NOT logged.
- Error_log - settings in httpd.conf :
The formats are defined here :
# LogLevel: values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn
For debugging, you can raise the LogLevel to debug. Apache 2.4 also has extra levels trace1 ... trace8, there are needed when you want to trace the raw data packets and SSL handshaking problems between the front end client and the httpd process. So for debugging often we can recommend :
LogLevel trace8
The apache error_log is also good place to find the exact httpd and mod_jk version numbers:
For more information refer: https://httpd.apache.org/docs/2.4/logs.html
2. Web Agent Logs (WebAgent.log, WebAgentTrace.log)
The Web Agent uses the following log files:
2.1 WebAgent.log
Introduction:
Contains program and operational-level errors. One example is when the Web Agent cannot communicate with Policy Server. The level of detail output in this log cannot be customized. Error logs contain the following types of messages:
Error messages
Contain program-level errors, which indicate incorrect or abnormal program behavior, or an inability to function as expected due to some external problem, such as a network failure. There are also operational-level errors.
This type of error is a failure that prevents the operation from succeeding, such as opening a file or authenticating a user.
Informational messages
Contain messages for the user or administrator that some event has occurred; that is, that a server has started or stopped, or that some action has been taken.
Warning messages
Contain warnings for the user or administrator of some condition or event that is unusual or indicative of a potential problem. This does not necessarily mean there is anything wrong.
How to enable WebAgent log:
1.Open Agent Configuration Object or local configuration file.
2.Set the value of the LogFile parameter to yes.
3.Specify the full path to the error file, including the file name, in below parameter:
LogFileName
Specifies the full path (including the file name) of the log file.
Default: No
Example: (Windows) web_agent_home\log\WebAgent.log
Example: (UNIX/LInux) /export/iPlanet/servers/https-jsmith/logs/WebAgent.log
Below parameters are optional:
LogAppend
Adds new log information to the end of an existing log file. When this parameter is set to no, the entire log file is rewritten each time logging is invoked.
Default: No
LogFileSize
Specifies the size limit of the log file in megabytes. When the current log file reaches this limit, a new log file is created.Archive or remove the old files manually.
Default: 0 (no rollover)
Example: 80
LogLocalTime
Specifies whether the logs use Greenwich Mean Time (GMT) or local time. To use GMT, change this setting to no. If this parameter does not exist, the default setting is used.
Default: Yes
If you use a local configuration file, your settings resemble the following example:
LogFile="yes"
LogFileName="/export/iPlanet/servers/https-myserver/logs/errors.log"
LogAppend="no"
LogFileSize="80"
LogLocalTime="yes"
LogFilesToKeep="20"
You can limit the number of log files that an agent keeps. For example, if you want to save disk space on the system that stores your agent logs, you can limit the number of log files using this parameter.
Specifies the number of agent log files that are kept. New log files are created in the following situations:
When the agent starts.
When the size limit of the log file (specified by the value of the LogFileSize parameter) is reached.
Changing the value of this parameter does not automatically delete any existing logs files which exceed the number that you want to keep. For example, If your system has 500 log files stored, and you decide to keep
only 50 of those files, the agent does not delete the other 450 files.
Setting the value of this parameter to zero retains all the log files. Default: 0
Follow these steps:
Archive or delete any existing log files from your system.
Set the value of the LogAppend parameter to no.
Change the value of the LogFilesToKeep parameter to the number of log files that you want to keep.
Webagent error log looks like below:
2.2 WebAgentTrace.log
Trace log contains detailed warning and informational messages, which you can configure. Examples include trace messages and flow state messages.
This file also includes data such as header details and cookie variables. Trace logs contain the following messages:
Trace messages
Provide detailed information about program operation for tracing and/or debugging purposes. Trace messages are ordinarily turned off during normal operation. In contrast to informational, warning, and error messages, trace messages are embedded in the source code and cannot easily be localized. Moreover, trace messages may include significant data in addition to the message itself; for example, the name of the current user or realm.
Note: For Agents on Windows platforms, set the EnableWebAgent parameter to yes to ensure that the Web Agent log gets created. If you leave EnableWebAgent set to no (the default) and set the logging parameters, the Agent log gets created only for Agents on UNIX platforms.
We can specify the location of both the error and trace log files when you configure the Web Agent.
Webagent Trace log looks like below:
How to enable Webagent Trace log:
1.Open Agent Configuration Object or local configuration file.
2. Set the TraceFile parameter to yes.
3. Specify the full path to the trace log files in following parameters:
TraceFileName
Specifies the full path to the trace log file.
Default: No default
Limits: Specify the file name in this parameter. Example: web_agent_home\log\trace.log
4. Determine what you want to record in the trace log by reviewing Trace Log Components and Subcomponents, Trace Message Data Fields and Data Field Filters.
Duplicate the default Trace Configuration File. Modify the duplicate file to include the items you want to record. Specify the full path to the duplicate copies of WebAgentTrace.conf file in the following parameter:
TraceConfigFile
Specifies the location of the WebAgentTrace.conf configuration file that determines which components and events to monitor.
Default: No default
Example:web_agent_home\config\WebAgentTrace.conf
(see next section for more details of this file)
5. Define the format of the information in your trace log file by setting the following parameters in your Agent Configuration Object or local configuration file:
TraceAppend
Adds new logging information to the end of an existing log file instead of rewriting the entire file each time logging is invoked.
Default: No
TraceFormat
Specifies how the trace file displays the messages. Choose one of the following options:
default—uses square brackets [] to enclose the fields.
fixed—uses fields with a fixed width.
delim—uses a character of your choice to delimit the fields.
xml—uses XML-like tags. A DTD or style sheet is not provided with the Web Agent.
Default: default (square brackets)
TraceDelimiter
Specifies a custom character that separates the fields in the trace file.
Default: No default
Example: |
TraceFileSize
Specifies (in megabytes) the maximum size of a trace file. The Web Agent creates a new file when this limit is reached.
Default: 0 (a new log file is not created)
Example: 20 (MB)
LogLocalTime
Specifies whether the logs use Greenwich Mean Time (GMT) or local time. To use GMT, change this setting to no. If this parameter does not exist, the default setting is used.
Default: Yes
Agent should then pickup the changes in some time, as these are dynamic ACO attributes. For some older agents a restart of the webagent may be required for the logging parameters to take effect.
WebAgentTrace.conf settings Trace Log Components and Subcomponents:
The WebAgentTrace.conf file determines the content of the trace log. You can control which components and data items appear in your trace log by modifying the settings of the WebAgentTrace.conf file on your web server. The following factors apply when editing the file:
Entries are case-sensitive.
When you specify a component, data field, or filter, the values must match exactly the options in the WebAgentTrace.conf file instructions.
Uncomment the configuration settings lines.
If you modify the WebAgentTrace.conf file before installing a new agent over an existing agent, the file is overwritten. Rename or back up the file first. After the installation, you can integrate your changes into the new file.
The CA Single Sign-On Agent can monitor specific CA Single Sign-On components. When you monitor a component, all of the events for that component are recorded in the trace log. Each component has one or more subcomponents that the agent can also monitor. If you do not want the agent to record all of the events for a component, you can specify only those subcomponents you want to monitor instead.
For example, if you want to record only the single sign-on messages for an agent on a web server, you would specify the WebAgent component and the SSO subcomponent.
Notes:
For components: - the default components are often enough, although the addiiton of AgentFunc and Agent_Con_Manager can be useful.
For data: - it is very useful to add PreciseTime, SrcFie, Function, as well as ensuring the Date, Pid, Tid, TransactionID and Messsage eg:
data: Date, PreciseTime, Pid, Tid, TransactionID, Function, SrcFile, Message
The following components and subcomponents are available:
AgentFramework
Records all Agent framework messages. (Applies only to framework agents.) The following subcomponents are available:
Administration,Filter,HighLevelAgent,LowLevelAgent,LowLevelAgentWP
AffiliateAgent
Records web Agent messages related to the 4.x Affiliate Agent, which is part of Federation Security Services, a separately-purchased product. (Applies only to framework agents.) The following subcomponent is available: RequestProcessing
SAMLAgent
Web Agent messages related to the SAML Affiliate Agent. (Applies only to framework agents.) The following subcomponent is available: RequestProcessing
WebAgent
Records all Web Agent log messages. Applies to all Agents except IIS 6.0 or Apache 2.0 Agents. The following subcomponents are available: AgentCore,Cache,authentication,Responses,Management,SSO,Filter
Agent_Functions
Records all Agent API messages. The following subcomponents are available:
Init,UnInit,IsProtected,Login,ChangePassword,Validate,Logout,Authorize,Audit,FreeAttributes,UpdateAttributes,GetSessionVariables,SetSessionVariables,DeleteSessionVariables,Tunnel,GetConfig,DoManagement
Agent_Con_Manager
Records messages related to internal processing of the Agent API. The following subcomponents are available:
RequestHandler,Cluster,Server,WaitQueue,Management,Statistics
WebAgentTrace.conf Data Fields:
The following data fields are available:
Message: Includes the actual trace message
SrcFile: Includes the source file and line number of the trace message
Pid: Includes the process ID
Tid: Includes the thread ID
Date: Includes the date
Time: Includes the time
PreciseTime: Includes the time, including milliseconds
Function: Includes the function in the code containing the trace message
User: Includes the name of the user
Domain: Includes the CA Single Sign-On domain
Realm: Includes the CA Single Sign-On realm
AgentName: Includes the Agent name being used
TransactionID: Includes the transaction ID
DomainOID: Includes the CA Single Sign-On domain OID
IPAddr: Includes the client IP address
RequestIPAddr: Includes the trace file displays the IP of the server where Agent is present
IPPort: Includes the client IP port
CertSerial: Includes the certificate serial number
SubjectDN: Includes the subject DN of the certificate
IssuerDN: Includes the Issuer DN of the certificate
SessionSpec: Includes the CA Single Sign-On session spec
SessionID: Includes the CA Single Sign-On session ID
UserDN: Includes the User DN
Resource: Includes the requested resource
Action: Includes the requested action
RealmOID: Includes the realm OID
ResponseTime: Includes the average response time in milliseconds of the Policy Servers associated with a CA Web Agent or SDK Agent and API application
Note:To output the ResponseTime to a trace log, include the component Agent_Con_Manager along with the data field ResponseTime in the WebAgentTrace.conf file or other file specified in the Policy Server Configuration Object (ACO) and restart the Policy Server. The Agent_Con_Manager component, or Agent API Connection Manager, calculates the ResponseTime each time a response is received from a Policy Server and keeps a running average. To locate the ResponseTime in the trace log, search for [PrintStats].
Some data fields are shown in the following example:
data:message,date,time,user,agentname,IPAddr
There may not be data for fields in each message, so blank fields my occur. For example, if you select RealmOID as a data field, some trace messages will display the realm's OID while others will not.
WebAgentTrace.conf Data Field Filters:
To focus on a specific problem, you can narrow the output of the trace log by specifying a filter based on the value of a data field. For example, if you are having problems with an index.html page, you can filter on resources with an html suffix by specifying Resource:==/html in the trace configuration file. Each filter should be on a separate line in the file.
Filters use the following syntax:
data_field:filter
The following types of filters are available:
== (exact match)
!= (does not equal)
The filters use boolean logic as shown in the following examples:
Action:!=get (all actions except get)
Resource:==/html (all resources ending in /html)
Determine the Content of the Trace Log:
Limit the Number of Trace Log Files Saved:
You can limit the number of trace logs that a CA Single Sign-On agent keeps. For example, if you want to save disk space on the system that stores your agent logs, you can limit the number of trace logs using the following parameter:
TraceFilesToKeep
Specifies the number of CA Single Sign-On agent trace log files that are kept. New trace logs are created in the following situations:
When the agent starts.
When the size limit of the trace log (specified by the value of the TraceFileSize parameter) is reached.
Changing the value of this parameter does not automatically delete any existing trace logs which exceed the number that you want to keep. For example, If your system has 500 trace logs stored, and you decide to keep only 50 of those files, the agent does not delete the other 450 trace logs.
Setting the value of this parameter to zero retains all the trace logs.
Default: 0
Follow these steps:
Archive or delete any existing trace logs from your system.
Set the value of the TraceAppend parameter to no.
Change the value of the TraceFilesToKeep parameter to the number of trace logs that you want to keep.
3. How to Enable Transport Layer Interface (TLI) Logging:
When you want to examine the connections between the agent and the Policy Server, enable transport layer interface logging.
1.Add the following environment variable to your web server.
SM_TLI_LOG_FILE
2.Specify a directory and log file name for the value of the variable, as shown in the following example:
directory_name/log_file_name.log
3.Verify that your agent is enabled.
4.Restart your web server. TLI logging is enabled.
<Picture coming>
4. Other useful Logs and Tools for Troubleshooting:
4.1 Fiddler:
Fiddler is an HTTP debugging proxy server application,it captures HTTP and HTTPS traffic.
To Enable HTTPS traffic decryption:
Open Fiddler-> Tools-> Fiddler Options-> HTTPS -> Check ‘Decrypt HTTS traffic’->Ok
Download: http://www.telerik.com/fiddler
4.2 Wireshark:
Wireshark is a free and open source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education.
Download: https://www.wireshark.org/
4.3 Netstat:
Netstat (network statistics) is a command-line network utility tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing),
routing tables,and a number of network interface (network interface controller or software-defined network interface) and network protocol statistics.
Refer: https://en.wikipedia.org/wiki/Netstat
4.4 strace:
strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state. The operation of strace is made possible by the kernel feature known as ptrace.
Refer: https://en.wikipedia.org/wiki/Strace
4.5 Process Monitor:
Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity.
Download: https://technet.microsoft.com/en-us/sysinternals/processmonitor.aspx
4.6 Event Viewer:
Event Viewer is a component of Microsoft's Windows NT line of operating systems that lets administrators and users view the event logs on a local or remote machine.
Refer: https://technet.microsoft.com/en-us/library/cc766042(v=ws.11).aspx
4.7 IIS Failed request tracing:
Refer: https://www.iis.net/learn/troubleshoot/using-failed-request-tracing
Thanks - Santosh
Santosh Reddy Rikkula - Support Engineer - Global Customer Success
This document is part of a series on Logging in SSO components:
Tech Tip:How to enable trace logging in SSO (aka Siteminder) Webagent
Tech Tip : Policy Server Loggings
Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server)