Blog Viewer

Tech Tip : Policy Server Loggings

By Gwan Yu Kim posted 03-27-2017 07:32 PM

  

Introduction 

The purpose of this blog entry is to show all the different types of logs that are available in CA SSO (aka Siteminder) Policy Server.  Tracing information would come in next blog. I will also be referring to some useful utilities commonly used with CA SSO for troubleshooting. 

 

CA-SSO Policy Server Logging Procedure: 

 The following gives an overview of the major components of Policy Sotre and also shows the name of (all) the logs that can be enabled and where they get their data from:

 

  Note: I may not be discussing the use all the utilities, but the diagram indicates where they would be used.

Log Files

Depending on the problem you are experiencing, Support may request one or more of the following log files:

Component

Files

Policy Server

The Policy Server log (smps.log)
The Policy Server profiler log (smtracedefault.log)
The audit log (smaccess.log)

Web Agent

The Web Agent log
The Web Agent trace log
The web server error log
The web server access log

WSS Agent

WSS Agent log
XML Processing Message Log
Web Agent trace log (WSS Agent for Web Servers only)
Application server or web server error log
Application server or web server access log

 

Policy Server Logs 

  • smps.log: The Policy Server log file records information about the status of the Policy Server and, optionally, configurable levels of auditing information about authentication, authorization, and other events in the Policy Server log file. When the Policy Server is started, its version information and configurations are recorded in the Policy Server log.

  • "smpolicysrv -stats" command will show the output on smps.log.

Cron job or Windows Scheduler can be configured to run "smpolicysrv -stats" command to get the policy server statistics.

** Explanation of the fields:

- Msgs = Number of thread pool messages handled
- Waits = # of times Dequeue had to wait for a message before timeout reached
- Misses = # of times waiting thread woke up to find no message and timeout reached
- Max HP Msg = # maximum number of High Priority messages on the queue since the last reset - of stats
- Max NP Msg = # maximum number of Normal Priority messages on the queue since the last reset of stats
- Current Depth = # message in the queue at the time of executing –stats
- Max Depth = # maximum number of messages on the queue since the last reset of stats
- Current High Depth = # High priority messages in the queue at the time of executing –stats
- Current Norm Depth = # Normal priority messages in the queue at the time of executing -stats
- Current Threads = # threads running at the time of executing -stats
- Max Threads = maximum thread number reached

Connections:

- Current = # of agent connections
- Max = maximum # of connections since last reset
- Limit = maximum allowable connections
- Exceeded limit = # of times exceeded the limit

"Busy threads" refers to the number of thread is currently active on stack and processing request.
"Wait" and "missed" are historical record, does not mean much unless an incident is happening, could be an indicator of how much threads are utilized.
"Reset" can either be policy server restarted or admin intentionally flushed the statistics by command line options

  • smtracedefault.log: Policy Server Trace log with configuring the Policy Server Profiler. Check the "Enable Profiling" option, then click on the "Configure Settings" button.


      * Config File – config file saved under C:\Program Files\CA\siteminder\config\smtracedefault

** More detail of “Configure the Policy Server Profiler” can be found from document.
https://docops.ca.com/ca-single-sign-on/12-6-01/en/configuring/configure-the-policy-server-profiler

- Configuration Settings: Useful configuration for troubleshooting

-   /Component

Functional group of components that will be logged.

Note: Often all except “Server/Policy_Object_Cache” generates lots of log lines – so often good to leave out.

-   / Data

Items:   <data item: PreciseTime, SrcFile, Function, Pid, Tid, Message  to give better results.>

-   / Filter


      (Expression can be added to exclude items – not widly used but occasionaly helpful)

 

Siteminder Policy Trace Analysis

https://communities.ca.com/message/97562726

“Siteminder Policy Trace Analysis” is a java Policy Log analysis tool that we have been using in CA Support for a while now for analysis of various SiteMinder logs.

 - Additional tips are available from author's blog post:
Tech Tip - [PreciseTime] gives better Graphs & Stats with SMTraceAnalysisTool 

Other Loggings

  • Xtrace – xTrace is an XPSConfig option that captures XPS errors in Policy Store. This option is available from CA SiteMinder Release 12.51.

 

 

 

 Type the number to enable
Need to do “U” to write the entries.

The config file Saves to :

C:\Program Files (x86)\CA\siteminder\config\XPS.cfg

** More detail of xTrace can be found from document.

How to Use xTrace - CA Single Sign-On - 12.8 - CA Technologies Documentation 

  • Smaccess.log – audit log

 

Useful Tools for Troubleshooting

  • Wireshark: Wireshark is a free and open source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol.

Usage: Decode SSL, LDAP protocol.
https://www.wireshark.org/ 

  • Netstat: netstat (network statistics) is a command-line network utility tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface (network interface controller or software-defined network interface) and network protocol statistics. 

 

  • Top: Top command displays processor activity of your Linux box and displays tasks managed by kernel in real-time. It'll show processor and memory are being used and other information like running processes.

 

  • Fiddler: Fiddler is an HTTP debugging proxy server application.

    - To enable HTTPS traffic decryption:
    Open Fiddler-> Tools-> Fiddler Options-> HTTPS -> Chec ‘Decrypt HTTS traffic’->
    http://www.telerik.com/fiddler

 

  • Debug Diagnostic tools: The Debug Diagnostic Tool (DebugDiag) is designed to assist in troubleshooting issues such as hangs, slow performance, memory leaks or memory fragmentation, and crashes in any user-mode process. 

      https://www.microsoft.com/en-us/download/details.aspx?id=49924

 

 

  • Strace: strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state. 

        -   Command: strace -p <pid>

  • Pstack: pstack attaches to the active processes named by the pids on the command line, and prints out an execution stack trace, including a hint at what the function arguments are.If the process is part of a thread group, then pstack will print out a stack trace for each of the threads in the group.
    If symbols exist in the binary (usually the case unless you have run strip(1)), then symbolic addresses are printed as well.

 

  • Core dumps: Core dump, memory dump, or system dump consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally. (Please refer to ‘Debug Diagnostic tools’ above in case Core Dump file was not generated properly.)
  • Pkgapp: Pkg_app is a script that can take a a core/gcore or process id and gather all the libraries required by Sun Support to debug a core/gcore file. (More detail of how to get stack trace with pkgapp can be found from following link- https://communities.ca.com/people/SungHoon_Kim/blog/2016/02/25/collecting-pkgapp-and-how-to-get-the-stack-trace)

 

Work with Support

If you require assistance from the CA Single Sign-On Support team, there is specific information you can gather and include when opening a Support ticket. Including as much information as possible helps to reduce the amount of time it takes the Support team to resolve the issue.

 

Note: If you are attaching log files as part of your Support engagement, be sure that the set of files matches. Also, ensure that all the files are from the same time as when the issue occurred.

 

Have a continuing nice time with your logging. 

 

Cheers - Gwan

---- Gwan Yu Kim Snr Support Engineer - Global Customer Success

10 comments
7 views

Comments

10-29-2018 07:09 PM

Thanks, updated with later link.

10-26-2018 03:14 AM

Looks like the link

 

https://docops.ca.com/ca-single-sign-on/12-6-01/en/troubleshooting/how-to-use-xtrace

 

lead to a 404. Could you correct it ?

 

Thanks,

 

Patrick

09-24-2017 09:03 PM

I experienced the same issue just now..apparently you need to run Policy server management console (smconsole) as Administrator..

09-19-2017 03:03 AM

Restart Policy Server service, and also check if the following file exists.

C:\Program Files (x86)\CA\siteminder\config\smtracedefault.txt

09-13-2017 03:38 PM

Thank you for this post.

In my case, the profiler configuration settings is empty! I don't options to  select and, if I try to load template, nothing is show! Anyone knows what's happening ? 

I Want to Debug the reason for SSO permits anonymous access to a protect resource! 

 

Thank you!

03-30-2017 11:31 AM

This is an outstanding article. I look forward to reading the next two postings. I can see using this content as an elective in a 300-level SSO Advanced Topics course.

03-28-2017 12:25 AM

Impressive!

Very detailed and comprehensive.

 

For tools, it would be good to add Policy Store reader as well.

This tool will check objects and their references to tell if there are any orphaned objects.

It would connect directly to the policy store to view the contents or can read smdif or xml export data.

 

And log log viewing, glogg and notepad++

glogg is very good at reading large files and focusing on specific string or transaction.

Filtering is very good.

 

notepad++, you know what it does

 

Others are like process monitor which will help to view if there are permission issues or if any files are missing.

03-27-2017 08:21 PM

Thank you for sharing this tip with the community Gwan!

Tech Tip : Policy Server Loggings 

03-27-2017 07:34 PM

Excellent ~~