Symantec SiteMinder

Summary:

In this guide, we will discuss about the steps performed during web agent initialization.

Then, we will also deep dive into some of the common agent initialization issues and discuss approaches to troubleshoot and resolve theses issues.

Environment:

• Web Agent : 12.5 and above
• OS : ANY 

For this tech tip , we will test on following platform :

• Web agent version : 12.52 SP1 CR7
• Web Server : Apache 2.2
• Web Server OS : RHEL 6.5 64 bit

Web Agent Startup Process 

On the high level the web agent startup process happens in the following order :

1. Read WebAgent.conf
2. Locate the path to the SmHost.conf file from WebAgent.conf and read SmHost.conf
3. Identify the following details from SmHost.conf :
   • Policy server IP ( this policy server is used only for the initial bootstrapping)
   • Shared Secret
   • Trusted Host Name (hostname)
   • Host configuration object (HCO)#agentname="<AgentName>, <IPAddress>"
HostConfigFile="/opt/CA/webagent/config/SmHost.conf"
AgentConfigObject="aco_rhel65"
EnableWebAgent="YES"
ServerPath="/etc/httpd/conf"
#localconfigfile="/etc/httpd/conf/LocalConfig.conf"
LoadPlugin="/opt/CA/webagent/bin/libHttpPlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libSessionLinkerPlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libAffiliate10Plugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libSAMLAffiliatePlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libeTSSOPlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libIntroscopePlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libSAMLDataPlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libOpenIDPlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libDisambiguatePlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libOAuthPlugin.so"
#LoadPlugin="/opt/CA/webagent/bin/libCertSessionLinkerPlugin.so"
AgentIdFile="/etc/httpd/conf/AgentId.dat"


     Figure : WebAgent.conf

     hostname="th-rhel65-4"
sharedsecret="{RC2}ovOEr7teMKP9xpKisg157/t4T1tqGXwNT0SWGsfi1QnajkcDjumFEmF9kBbw1d2MZb8CGf2ueSfWkfKmZEWrYVeM3hZ/HRI1F2bh6v4lBQq9uqi0Lp2iIQJb0flOY2oUGLmDE/iZFDQt3ceo7aCij9YQY72YD2iLLtJTKJGlu6e2nJcMzGlf2roiNfd2MwMV"
sharedsecrettime="0"
enabledynamichco="NO"
hostconfigobject="hco"
#Add additional bootstrap policy servers here for fault tolerance.
policyserver="shruj01-i1849.ca.com,44441,44441,44441"
requesttimeout="60"
cryptoprovider="ETPKI"
fipsmode="COMPAT"

         Figure : SmHost.conf

4. Establish Agent API connection with the Policy server listed in the SmHost.conf file. This includes 3 way handshake (more details below ).
5. Read HCO (Host configuration object) from the policy server/policy store.
6. Establish the Agent API connection with the primary policy server listed in the HCO ( this could be same or different from the bootstrap policy server as listed in SmHost.conf)
7. Once the connection is established with the primary policy server listed in the HCO , read ACO (Agent configuration object as per the WebAgent.conf)
8. Initialize agent log/trace file as per the ACO configuration.

Three way agent to policy server handshake

1. Agent opens a TCP socket connection with the policy server.
2. Agent sends a Hello message which includes following info among other details :
   • MD5 Hash of shared secret and trusted host name combined (stronger encryption is used if using FIPS only mode)
   • Trusted host name
3. Policy server validates the shared secret based on the trusted host name passed. It may validate both current and previous shared secret from the policy store against the shared secret sent by agent. If the shared secret validation is successful , policy server sends Hello Reply message which consists of following info among other details :
   • Session Keys
   • New shared secret (optional - this is sent only if the agent currently doesn't have the current shared secret)
   • New shared secret generated time (optional)
4. Agent sends Hello Confirm message encrypted with the Session Keys previously sent by Policy server.
5. (Optional ) Agent updates the SmHost.conf file with the new shared secret.

                                             Figure : Three way agent to Policy server handshake

Web Agent Initialization Error Codes :

Basic troubleshooting:

• On UNIX platform, ensure you have sourced the web agent environment script (ca_wa_env.sh) before starting the webserver.[root@rhl65 webagent]# pwd
/opt/CA/webagent
[root@rhl65 webagent]# source ./ca_wa_env.sh
[root@rhl65 webagent]#



  On Windows, the web agent environment variable are set as system environment variable. Ensure that the user running the web server process has access to these system environment variables.

  
• Ensure path to the host configuration file (HostConfigObject) is valid in WebAgent.conf. 
• Ensure the name of the agent configuration object (AgentConfigObject) is valid in WebAgent.conf (This is case sensitive field and need to match against the name of the ACO in the policy store)
• Ensure that the user under which web server process runs has write permission to SmHost.conf (This is optional requirement. This is required only if the shared secret rollover functionality is used. )
• Ensure that either defaultagentname or the agentname ACO parameter must be set in the policy store.
• Ensure that the policy server FQDN/IP is specified in SmHost.conf file. Also ensure that you can ping and resolve the DNS for the specified policy server from the web server host.
• Ensure that a valid Policy server FQDN/IP is specified in Host configuration object in the policy store. Also ensure that you can ping and resolve the DNS for the specified policy server from the web server host.
• If there are multiple policy server IP listed in SmHost.conf and HCO , it is usually best to start with just one policy server and comment the remaining servers out. This will help and ease the troubleshooting.
• On Unix platform, if there are multiple web agent instance running on the same box, ensure that a unique ServerPath is specified for each instance. 
• Ensure that you can telnet to the Policy server ports from web server

Test connectivity to all the ports - accounting, authentication , authorization 

Advance troubleshooting:

Following logs will be required to be analyzed for advanced troubleshooting:

Linux :

• web server error/startup log
• policy server logs (smps.log) and trace log (smtracedefault.log)

At minimum use following profiler for policy server trace log :

• strace log from the web server startup ( this will help to identify any issue related to file permission/library dependency/environment variable etc )

  strace -Ff -t -i -v -o strace.log -s 16384 <command to start web server>

  e.g.

• tcpdump from webserver

tcpdump -i <network interface> -s 65535 -w <some-file.pcap>
To find the available network interface, you can run following command : ifconfig

then, run the tcpdump command as below :

• web agent logs & trace ( note : if there is an agent initialization issue, these logs will most likely not be created as they are created only at the end of init process, however it is good to get this configured just in case )At minimum enable following profiler for web agent trace :components:  AgentFramework, HTTPAgent, WebAgent, Agent_Con_Manager
data: Date, Time, Pid, Tid, SrcFile, Function, ResponseTime, IPAddr, IPPort, AgentName, Resource, User, Threshold, Throughput, MinThroughput, MaxThroughput, HandleCount, BusyHandleCount, FreeHandleCount, State, ClusterID, Message


Solaris/AIX : 

All the logs from Linux except strace logs are applicable for Solaris/AIX based system.

The Linux's strace equivalent is truss in Solaris.

You can capture truss output from web server startup as below :

Windows:

• For windows the equivalent of strace is process monitor logs (procmon.exe)

Process Monitor - Windows Sysinternals | Microsoft Docs 

• TCPDump can also be replaced with the wireshark network capture.

Wireshark · Go Deep. 

• Event Viewer would also be helpful.

Commons Issues :

Problem : Policy server smps.log shows following error :

The web server log (Apache for our test) shows agent initialization errors:

No Agent Log/Trace is created.

Cause :

If no changes has been done either in the policy server side or on the agent side since the last working state, then this error indicate a possible change in the hostid (for unix based system) on the web agent side.

On all non-Windows platforms, the agent code used to encrypt and decrypt the shared secret uses a key that is derived from a hard coded value (Web Agent Host Key) combined with the results of calling gethostid() on the platform in question. gethostid() is a standard C Library function that returns a 32-bit long value.

Different UNIX system implements this function differently. For e.g Linux, AIX and solaris , the system implementation for the gethostid() C library function is not the same.
As such, SiteMinder web agent might not be able to decrypt the shared secret generated in one UNIX system when moved to other system. Not only that, if the host ID of the same system changes (due to change in IP, hostname etc ) , the webagent will not be able to decrypt the shared secret which was originally generated on the same system.

Testing :

Set

• hostname = rhel65_1.ca.com
• IP =192.168.0.6 (in the hosts file)

hostid gives output as a8c00600

Agent initializes fine with the 3 way agent to PS handshake being successful

Now, change the IP address to 192.168.0.7 in the hosts file with everything else remaining the same.

This time hostid command gives a different result : a8c00700

3 way agent to PS handshake now fails with the following error in smps.log :

Also, note in the network capture, the encrypted data in front of the trusted hostname is now different.

If the shared secret+trusted hostname+hostid combination is same, the encrypted data should remain same.

Resolution :

As we saw above, a simple change in the IP address resulted in the change in the hostid in RHEL system. This in turn invalidated the shared secret stored in SmHost.conf. There could be more factor contributing to the change in the hostid which is dependent on the platform being used.

The only way to fix this issue is by re-registering the trusted host or reverting to the previous hostid( reverting to previous IP address in this case).

smreghost -i <policyserver_ip>:44441,44442,44443 -u "siteminder" -p <siteminder super user password> -hn <trustedhost> -hc  <hco> -cf COMPAT -f <Path_To_SmHost.conf> -o

e.g

If it is expected to have frequent changes in the web server IP address (due to reboot/change in network interface/dns server etc.) , it is recommended to specify a static hostid.

In RHEL you can do this by running command : genhostid. The static hostid is then stored in /etc/hostid file.

(Please refer to your respective OS documentation on how to set static hostid)

To be continued....