Causes for outage due to trusted host issue

[Jim Lundell]

Hi,

 

We had a recent outage that impacted all 'related' web server running on two RHEL6 hosts.  The web servers were Apache on two separate hosts. (We are running R12.52 SP on 2012R2.)

 

The purpose of my post is not to 'solve' the problem but rather ensure I understand all possible causes for this issue.  The web agents simply stated they were unable to contact policy server and/or find the HCO object. We resolved this issue by running smreghost on each host using the new smhost.conf file generated.  T

 

Below is a partial list I have so far...

 

1) Connection/firewall rules

        ==>Does not seem likely as running smreghost again solved the problem.

 

2) Trusted Host object modified in policy store.  

          ==>I've confirmed via the audit log that no changes were made; my assumption is these keys were not corrupted.

 

3) Corruption to the two smhost.conf files on the two hosts.  Numerous web servers share the same host file; but each host used smreghost independently for each of their own smhost files. 

     ==>Given that two both hosts had the same issue around the same time, this does not seem likely

 

4) A system change that impacted the agents algorithm to decrypt the session key in the smhost.conf file

     ==>My current theory

 

Question: Is it true the web agent on RHEL6 will encrypt using system-specific info. for the trusted host key in the smhost.conf file?   If so, what properties are used?

 

5) Others?

 

 

Thanks in advance! 

 

Cheers, Jim

[Ujwol]

I agree with your theory Jim. Yes, the encryptio/decryption of the shared secret depends on the system-specific in case of unix based systems, specifically it depends on the "hostid" of the system which in turn depends on various other parameters of the system.

 

I created a KB explaining this earlier, please refer to the same for details:

Failed Handshake between Webagent and Policy Server.

 

For your qucik refernce, here is the content of the KB:

 

Introduction:

Already successfully running webagent suddenly reports following error in webserver log.
[Error] SiteMinder Agent Unable to load SiteMinder host configuration object or host configuration file.
Path to the SiteMinder host configuration File is Empty.

Policy Server smps.log shows failed handshake errors:

[1860/2604][Mon Jul 18 2016 13:59:03][CServer.cpp:1959][ERROR][sm-Tunnel-00050] Handshake error: Shared secret incorrect for this client
[1860/2604][Mon Jul 18 2016 13:59:03][CServer.cpp:2121][ERROR][sm-Server-01070] Failed handshake with 155.35.245.129:49184

Question:

What are the reason of a Failed Handshake between Webagent and Policy Server (need to re-register the Agent)?

Environment:

All Unix environments

Answer:

On all non-Windows platforms, the agent code used to encrypt and decrypt the shared secret uses a key that is derived from a hard coded value combined with the results of calling gethostid() on the platform in question.  gethostid() is a standard C Library function that returns a 32-bit long value.

Different UNIX system implements this function differently. For e.g  Linux, AIX and solaris , the system implementation for the gethostid() C library function is not the same.

As such, SiteMinder web agent might not be able to decrypt the shared secret generated in one UNIX system when moved to other system.

Not only that, if the host ID of the same system changes (due to change in IP, hostname, mac address etc ) , the webagent will not be able to decrypt the shared secret which was originally generated on the same system, in which case you need to re-register the trusted host.

Additional Information:

gethostid Linux Man Page : http://linux.die.net/man/2/gethostid

[Jim Lundell]

Ujwol,

 

Thanks for your quick and detailed reply to my question.  This does help a lot.

 

Kudos to CA for providing this community for your clients!!!  It definitely is valuable.

 

Cheers, Jim