Siteminder R12 Slowness issue

[Saravanan Velusamy]

Hello All,

 

 

I am facing slowness issue in one of the Siteminder Policy Server...

 

Checking with you to share my analysis and get your suggestions to check further...

 

 

My Env:

 

FullVersion=12.0.302.328

 

OS :SunOS 5.10

 

 

When Server is slow:

 

 

Errors in smps.log:

When we observe slowness and policyserver console not responding we could see the below in smps.logs

 

 

socket 32 error from multiple truster hosts.. Ex.:[ERROR] Failed to send response on session # Number : <trusted host name and port> Socket error    32

   

 

Increase in current depth queue:

 

[7822/41][Fri Jan 08 2016 09:04:28][CServer.cpp:4406][INFO] ===================================================================================

[7822/41][Fri Jan 08 2016 09:04:28][CServer.cpp:4407][INFO] System Statistics

[7822/41][Fri Jan 08 2016 09:04:28][CServer.cpp:4411][INFO] Available file descriptors: 65536

[7822/41][Fri Jan 08 2016 09:04:28][CServer.cpp:4424][INFO] Thread pool limit: 75

[7822/41][Fri Jan 08 2016 09:04:28][CServer.cpp:4444][INFO] Thread pool: Msgs=18366 Waits=7667 Misses=29383 Max HP Msg= 158 Max NP Msg=  222 Current Depth=   214 Max Depth=    251 Current High Depth=       0 Current Norm Depth=      214 Current Threads=        75 Max Threads=         75

[7822/41][Fri Jan 08 2016 09:04:29][CServer.cpp:4452][INFO] Connections: Current=333 Max=375 Limit=7992 Exceeded limit=   0

[7822/41][Fri Jan 08 2016 09:04:29][CServer.cpp:4455][INFO] ===================================================================================

 

[7822/41][Fri Jan 08 2016 12:55:49][CServer.cpp:4444][INFO] Thread pool: Msgs=18318 Waits=12553 Misses=76866 Max HP Msg= 244 Max NP Msg=  205 Current Depth=    98 Max Depth=    273 Current High Depth=       0 Current Norm Depth=       98 Current Threads=        75 Max Threads=         75

[7822/41][Fri Jan 08 2016 12:55:49][CServer.cpp:4452][INFO] Connections: Current=211 Max=380 Limit=7992 Exceeded limit=   0

 

 

Sample Logs: When working normally..

 

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4406][INFO] ===================================================================================

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4407][INFO] System Statistics

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4411][INFO] Available file descriptors: 65536

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4424][INFO] Thread pool limit: 75

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4444][INFO] Thread pool: Msgs=164671 Waits=146253 Misses=927104 Max HP Msg= 244 Max NP Msg=  281 Current Depth=     0 Max Depth=    281 Current High Depth=       0 Current Norm Depth=        0 Current Threads=        75 Max Threads=         75

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4452][INFO] Connections: Current=72 Max=380 Limit=7992 Exceeded limit=   0

 

[7822/41][Sun Jan 10 2016 00:00:16][CServer.cpp:4455][INFO] =========

 

 

It is happening intermidently... We have removed this server from all the cluster to avoide load and using this only for SM Admin operation like FSSUI...

I am thinking the sloness issues is co-related with the Queue Depth increase and hence server is not responding...

 

Could you help me to take it further?

[Ujwol]

Hi Saravanan,

 

The main issue I see here is that the Policy server thread usage is maxing out.

As you could see, Current Threads is equal to Max Thread and at one point the Queue depth increased to 281.

 

Current Threads=        75

Max Threads=         75

 

Couple of suggestions:

 

*) Using SiteMinder Policy Trace analysis tool find out the Long running transactions. It will most likely also indicate, if there is any deadlocks.

  Then depending upon where the problem your configuration could be tuned. For e.g if the bottleneck is happening at LDAP connections.you can consider increasing number of LDAP banks.

Siteminder Policy Trace Analysis

 

*) As the Max Depth is not drastically that high, and your Thread pool limit is quite low, you might even considering increasing your Thread pool limit to say 256.

However, even if you do this , I would still suggest that you do the analysis on the current usage and find out where the bottleneck is.

 

*) You have not indicated how many policy servers are in the cluster and if other Policy servers in the cluster are also experiencing similar issue. One other suggestion would be to add another load balancing Policy server in the mix in the cluster.

 

Cheers,

Ujwol