How to set a benchmark for Login: Average response time for a SM policy server environment monitored by CA APM for CA SSO

[Naveen A]

Hello folks,

 

we are running four SiteMinder policy servers in load-balancing mode (which is configured in HCO). All the four policy servers are monitored by CA APM for CA SSO product. currently we set the Login: Average response time for 100ms. But sometime we see it increases in between 200 to 600ms for some minutes and drop back to less than 100ms. we are seeing this more frequently after we upgrade the smps from 12.52 SP1 CR06 to 12.8 and at the same time we upgraded the CA APM version as well. 

 

Alert from APM:

Alert Name: Siteminder ES - PROD Operations: Login: Avg RT - 100ms
Time Triggered: 12/27/18 4:10:00 PM CST
Alert Status: Danger

Problem Details:
Danger SuperDomain|ourservername|EPAgentProcess|EPAgent|SiteMinder|Policy Server|ourservername.domainname.com|Operations|Login:Average Response Time (ms): 169

 

At the same time i gathered the smpolicysrv stats on that server:

[102269/139648894490368][Thu Dec 27 2018 16:10:01][CServer.cpp:4841][INFO][sm-Server-02000] System Statistics
[102269/139648894490368][Thu Dec 27 2018 16:10:01][CServer.cpp:4847][INFO][sm-Server-02010] Available file descriptors: 44032
[102269/139648894490368][Thu Dec 27 2018 16:10:01][CServer.cpp:4858][INFO][sm-Server-02020] Thread pool limit: 50
[102269/139648894490368][Thu Dec 27 2018 16:10:01][CServer.cpp:4878][INFO][sm-Server-02030] Thread pool: Msgs=50505813 Throughput=76.473672/sec Response Time=35.227417ms Wait Time In Queue=94.885787ms Max HP Msg=7 Max NP Msg=8417 Current Depth=0 Max Depth=8417 Current High Depth=0 Current Norm Depth=0 Current Threads=50 Max Threads=50 Busy Threads=21
[102269/139648894490368][Thu Dec 27 2018 16:10:01][CServer.cpp:4886][INFO][sm-Server-02040] Connections: Current=2832 Max=8839 Limit=32768 Exceeded limit=0
[102269/139648894490368][Thu Dec 27 2018 16:10:01][CServer.cpp:4889][INFO][sm-Server-01990] ===================================================================================

 

Based on this data can we determine why there is an increase in login: Avg Response time?

 

Environment details:

smps version : 12.80.100.1775

policy server OS environment : RHEL 6.10 x86 64

CA APM agent version : 13.2.0

CA Interscope version: 10.7.0.115

 

Can someone please help me on this?

 

Thanks

Naveen

 

[Patrick Dussault]

Hi Naveen007,

 

The server isn't permanently loaded, but the Max NP Msg shows that at
a given time, the threads were all busy, and the queue piled up for a
time.

 

In order to point out why you get an increase in the average response
at login, you do need to put the Policy Server traces (Profiler), and
then once you get the traces, run the Policy Server traces tool
analyser to identify the bottle necks.

 

Siteminder Policy Trace Analysis
https://communities.ca.com/message/97562726-siteminder-policy-trace-analysis

 

Configure the Policy Server Profiler
https://docops.ca.com/ca-single-sign-on/12-8/en/configuring/configure-the-policy-server-profiler

 

From the log line :

 

Thread pool:
Msgs=50505813
Throughput=76.473672/sec
Response Time=35.227417ms
Wait Time In Queue=94.885787ms
Max HP Msg=7
Max NP Msg=8417
Current Depth=0
Max Depth=8417
Current High Depth=0
Current Norm Depth=0
Current Threads=50
Max Threads=50
Busy Threads=21

 

We note :

 

- When a thread goes to the queue to get a new request to process, it
has to wait an average of 94.8ms, which seems to indicate that the
Policy Server isn't loaded.
- The maximum amount of High Priority requests in queue is 7 which seems to indicate that the
Policy Server isn't loaded.
- The maximum amount of Normal Priority requests in queue is 8417
which seems to indicate that at a given moment the Policy Server
face a bottle neck to process request (in many situation, it is
because of a backend server performance or network latency)

 

ref.:

Tech Tip : CA Single Sign-On :: Policy server :: STATS Command
https://communities.ca.com/thread/241734066

 

I hope that helps you,

 

Best Regards,
Patrick