Determine Server Queue Length using smpolicysrv -stats

[Richard Leto]

Hi All,

 

Just wanted to make sure what value we should look for Server Queue Length, in output of smpolicysrv -stats.

 

[5752/5108][Wed Sep 30 2015 15:34:51][CServer.cpp:7500][INFO][sm-Server-02360] Server 'Stats' command received

[5752/5108][Wed Sep 30 2015 15:34:51][CServer.cpp:4638][INFO][sm-Server-01990] ===================================================================================

[5752/5108][Wed Sep 30 2015 15:34:51][CServer.cpp:4639][INFO][sm-Server-02000] System Statistics

[5752/5108][Wed Sep 30 2015 15:34:51][CServer.cpp:4656][INFO][sm-Server-02020] Thread pool limit: 8

[5752/5108][Wed Sep 30 2015 15:34:51][CServer.cpp:4677][INFO][sm-Server-02030] Thread pool: Msgs=26 Waits=22 Misses=62 Max HP Msg=0 Max NP Msg=3 Current Depth=0 Max Depth=3 Current High Depth=0 Current Norm Depth=0 Current Threads=8 Max Threads=8 Busy Threads=0

[5752/5108][Wed Sep 30 2015 15:34:51][CServer.cpp:4685][INFO][sm-Server-02040] Connections: Current=25 Max=27 Limit=1000 Exceeded limit=0

===================================================================================

This question is raised because the value showed in OneView Monitor for Server Queue Length is not available in the output, enclosing the snippet for oneView monitor, is there something we are missing? Any suggestions. We are running R12.52sp1cr02

 

[wonsa03]

Hi Richard.Leto,

 

ServerQueueLength indicates the number of entries in the normal queue, which is corresponding to the Current Norm Depth in the stats.

[Richard Leto]

@wonsa03 Thanks for your reply, but why the value is not same in one View monitor, for instance the screen shot I have attached it showing 1 server Queue length but -stats command is showing 25. Why this difference.

[wonsa03]

Hi Richard.Leto,

 

Current Norm Depth is 0 from the stats. Did you try refreshing the OneView Monitor page at the same point you output the stats?

 

By default, OneView Monitor contents are updated every 30 seconds. However, you can configured to manual updates which require user to reload the browser page.

[Richard Leto]

Hi wonsa03

 

   We had a database issue due to which the connection went high and policy server was not serving the requests.

Looking at the below 2 snippets Current Norm Depth=0 but it has exceeded the limit. Why this difference will happen, The current Norm Depth should also have showed the same value I hope as exceeded Limit. IF that's Server Queue Length. Please correct me if I am wrong.

 

[45220/45972][Tue Jun 23 2015 10:50:12][CServer.cpp:7405][INFO][sm-Server-02360] Server 'Stats' command received

[45220/45972][Tue Jun 23 2015 10:50:12][CServer.cpp:4599][INFO][sm-Server-01990] ===================================================================================

[45220/45972][Tue Jun 23 2015 10:50:12][CServer.cpp:4600][INFO][sm-Server-02000] System Statistics

[45220/45972][Tue Jun 23 2015 10:50:12][CServer.cpp:4617][INFO][sm-Server-02020] Thread pool limit: 8

[45220/45972][Tue Jun 23 2015 10:50:12][CServer.cpp:4637][INFO][sm-Server-02030] Thread pool: Msgs=24063628 Waits=23385650 Misses=157297694 Max HP Msg=18 Max NP Msg=110 Current Depth=0 Max Depth=110 Current High Depth=0 Current Norm Depth=0 Current Threads=8 Max Threads=8

[45220/45972][Tue Jun 23 2015 10:50:12][CServer.cpp:4645][INFO][sm-Server-02040] Connections: Current=19 Max=256 Limit=256 Exceeded limit=394

====================================================================================

 

 

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:7500][INFO][sm-Server-02360] Server 'Stats' command received

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:4638][INFO][sm-Server-01990] ===================================================================================

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:4639][INFO][sm-Server-02000] System Statistics

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:4656][INFO][sm-Server-02020] Thread pool limit: 8

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:4677][INFO][sm-Server-02030] Thread pool: Msgs=198569 Waits=191900 Misses=1196644 Max HP Msg=11 Max NP Msg=142 Current Depth=142 Max Depth=142 Current High Depth=0 Current Norm Depth=142 Current Threads=8 Max Threads=8 Busy Threads=8

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:4685][INFO][sm-Server-02040] Connections: Current=150 Max=150 Limit=256 Exceeded limit=0

[5688/3600][Tue Sep 22 2015 05:16:49][CServer.cpp:4688][INFO][sm-Server-01990] ===================================================================================

[5688/5608][Tue Sep 22 2015 05:24:33][CServer.cpp:3195][ERROR][sm-Server-07017] Connection request rejected. Connection limit of 256 exceeded.

[wonsa03]

Hi Richard.Leto,

 

The stats covers 2 areas:

• Threads usage:

          Policy Server is running with 8 worker threads (Thread pool limit: 8).

 

          The Policy Server worker thread pool consists of two separate thread pools that independently process High Priority and Normal Priority messages. A reactor thread receives all incoming Web Agent requests and           depending on the message type, passes them to either the High Priority or Normal Priority queue. High Priority messages include Agent connection requests. Normal Priority messages include user messages,           such as authentication and authorization requests.

 

• High Priority messages—the default number of worker threads in the thread pool available for High Priority messages is five and the maximum number is 20. You can change the default value by adding and setting the PriorityThreadCount registry key.
• Normal Priority messages—the default number of worker threads in the thread pool available for Normal Priority messages is eight. You can add additional worker threads by modifying the Maximum Threads setting field on the Data tab in the Policy Server Management Console.

 

          Current Norm Depth=142 indicates that the normal priority messages are waiting in the queue for available worker threads. You can increase the worker threads to accommodate the load.

 

• Connection utilization:

          Policy Server is running with maximum of 256 agent connections (Limit=256). All connections share the thread pool to fulfill requests.

 

          Connections: Current=150 Max=150 Limit=256 Exceeded limit=0

 

          Current is the number of agent connections currently attached to the Policy Server service.  Max is the maximum number ever attached at once to the service. 

          Exceeded limit notes if the Limit has been reached and further agent connections have been attempted (the connections attempted past the limit will be rejected)

 

In short, Current Norm Depth is not not the same as Exceeded limit.

[Satyendra Singh]

Is Current Depth related to number of users reqesting to login?

How can we correlate with application login?

[banst03]

Kelly (wonsa03) had an excellent response.  So there is not much to add there.

 

but I do want to mention that having the policy server queuing up messages does not really mean that it is a bad thing.  I see sometimes there is a kneejerk reaction that if the queue depth is greater than zero, people will increase the number of threads (to a detriment).

 

Set up your -stats a cron or task to generate -stats every 2 to 5 minutes.  Then graph it, or use the tools available on the community page to graph it.  The 142 might be a small spike.  Or it may be sustained at that level.  But if you are servicing your requests per your defined SLA, then I would leave it.

 

The suggestion below to scale the threads base on the number of cpus/cores is fine.  As long as you have the data to measure the before and after to see if it is actually helping your transaction times.

[Ujwol]

Also have a look at this KB :

Information about the Stats command in Policy server logs

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec458114.aspx

 

and this :

 

OneView Monitor Overview

[Anon Anon]

Seems the link for TEC458114 is not found any more.

[Saravanan Velusamy]

Ujwol Looks TEC458114  is not available.. Would you be able to help?

[Ujwol]

Hi Saravanan,

 

That KB is deprecated. 

Please refer this : Tech Tip : CA Single Sign-On :: Policy server :: STATS Information (Updated) 

 

Regards,

Ujwol

[Saravanan Velusamy]

Thanks Ujwol

[Robert Maiello]

Richard,

 

Looking at your stats without knowing the size of your server or environment I'm still concerned.    You have the max threads set to 8.  And the

max connections set to 256.     I think these may be the CA defaults?     

 

Unless you have a very small policy server and a small number of webagents these limits will not work for anything but the lab or classroom.

 

First  the connections;   You should set the number of open files for your OS user(I only know unix) to a number as large as or larger than

your Siteminder Max Connection setting.     As for the number of Max connections to set... the limit your hitting here (256) becomes artificial..

is your policy server CPU and directory so slow and small that you think it cant serve 384 connections.     I would set it to something higher

like 2048 to start.   It should be a limit that you think the server can serve but not so low as to make the server stop servicing requests when

its CPU etc can handle more.

 

 

As for the number of threads.  This is probably why you have queueing..   It depends on your server  but if Intel..too high and you get a lot of context switching.. too low   (8 threads) and the server does not have enough threads to service all the requests..you get queueing  (142). 

You should look at the number of cpus presented to your OS... CAs rule of thumb of 2x # of cpus  would be 16 if you have 8 cores/cpus presents to the OS.     But even if you have 4 CPUs and set max threads to 8 it seems like the software may still not have enough to do the work.    I would argue that 8 is still an artificial limit and 16 should be the minimum.    CA will say it all varies based on server, env etc and

that this is beyond the scope of tech support.   That may be true but the defaults will give you problems very quickly.

 

Just wanted to throw that out there..   I hope your server is doing better.