Intermittent webagent errors

[Vivek_S]

Hi we are getting intermittent errors in agent logs , just for 1 server, Restarting the server didn't help. On turning additional tracing on the agents it seems that the agent is marking policy server as not responding . We are getting isprotected -1 and -2 errors in webabent logs and in agent trace we see that its marking policy server as not responding however policy servers are working fine without any issues during this time , i dont see any queues or socket errors and all other 100+ apps are working fine , Only issues seems to be occurring from the 1 system only which has 3-4 apps on it. We set agent wait time in Webagnt.conf ~100 seconds but nothing seems to hlep.

 

This is what i see in the trace logs 

 

Line 91546: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2736][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][Entered Function][][][][][][][][][][][]
Line 91547: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2740][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][--------------------
Line 91549: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2744][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][Server xxxxxxx: Current total capacity: 0, current throughput: 0
Line 91551: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2744][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][Server xxxxxxx: Current total capacity: 0, current throughput: 0
Line 91553: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2744][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][Server xxxxxxx: Current total capacity: 48, current throughput: 1
Line 91555: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2749][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][--------------------
Line 91557: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2767][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][Changing Best server index was 0 is now 2
Line 91559: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2791][GetServer][][][][][][][][][][][][3][][][ACTIVE][3][Selected server xxxxxxx: Current total capacity: 47, current throughput: 1][][][][][][][][][][][]
Line 91560: [10/13/2016][12:25:54.154][8454336][106657][SmClient.cpp:2796][GetBestServerIndex][][][][][][][][][][][][3][][][ACTIVE][3][Leaving Function][][][][][][][][][][][]
Line 92072: [10/13/2016][12:26:54.154][8454336][106657][SmClient.cpp:2576][ProcessRequest][][][][][][][][][][][][3][][][ACTIVE][3][Request failed with 2 error code.][][][][][][][][][][][]

[Karmeng]

Hi Vivek,

 

It seees you have CA Support issue open for this 00532360 and this is with PS R12SP3CR12.

Based on the snippet of web agent log, it looks like web agent is selecting best server to work with. Not sure why it return error.

Policy server trace log is best place to start with.

If the policy servers are working fine with other web agents and only this particular agent having issue, I suspect it could be either configuration on the web server (compare httpd.conf with other similar web servers that are working fine) OR some network issue.

 

Regards,

Kar Meng

[Vivek_S]

Hi kar,

 

thanks for the response , policy Server queue's look clean , no queuing on the server , there are 4-5 webservers on this server and only 2-3 are having issue . I am trying to understand why would it webagent report policy server server capacity as 0 and througput at 0 .

 

could it be soemthing that it is not able to get to the policy server to get capacity and it returns 0 for cpacity and throughput.

 

the requests that report -1 and -2 dont make it to the policy setver obviosuly so not sure how to track this issue.

 

thanks

[Karmeng]

Hi Vivek,

 

Network trace is the best to trace the request from web server to policy server. It seems the issue is related to firewall in between the web server and policy server that reset the connection (based on issue 00532360).

This align to initial investigation in the case that it related to Network device in between web server and policy server cause the request didn't reach to policy server. Related KB

 

http://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1227692.html?intcmp=searchresultclick&resultnum=2

 

Hope the issue fix after firewall setting change.

 

Regards,

Kar Meng