Web Agent Processing Clarification:
Hello all, We have been getting some errors in our Production where a user gets internal server error. Firewall is dropping connectins saying 'Out Of State'.. FW idle time out is 30 mins where as Policy server idle timeout is 10 mins. It happens only intermittently.
Policy Server is well under max sockets used, and it does nt show any errors related. I am using Apache 2.4 on RHEL 6.
My strong feeling is , the connection at the firewall is not terminated graciously, web agent is not informed about this termination, and when tries to use the same connection, it might throw errors similar to what I am seeing. But how is it working in subsequent requests.Does this process immediately opens another socket and connection to firewall?
Sometimes , a PID has 20 of these errors continuously with different thread IDs, and eventually it will start working fine. I have TLI logging enabled but I dont see tli logs geneated for many PIDs.
Any ideas on why this would/might happen? What is the best way to troubleshoot? It happens to all function calls such as 'Sm_AgentApi_IsProtectedEx' 'Sm_AgentApi_IsAuthenticated' 'Sm_AgentApi_IsAuthorized ' etc..
Can someone explain back ground of work of how WebAgent/LLAWP interacts with OS to create sockets to communicate with Policy Server using TCP Connection through Firewall? I want to know where are all we can enable some tracing and validation.
Agent trace shows:
[08/31/2015][15:16:04][32342][1444923136][000000000000000000000000f1f9980a-7e56-55e4b604-561fc700-4c065cdb23d0][IsResourceProtected][Communication failure between SiteMinder policy server and web agent.]
[08/31/2015][15:16:04][32342][1444923136][000000000000000000000000f1f9980a-7e56-55e4b604-561fc700-4c065cdb23d0][CSmProtectionManager::DoIsProtected][LowLevelAgent returned SmFailure.]
[08/31/2015][15:16:04][32342][1444923136][000000000000000000000000f1f9980a-7e56-55e4b604-561fc700-4c065cdb23d0][ProcessAdvancedAuthentication][ProtectionManager returned SmNoAction or SmFailure, end new request.]
WebAgentTrace.conf:
data: Date, Time, Pid, Tid, TransactionID, Function, Message
Corresponding Agent Log shows:
[723/1476392704][Mon Aug 31 2015 15:34:55][CSmLowLevelAgent.cpp:492][ERROR] LLA: SiteMinder Agent Api function failed - 'Sm_AgentApi_IsProtectedEx' returned '-1'.
[723/1476392704][Mon Aug 31 2015 15:34:55][CSmProtectionManager.cpp:192][ERROR] HLA: Component reported fatal error: 'Low Level Agent'.
[723/1476392704][Mon Aug 31 2015 15:34:55][CSmHighLevelAgent.cpp:776][ERROR] HLA: Component reported fatal error: 'Session Manager'.
Appreciate your time in advance.