Symantec Access Management

 View Only
Expand all | Collapse all

WebAgent Losing connection with Policy Server

  • 1.  WebAgent Losing connection with Policy Server

    Posted Jan 07, 2020 08:32 PM
    Hi Team,

    One of our team is running automated scripts to test their application which is getting interrupted multiple times because their webagent is losing connection with policy server. I don't see any errors in the policy server logs during the time frame they mentioned where it is not working. One of the reasons that I could think of is network glitches causing the connectivity unstable. Could you provide any other possible causes for this issue? 

    Please find the below webagent logs for your reference - 

    [12/20/2019][15:41:27][22539][469759744][CSmHighLevelAgent.cpp:395][ProcessRequest][000000000000000000000000c64012ac-580b-5dfd31f7-1bfff700-1bfa65057d54][*172.18.91.84][][qa3_myadm_wa][/cbma/index.html][][ProtectionManager returned SmNo, end new request.]
    [12/20/2019][15:41:27][22539][469759744][CSmLowLevelAgent.cpp:3466][ReportHealthData][][][][][][][Accumulating HealthMonitorCtxt.]
    [12/20/2019][15:41:27][22539][807704320][CSmHighLevelAgent.cpp:204][Shutdown][][][][][][][High Level Agent shutting down.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Resource Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Session Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Response Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Session Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmProtectionManager.cpp:125][CSmProtectionManager::Shutdown][][][][][][][ProtectionManager shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Credential Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Challenge Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Response Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Session Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmAuthenticationManager.cpp:124][CSmAuthenticationManager::Shutdown][][][][][][][AuthenticationManager shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Response Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmManager.cpp:82][Variable Manager][][][][][][][Shutdown.]
    [12/20/2019][15:41:27][22539][807704320][CSmAuthorizationManager.cpp:124][CSmAuthorizationManager::Shutdown][][][][][][][AuthorizationManager shutdown.]

    After we noticed these logs, we asked the team to re-register the webagent  and check if they are facing similar problem. They came back to us stating the same issue -  "their webagent is losing connection with policy server"

    Hence, we couldn't come to a conclusion in this usecase, Kindly please help out with a solution. 




    ------------------------------
    Lavanya
    ------------------------------


  • 2.  RE: WebAgent Losing connection with Policy Server

    Broadcom Employee
    Posted Jan 20, 2020 11:31 AM

    Hi Lavanya,

    Are you still having this problem?

    The High Level Agent (HLA) shutdown messages can be normal depending on web server type and configuration (such as during idle time) and are not the result of the web agent losing connection with the policy server.

    What does the agent trace log show when a request is received?  That log will show the resolved URL, resolved agentname, and the results of the IsProtected call made to the policy server.

    Regards,

    Pete




  • 3.  RE: WebAgent Losing connection with Policy Server

    Posted Jan 21, 2020 02:38 PM
    Hi Pete, 

    Yes, the issue still persists, as the webagent goes down, it doesn't record any trace log, so we don't see any trace log coming up with it. 

     

     



    ------------------------------
    Siteminder Engineer
    ------------------------------



  • 4.  RE: WebAgent Losing connection with Policy Server

    Broadcom Employee
    Posted Jan 22, 2020 02:16 AM
    Hi Lavanya,

    Could you share with us then the Apache error logs, you should see an
    error or what happens when the LLAWP process is removed from memory.

    Best Regards,
    Patrick


  • 5.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 04, 2020 09:21 AM
      |   view attached
    Hi Patrick, 

    Sorry for the late reply. We don't have an offshore team member to capture the error logs at their end, hence it was delayed. Please find the attached error logs and below are smwa.logs for your reference - 

    [28462/3380590336][Tue Feb 04 2020 02:30:49][CSmHttpPlugin.cpp:2176][WARNING][sm-HTTPAgent-00190] Unable to process CBSESSION cookie.
    [28462/3380590336][Tue Feb 04 2020 02:30:50][CSmHttpPlugin.cpp:2176][WARNING][sm-HTTPAgent-00190] Unable to process CBSESSION cookie.
    [28462/3359610624][Tue Feb 04 2020 02:30:50][CSmHttpPlugin.cpp:2176][WARNING][sm-HTTPAgent-00190] Unable to process CBSESSION cookie.
    [28462/3359610624][Tue Feb 04 2020 02:30:50][CSmHttpPlugin.cpp:2176][WARNING][sm-HTTPAgent-00190] Unable to process CBSESSION cookie.
    [28462/3359610624][Tue Feb 04 2020 02:30:50][CSmHttpPlugin.cpp:2176][WARNING][sm-HTTPAgent-00190] Unable to process CBSESSION cookie.
    [28462/3812370176][Tue Feb 04 2020 02:34:17][CSmHighLevelAgent.cpp:206][INFO][sm-AgentFramework-00390] HLA: Stopping.
    [28462/3812370176][Tue Feb 04 2020 02:34:17][SmPlugin.cpp:103][INFO][sm-AgentFramework-00180] Agent Framework plug-in 'SM_WAF_HTTP_PLUGIN' shutdown.
    [28462/3812370176][Tue Feb 04 2020 02:34:18][SmAgentAPI.cpp:1679][INFO][sm-AgentFunc-00040] Agent API has been released.
    [28957/3554739968][Tue Feb 04 2020 07:33:55][SmAgentAPI.cpp:1390][ERROR][sm-AgentConMgr-00020] Failed to enable cluster 0.
    [28957/3554739968][Tue Feb 04 2020 07:33:55][SmAgentAPI.cpp:1404][ERROR][sm-AgentConMgr-00010] Failed to initialize clusters.
    [28957/3554739968][Tue Feb 04 2020 07:33:55][SmAgentAPI.cpp:1484][ERROR][sm-AgentFunc-00010] Agent API initialization failed.

    smwatrace.logs

    [02/04/2020][02:34:02][28462][2898056960][CSmHighLevelAgent.cpp:911][ProcessRequest][000000000000000000000000c64012ac-6f2e-5e391e6a-acbcd700-c1e4215b734e][*172.18.91.84][][qa3_ma_wa][/api/cbma/datainternetgateway/services/datainternetgateway/cbEmail/configuration?profileId=68356][93401101@mailinator.com][End new request.]
    [02/04/2020][02:34:02][28462][2898056960][CSmLowLevelAgent.cpp:3466][ReportHealthData][][][][][][][Accumulating HealthMonitorCtxt.]
    [02/04/2020][02:34:17][28462][3812370176][CSmHighLevelAgent.cpp:204][Shutdown][][][][][][][High Level Agent shutting down.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Resource Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Session Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Response Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Session Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmProtectionManager.cpp:125][CSmProtectionManager::Shutdown][][][][][][][ProtectionManager shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Credential Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Challenge Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Response Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Session Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmAuthenticationManager.cpp:124][CSmAuthenticationManager::Shutdown][][][][][][][AuthenticationManager shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Response Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmManager.cpp:82][Variable Manager][][][][][][][Shutdown.]
    [02/04/2020][02:34:17][28462][3812370176][CSmAuthorizationManager.cpp:124][CSmAuthorizationManager::Shutdown][][][][][][][AuthorizationManager shutdown.]
    [02/04/2020][02:34:17][31171][2239538944][LLAWPMsgBus.cpp:221][ProcessMessage][][][][][][][Close message received from client '28462.-1333971200']
    [02/04/2020][02:34:18][28462][3812370176][SmAgentAPI.cpp:1679][][][][][][][][LogMessage:INFO:[sm-AgentFunc-00040] Agent API has been released.]
    [02/04/2020][02:34:18][28462][3812370176][CSmAgentApiBase.cpp:612][CSmAgentApiBase::Shutdown][][][][][][][AgentApiBase Shutdown.] [02/04/2020][02:40:49][31171][2239538944][LLAWPMsgBus.cpp:181][ProcessMessage][][][][][][][Open message received from client '28957.-1417890048']
    [Date][Time][Pid][Tid][SrcFile][Function][TransactionID][IPAddr][IPPort][AgentName][Resource][User][Message]
    [====][====][===][===][=======][========][=============][======][======][=========][========][====][=======]
    [02/04/2020][02:40:49][28957][2877077248][CSmLowLevelAgent.cpp:5126][ConfigureMP][][][][][][][Tracing initialized.]
    [Date][Time][Pid][Tid][SrcFile][Function][TransactionID][IPAddr][IPPort][AgentName][Resource][User][Message]

    Kindly suggest us the solution to resolve it asap.

    ------------------------------
    Siteminder Engineer
    ------------------------------

    Attachment(s)

    txt
    error_log.txt   901 B 1 version


  • 6.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 05, 2020 07:34 AM
    Hi Lavanya,

    04/Feb/2020:02:34:52] [Error] SiteMinder Agent
    Unable to load SiteMinder host configuration object or host configuration file.
    /cbsiteminder/webagent/config/SmHost.conf
    06 00 00 00
    [04/Feb/2020:02:34:52] [Warning] SiteMinder Agent
    SiteMinder agent has encountered initialization errors and will not service requests.
    [04/Feb/2020:02:34:52] [Information] SiteMinder Agent
    SiteMinder agent is enabled.
    [04/Feb/2020:02:34:52] [Information] SiteMinder Agent
    Configuration file path:
    '/opt/app/httpd/conf/WebAgent.conf'.
    [04/Feb/2020:02:34:52] [Warning] SiteMinder Agent
    Siteminder Web Agent not having write permissions on host configuration file. Shared secret roll-over may not be supported.
    Permission denied. Please assign write privileges to the user apache for the file /cbsiteminder/webagent/config/SmHost.conf


    As per the attached logs, smHost.conf is not able to load, please check if smHost.conf is on correct location and  correct policy server details, moreover please check for the permission web agent on the server it is installed.. 

    Thanks
    Ank


  • 7.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 05, 2020 10:41 AM
    Hi Singh, 

    This is not an issue with configurations. Because the webagent is able to make a connection to policy server with the same configuration (SmHost.conf). Nothing is being changed from the configuration perspective. It is automatically disconnecting/connecting to policy server intermittently. When it happens, it is giving that error message. I have asked them to do a telnet to the policy server when the webagent was going down and telnet works fine. So, it is not network connectivity either.

    We have also checked policy server connections during that time frame if connections are being hosed, but that didn't happen either. We ran the smpolicysrv -stats command on the policy server to make sure that threads and connections are available when this issue happened.

    Is there are any other reason that make a webagent go down automatically?? 



    ------------------------------
    Siteminder Engineer
    ------------------------------



  • 8.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 06, 2020 06:24 AM
    Hi Lavanya,

    This is an issue which we also face at peak load. This issue occur when the policy server crashes due to high load, the issue get resolved post policy server restart. You have to scale up your infrastructure to mitigate the issue. 

    Thanks
    Ank


  • 9.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 07, 2020 11:02 AM
    Edited by Lavanya Ravula Feb 07, 2020 11:02 AM
    Hi Singh,

    As I mentioned in my previous comment, We have also checked policy server connections during that time frame if connections are being hosed, but that didn't happen either. We ran the smpolicysrv -stats command on the policy server to make sure that threads and connections are available when this issue happened.

    Regards, 
    Lavanya .R

    ------------------------------
    Siteminder Engineer
    ------------------------------



  • 10.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 29, 2020 01:13 AM
    ​Do you see any handshake errors in PS smps log for this agent specifically?

    Is it Apache web server, if what is the apache process mode i.e. prefork or worker?


  • 11.  RE: WebAgent Losing connection with Policy Server

    Posted Feb 20, 2020 10:57 AM
    Hi Singh, 

    Any solution for this issue??? 

    Regards,
    Lavanya R

    ------------------------------
    Siteminder Engineer
    ------------------------------



  • 12.  RE: WebAgent Losing connection with Policy Server
    Best Answer

    Posted Feb 21, 2020 03:19 PM
    Hi Lavanya,

    Please check if there is any firewall or other network devices between web and policy server is timing out the connection. Try reducing the idle time-out at policy server and validate.

    Thanks
    Vijay