Symantec Access Management

 View Only
Expand all | Collapse all

Semaphore error

  • 1.  Semaphore error

    Posted Nov 01, 2016 09:55 AM

    Hi All,

     

        we are getting below error on Apache webservers:

     

    [01/Nov/2016:07:27:10] [Error] [CA WebAgent IPC] [18911] [CSmSem::SemWait] Error performing decrement operation on semaphore 543490050 - Identifier removed (43)
    [01/Nov/2016:07:27:10] [Error] [CA WebAgent IPC] [18911] [CSmIpcEvent::WaitForEvent] Error waiting for event semaphore 543490050 - Identifier removed (43)
    [01/Nov/2016:07:27:10] [Error] [CA WebAgent IPC] [18911] [CSmSem::SemWait] Error performing decrement operation on semaphore 543490050 - Identifier removed (43)

     

    Due to this error the disk space is getting full within few hours.

    we restarted the webserver, cleared the Semaphores, even rebooted the physical server.

     

    [18911/-708999424][Tue Nov 01 2016 05:57:16] SiteMinder APACHE 2.2 WebAgent, Version 12.5, Update None, Label 732.
    [18911/-708999424][Tue Nov 01 2016 05:57:16] FileVersion: 12.5.0000.732.

     

    Policy server R12.52 SP1.

     

    We are not seeing these errors coming in other environments but this error is coming in this environment.

     

    Any suggestions to resolve these errors



  • 2.  Re: Semaphore error

    Broadcom Employee
    Posted Nov 01, 2016 03:19 PM

    Hi Richard,

     

    The error message could be attributed by multiple different causes.

    1. Is your web server apache restarted automatically during the same time?

    2. Is your semaphore configuration on the server ever been properly tuned?

    kernel.sem = 250 32000 32 128
    (SEMMSL, SEMMNS, SEMOPM, SEMMNI)
    TEC529712 staes ” to modify SEMMNI and SEMMNS”.
    (1) SEMOPM (maximum operations per semop call)
    it is often recommended to set SEMOPM equal to SEMMSL.
    http://www.puschitz.com/TuningLinuxForOracle.shtml#SettingSemaphores
    The SEMOPM Parameter
    (2) SEMMNS = SEMMSL * SEMMNI
    "The SEMMNS Parameter"
    http://gerardnico.com/wiki/linux/semaphore

    3. Is the apache running in prefork or worker mode?

    4. Is the working apache agent on the same version of OS and same version of apache server, same bit level?

    5. Have seen defect on certain old release of agent too with this problem, it does not hurt to upgrade to R12.52 SP1 agent and see if problem resolves.

     

    Hope this helps.

     

    Hongxu



  • 3.  Re: Semaphore error

    Broadcom Employee
    Posted Nov 02, 2016 04:55 AM

    Hi Richard,

     

    You can also stop the Web Agent and then, make sure
    that there's no orphan semaphore trailing in memory :

     

    https://communities.ca.com/message/241802874

     

    Once stopped, get list of any trailing semaphores in memory
    and remove all of them.

     

    Once the memory's cleaned, start the Web Server and see
    if you still have the error.

     

    Hope that helps,

     

    Best Regards,
    Patrick



  • 4.  Re: Semaphore error

    Posted Nov 02, 2016 09:41 AM

    If issue still persist post semaphore clearing ( as suggested above ) , take a reboot of the OS. It might resolve the issue.

     

    Thanks,

    Ankush



  • 5.  Re: Semaphore error

    Posted Nov 07, 2016 01:24 PM

    Thanks @ankush raj Patrick-Dussault liuho03 for your inputs

     

    we verified all other environments and settings for semaphores are similar on all environments but only this individual environment is having issue.

     

    One more weird issue we are noticing that whenever  we notice below error in Policy server we see those errors in webagents and fill up the disk space within 30 mins or so.

     

    [4552/4544][Mon Nov 07 2016 01:53:09][HSTMT.cpp:227][CHSTMT::Error][ERROR][sm-xpsxps-00800] While calling "SQLConnect(SiteMinder Oracle)", the following error occurred:
    [4552/4544][Mon Nov 07 2016 01:53:09][HSTMT.cpp:241][CHSTMT::Error][ERROR][sm-xpsxps-00810] Native Diagnostic: HYT00:0 [DataDirect][ODBC Oracle Wire Protocol driver]Timeout expired.
    [4552/4544][Mon Nov 07 2016 01:53:09][HSTMT.cpp:241][CHSTMT::Error][ERROR][sm-xpsxps-00810] Native Diagnostic: 08001:-6 [DataDirect][ODBC Oracle Wire Protocol driver][Oracle]Network Operation Timed Out.
    [4552/4544][Mon Nov 07 2016 01:53:09][XPSODBC.cpp:722][CXPSIOODBC::CXPSIOODBC][ERROR][sm-xpsxps-00960] Error obtaining ODBC handle.
    [4552/4544][Mon Nov 07 2016 01:53:09][XPSPolicyData.cpp:1150][CXPSPolicyData::Begin][ERROR][sm-xpsxps-00730] XPS Transaction BEGIN has failed.
    [4552/1112][Mon Nov 07 2016 01:53:45][HSTMT.cpp:227][CHSTMT::Error][ERROR][sm-xpsxps-00800] While calling "SQLConnect(SiteMinder Oracle)", the following error occurred:
    [4552/1112][Mon Nov 07 2016 01:53:45][HSTMT.cpp:241][CHSTMT::Error][ERROR][sm-xpsxps-00810] Native Diagnostic: HYT00:0 [DataDirect][ODBC Oracle Wire Protocol driver]Timeout expired.
    [4552/1112][Mon Nov 07 2016 01:53:45][HSTMT.cpp:241][CHSTMT::Error][ERROR][sm-xpsxps-00810] Native Diagnostic: 08001:-6 [DataDirect][ODBC Oracle Wire Protocol driver][Oracle]Network Operation Timed Out.
    [4552/1112][Mon Nov 07 2016 01:53:45][XPSODBC.cpp:722][CXPSIOODBC::CXPSIOODBC][ERROR][sm-xpsxps-00960] Error obtaining ODBC handle.
    [4552/1112][Mon Nov 07 2016 01:53:45][XPSPolicyData.cpp:1150][CXPSPolicyData::Begin][ERROR][sm-xpsxps-00730] XPS Transaction BEGIN has failed.

     

    which actually doesn't make sense why this could be happening.

    We are trying to get rid of those errors in policy server, hoping that if those errors are gone webagents will work fine. But whenever policy server reports those errors, webagents start to give Semaphore errors, any suggestions why this behavior might be happening?



  • 6.  Re: Semaphore error

    Broadcom Employee
    Posted Nov 08, 2016 03:13 AM

    Hi Richard,

     

    As you mentionned, this is a different issue, and it would be better to
    open a separate communities to handle it to avoid confusion.

     

    But here's a hint about this Policy Server error :

     

        Native Diagnostic: HYT00:0 [DataDirect][ODBC Oracle Wire Protocol driver]Timeout expired.
        [sm-xpsxps-00730] XPS Transaction BEGIN has failed.

     

    This is a connection error to the Policy Store Database. This may be due to :

     

         - Network issue bring the connection to timeout;
         - The DB is too busy to be able to accept the connection before getting a timeout;

     

    You need to investigate the network and Policy Store DB sanity.
    Because this is related to the Policy Store, it might be related to the semaphores problem
    considering that by starting the Web Agent requests the Policy Server to search for trustedhost
    in the Policy Store. If the connection fails due to that, may the Web Agent connection to
    hang and corrupt memory.

     

    Also, you can increase the connection timeout :

     

         https://docops.ca.com/ca-single-sign-on/12-52-sp1/en/administrating/configuring-policy-server-data-store-options/configure-odbc-data-store-options

     

         "Configure ODBC Registry Settings for Timeout"

     

    Please use a separate community thread for other questions.

     

    Best Regards,
    Patrick



  • 7.  Re: Semaphore error

    Posted Nov 08, 2016 09:21 AM

    Patrick-Dussault I agree with you that these are two different issues, as I said whenever policy server reports those Network timeout errors I see those semaphore errors in webagents, I have increased the ODBC timeout since then errors in policy sever and webagents are gone. Will keep track of errors and post what happens.



  • 8.  Re: Semaphore error

    Posted Nov 08, 2016 09:56 AM

    Richard Richard.Leto

     

    What is the Operation System for this Apache Server where we are seeing these Semaphore Errors. If it is RHEL7 then there is a Design change in the RHEL7 OS, which clears Semaphores on User Session Logout from Server. 

     

    CA SSO : RHEL7 and Semaphores 



  • 9.  Re: Semaphore error

    Posted Nov 08, 2016 11:41 AM

    HubertDennis we are running Red Hat Enterprise Linux Server release 6.2 (Santiago)



  • 10.  Re: Semaphore error

    Posted Nov 15, 2016 12:03 PM

    Hello All,

     

       Any other suggestions please what can be done to resolve below error:

     

    [01/Nov/2016:07:27:10] [Error] [CA WebAgent IPC] [18911] [CSmIpcEvent::WaitForEvent] Error waiting for event semaphore 543490050 - Identifier removed (43)
    [01/Nov/2016:07:27:10] [Error] [CA WebAgent IPC] [18911] [CSmSem::SemWait] Error performing decrement operation on semaphore 543490050 - Identifier removed (43)

     

    Its happening randomly on different servers at different times.



  • 11.  Re: Semaphore error

    Posted Nov 15, 2016 05:16 PM

    Hi Richard,

     

    You mentioned that this error is consistently happening in one environment (random server at random time) and not the other.

     

    Please compare the output of "sysctl kernel.sem" (semaphore settings) and "httpd -V" (prefork/worker mode) between the environments.



  • 12.  Re: Semaphore error

    Posted Nov 28, 2016 09:32 AM

    Hello All,

     

        Thanks for your suggestions, we upgraded the webagents to R12.52SP1CR03, and since then we didn't noticed those errors. Assuming the issue is resolved after the upgrade.