DX Application Performance Management

 View Only
  • 1.  Outgoing message queue is not moving

    Posted Aug 01, 2019 03:53 PM
    Good afternoon team,

    I would like to check if someone has happened the following:

    • Currently in the MOM log, the following messages are being received:

    8/01/19 02:16:45.088 PM COT [WARN] [Dispatcher 1] [Manager] Outgoing message queue is not moving. Terminating connection: Node=Agent_25, Address=pgn-ms-06.procuraduria.gov.co/172.16.8.195:50129, Type=socket

    8/01/19 02:19:38.683 PM COT [WARN] [Dispatcher 1] [Manager] Outgoing message queue is not moving. Terminating connection: Node=Agent_24, Address=172.16.9.123/172.16.9.123:63251, Type=socket

    8/01/19 02:19:38.687 PM COT [WARN] [Dispatcher 1] [Manager] Outgoing message queue is not moving. Terminating connection: Node=Agent_26, Address=pgn-ms-06.procuraduria.gov.co/172.16.8.195:50151, Type=socket

    8/01/19 02:24:42.063 PM COT [WARN] [pool-14-thread-13] [Manager] Waited 2000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerExtService.clearAllAutoTracingTriggers, v1, [], source: Server.main:535, remoteHost: {Unknown}} from address Server.main:535 to service address Agent_32.main:258 from thread pool-14-thread-13 -- We will keep waiting and don't log further messages until we receive the reply or time out

    8/01/19 02:24:44.063 PM COT [WARN] [pool-14-thread-13] [Manager] Waited 2000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerExtService.clearAllAutoTracingTriggers, v1, [], source: Server.main:535, remoteHost: {Unknown}} from address Server.main:535 to service address Agent_32.main:258 from thread pool-14-thread-13 -- We will not wait any longer

    • While in the collectors, you have

    8/01/19 12:52:48.594 PM COT [WARN] [pool-10-thread-5] [Manager] Waited 2000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerExtService.clearAllAutoTracingTriggers, v1, [], source: Server.main:1996, remoteHost: {Unknown}} from address Server.main:1996 to service address Agent_28.main:258 from thread pool-10-thread-5 -- We will not wait any longer

    8/01/19 01:05:51.649 PM COT [WARN] [pool-10-thread-20] [Manager] Waited 2000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerService.clearAllAutoTracingTriggers, v1, [], source: Server.main:2154, remoteHost: {Unknown}} from address Server.main:2154 to service address Agent_26.main:258 from thread pool-10-thread-20 -- We will keep waiting and don't log further messages until we receive the reply or time out

    8/01/19 01:05:53.650 PM COT [WARN] [pool-10-thread-20] [Manager] Waited 2000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.agent.beans.autotracing.IAutoTracingTriggerService.clearAllAutoTracingTriggers, v1, [], source: Server.main:2154, remoteHost: {Unknown}} from address Server.main:2154 to service address Agent_26.main:258 from thread pool-10-thread-20 -- We will not wait any longer

    8/01/19 01:07:01.470 PM COT [INFO] [PO Route Down Executor] [Manager] Lost connection at: Node=Agent_25, Address=172.16.9.126/172.16.9.126:60383, Type=socket

    The following adjustments were made at the configuration level:

    MOM 

    *  IntroscopeEnterpriseManager.properties

    transport.outgoingMessageQueueSize=8000

    transport.override.isengard.high.concurrency.pool.min.size=14

    transport.override.isengard.high.concurrency.pool.max.size=14

    transport.override.isengard.high.concurrency.pool.queue.size=8000

     
    *  Introscope_Enterprise_Manager.lax

    lax.nl.java.option.additional=-Xms8192m -Xmx8192m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xss512k

     

     

    Colectors

    *   IntroscopeEnterpriseManager.properties


    transport.outgoingMessageQueueSize=8000

    transport.override.isengard.high.concurrency.pool.min.size=14

    transport.override.isengard.high.concurrency.pool.max.size=14

    transport.override.isengard.high.concurrency.pool.queue.size=8000


        *  Introscope_Enterprise_Manager.lax

    lax.nl.java.option.additional=-Xms4196m –Xmx4196m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xss512k



    You only have 15 agents installed;
    When validating in the Webview, the agents are connected but only shows:
    EM_Host
    EM_Port

    Thanks,
    Richard







  • 2.  RE: Outgoing message queue is not moving

    Posted Aug 02, 2019 09:36 AM
    Hello Richard;

    Did you red the agent side logs?

    Regards

    ------------------------------
    Sr. Analyst
    OSDE
    ------------------------------



  • 3.  RE: Outgoing message queue is not moving
    Best Answer

    Broadcom Employee
    Posted Aug 02, 2019 09:48 AM
    Hi Richard,
    Check if the below recent RH Patch OS defect is affecting your environment, so far it has affected mutliple customers and it can affect any APM version.

    Red Hat Patch Update (RHSA-2019:1481) Causes EM Slowness and Disconnects
    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=135344

    If not valid for your case, I suggest you to check the recommendations in the below KB

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=93176

    I hope this helps,
    Sergio


  • 4.  RE: Outgoing message queue is not moving

    Posted Aug 06, 2019 06:44 AM
    Hi
    We have just ran into this same issue.  Working with our Operating System to resolve, will post what we end up doing, the workaround, or backing out the patch.

    Lee​


  • 5.  RE: Outgoing message queue is not moving

    Posted Aug 06, 2019 09:19 AM
    Hi  Team,

    According to Redhat's documentation, I did the following in all MSs, and with this I managed to solve the problem. Considering that the patch effectively affected the network card

    # ethtool -K <interface> tso off gso off

    Redhat: https://access.redhat.com/solutions/4302501

    Richard Briceño


  • 6.  RE: Outgoing message queue is not moving

    Posted Aug 09, 2019 07:03 AM
    Our Unix Admins took the same action and it has resolved it for us too.  Be warned they said it may impact throughput at higher loads - we haven't seen any issues yet but just wanted to mention that.​


  • 7.  RE: Outgoing message queue is not moving

    Posted Aug 06, 2019 11:48 AM
    Change Out going message queue size
    transport.outgoingMessageQueueSize=8000 to 10000

    change heap from 4 GB to 8 GB if you have 16 GB. If not Please add 8 GB more

    lax.nl.java.option.additional=-Xms4196m –Xmx4196m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xss512k

    Xms8192m -Xmx8192m

    run fillowing command in linux

    ulimit -a
    ulimit open files should be 65536. If it is not ask your linux admin to increase it to 6436.
    open files (-n) 65536

    ------------------------------
    Business Solutions Analyst III
    ------------------------------