DX Application Performance Management

  • 1.  Collectors disconnecting and connecting

    Posted Mar 28, 2018 10:09 PM

    I have a problem with the collectors. It is disconnecting and connecting.
    I have a MOM and two collectors.
    Does not show anything in the logs.
    Only problem I found was the ping time too high, but I did all tests MTR, network, TRACEROUTE and this one with very low ping..
    I already saw all the tests and installed the fixes but it did not work.
    Can someone help me?

     



  • 2.  Re: Collectors disconnecting and connecting
    Best Answer

    Broadcom Employee
    Posted Mar 28, 2018 10:43 PM

    The Ping value we show from MOM to each collector is not an ICMP ping, but rather something proprietary we're doing on the Isengard protocol.

    We except this ping to be less than 3.5 seconds.

     

    If you are finding your network latency is a problem, then please speak with your network engineers.



  • 3.  Re: Collectors disconnecting and connecting

    Posted Mar 28, 2018 10:48 PM

    I talked to the network engineer, and he told me that the problem is not in the network.

    Do you have a test to show it to me?
    Which protocol do I use?



  • 4.  Re: Collectors disconnecting and connecting

    Broadcom Employee
    Posted Mar 28, 2018 10:57 PM

    No, there is no test (that I'm aware of) and no, Isengard is a proprietary protocol.

    Please open a support case. They will likely want your perflog.txt from all of your EMs in your cluster.



  • 5.  Re: Collectors disconnecting and connecting

    Posted Mar 28, 2018 11:22 PM

    I already have an open the case and nobody can solve it yet



  • 6.  Re: Collectors disconnecting and connecting

    Broadcom Employee


  • 7.  Re: Collectors disconnecting and connecting

    Posted Apr 02, 2018 10:47 AM

    HI

    I did this procedure and it didn't solve



  • 8.  Re: Collectors disconnecting and connecting

    Posted Apr 10, 2018 01:13 PM

    I have this error message too.

     

    What could I do to improve it?

     

     

    /10/18 12:00:29.539 PM BRT [WARN] [Collector Colletor1@5001] [Manager.Cluster] Waited 15000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.console.IConsoleService.getCurrentHarvestTime, v1, [], source: Workstation_21.client_main:263, remoteHost: {Unknown}} from address Workstation_21.client_main:263 to service address Server.main:273 from thread Collector Colletor1@5001 -- We will keep waiting and don't log further messages until we receive the reply or time out
    4/10/18 12:00:31.281 PM BRT [WARN] [Collector Colletor2@5001] [Manager.Cluster] Got the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.console.IConsoleService.ping, v1, [], source: Workstation_286.client_main:263, remoteHost: {Unknown}} to address Workstation_286.client_main:263 service address Server.main:273 from thread Collector Colletor2@5001
    4/10/18 12:00:31.395 PM BRT [WARN] [Collector Colletor1@5001] [Manager.Cluster] Got the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.console.IConsoleService.getCurrentHarvestTime, v1, [], source: Workstation_21.client_main:263, remoteHost: {Unknown}} to address Workstation_21.client_main:263 service address Server.main:273 from thread Collector Colletor1@5001



  • 9.  Re: Collectors disconnecting and connecting

    Posted Apr 11, 2018 07:48 AM

    Hi Liuz,

     

    In the example directory of the enterprise manager, there are two management modules: MOM_Infra_Monitoring_MM.jar and collector.jar.  Create a copy of the collector.jar for each of your collectors.  Deploy the management modules to your MOM.  Customize the agent management module expression for each module.  Add your collector management modules alerts and dashboard links to the MOM's dashboards.  Open the dashboard "MOM Infrastructure Overview"

    Now, with everything set up, you can start to understand what metrics are in the APM to help address your issue.

     

    Ping could be three factors, network, destination system, source system.  Now, with the APM ping, as Hiko mentioned it isn't your typical ping.  The MOM will connect to the collector's message queues, post a message and wait for a response, so if the MOM's queues/processes are too busy to handle the response or if the collector's queues/process are too busy then you will get an increased ping time.

     

    Now with the MOM Infrastructure Overview, there there a branch that has "Performance", it is a good place to start.

     

    On the Collector's - Collector Performance dashboard, there is harvest duration, smartstor duration, number of event inserts, query performance and CPU & Memory.

     

    Try to correlate the ping (Dashboard - MOM Cluster Ping Time) to the enterprise managers and see if there is anything that might help determine the root cause of the problem.

     

    Hope this helps,

     

    Billy