DX Application Performance Management

Expand all | Collapse all

MetricsDataService SOAP Service returning partial data

  • 1.  MetricsDataService SOAP Service returning partial data

    Posted Sep 05, 2018 05:07 AM

    Hello Community.

    I am trying to gather data from introscope wily using the SOAP CALL to the 8080 MetricsDataService for a specific metric (say GC Heap:Bytes Total)

    For some reason sometimes the introscope server is not responding with all the data form all the agents, but a 10% is missing.

    Can you explain what is the problem and to fix it?

     

    Thank you

    Luca Razzi



  • 2.  Re: MetricsDataService SOAP Service returning partial data

    Broadcom Employee
    Posted Sep 05, 2018 08:35 AM

    Dear Community:

    Can you offer Luca any suggestions?

     

    Dear Luca:

    Are there any patterns on the missing 10% in terms of being the same agents or class of metrics? How do you know they are missing? Converting to a discussion 



  • 3.  Re: MetricsDataService SOAP Service returning partial data

    Posted Sep 06, 2018 01:59 AM

    Hi Hallet.

     The query is like give me back All the GC Heap:Total Bytes for all the agents you have.I use it as a counter for the agents with the agent name to say which agent has gone OOM or down.

    some times i get 1000 back which is truly the total agents.

    but one minute later i get 800 or 700

    and the i get back 1000 most of time.is like some time the query is not gathering all the agents or some collector is not rispondig in timely fashion. 

    I can say the agent not responding seem to be always the same.

    What can be the problem?

    thx

    luca



  • 4.  Re: MetricsDataService SOAP Service returning partial data

    Broadcom Employee
    Posted Sep 06, 2018 02:22 AM

    Hi Luca,

    Are you using the MetricsDataService getLiveMetricData operation?

    Metrics Data Web Service WSDL Definition - CA Application Performance Management - 10.7 - CA Technologies Documentation 

    How frequently are you executing it?

    This is a guess but if most of the time you get all of the agents I am wondering if the temporary loss is due to load rebalancing and agents being moved around Collectors. Default load balancing is 10 minutes so if you are using that value does the frequency of losing agents map to approximately to 10 minutes or multiples of 10 ?

     

    Thanks

     

    Lynn



  • 5.  Re: MetricsDataService SOAP Service returning partial data
    Best Answer

    Broadcom Employee
    Posted Sep 06, 2018 02:32 AM

    The issue could also be due to cluster performance problems i.e. the MOM not getting back all the results. Ultimately the WS call will trigger a query and you can log what the query returns (amount of data not actual result) by setting

     

    log4j.logger.Manager.QueryLog=DEBUG, querylog

     

    in the IntroscopeEneterpriseManager.properties files - you should set it on all EMs

     

    Also I would check the EM logs at the times of the incorrect result for any other tell tail signs of problems like collectors disconnected etc

     

    Hope this helps

    Mike



  • 6.  Re: MetricsDataService SOAP Service returning partial data

    Posted Sep 07, 2018 02:35 AM

    I am using the operation getMetricData because I need to specify start time and end time. But perhaps here is the problem. I must use getLiveMetricData?



  • 7.  Re: MetricsDataService SOAP Service returning partial data

    Broadcom Employee
    Posted Sep 10, 2018 12:21 AM

    Hi Luca,

    Not sure, but if you can compare the results it would be interesting.

    getLiveMetricData will return the last 2 minutes worth of data (8 values for 15 second data interval) from Colector EM memory.

    If you are also already using getMetricData with current timeframe i.e within the last 8 minutes then the data should also be getting retrieved from Collector EM memory rather than SmartStor on disk. 

    If Mike's suggestion on cluster performance is occurring then you may get similar behaviour for both.

    Regards,

    Lynn



  • 8.  Re: MetricsDataService SOAP Service returning partial data

    Posted Sep 11, 2018 07:44 AM

    I am using APM 10.5 and the QueryLog in debug mode did not report any information.

    I am getting the same error using the getLiveMetricsData operation.

    I am still looking for a reason for that.

     

    Let you know if I get something

     

    Luca



  • 9.  Re: MetricsDataService SOAP Service returning partial data

    Posted Sep 12, 2018 04:15 AM

    Hello,

    I am trying to isolate the problem.

    I have set the following to DEBUG in the configuration file:

    Manager.IntroscopeWebServices to DEBUG

    Manager.Cluster to DEBUG

     

    What i get is the following

     

    9/12/18 10:05:02.623 AM CEST [DEBUG] [btpool0-54853] [Manager.IntroscopeWebServices] Agent Regex: (.*) Metric Prefix: GC Heap:Bytes Total

     

    9/12/18 10:05:02.694 AM CEST [DEBUG] [PO:client_main Mailman 1] [Manager.Cluster] Message sent to unregistered mailbox [Server.main:149756 -> Workstation_0.client_main:425708] com.wily.isengard.messageprimitives.MethodCallMessage: {com.wily.introscope.spec.server.beans.metricdata.IChunkedMetricDataNotification.handleData, v1, [[Lcom.wily.introscope.spec.server.beans.metricdata.TimeslicedResultSet;@2db255ec], source: Server.main:149756, remoteHost: Node=Server, Address=ip_here:5001, Type=socket}
    9/12/18 10:05:02.694 AM CEST [DEBUG] [PO:client_main Mailman 1] [Manager.Cluster] Message sent to unregistered mailbox [Server.main:149756 -> Workstation_0.client_main:425708] com.wily.isengard.messageprimitives.MethodCallMessage: {com.wily.introscope.spec.server.beans.metricdata.IChunkedMetricDataNotification.handleData, v1, [[Lcom.wily.introscope.spec.server.beans.metricdata.TimeslicedResultSet;@2db255ec], source: Server.main:149756, remoteHost: Node=Server, Address=/26.2.44.243:5001, Type=socket}
    9/12/18 10:05:02.884 AM CEST [DEBUG] [btpool0-54859] [Manager.IntroscopeWebServices] CAAPM_MAINTAIN_SESSION Value is NULL.
    9/12/18 10:05:02.884 AM CEST [DEBUG] [btpool0-54859] [Manager.IntroscopeWebServices] CAAPM_MAINTAIN_SESSION Value is NULL.
    9/12/18 10:05:03.515 AM CEST [DEBUG] [btpool0-54853] [Manager.IntroscopeWebServices] >>> Data Size <<< 8
    9/12/18 10:05:03.515 AM CEST [DEBUG] [btpool0-54853] [Manager.IntroscopeWebServices] >>> Data Size <<< 8
    9/12/18 10:05:03.533 AM CEST [DEBUG] [btpool0-54853] [Manager.IntroscopeWebServices] Setting Cookie CA-APM=null
    9/12/18 10:05:03.533 AM CEST [DEBUG] [btpool0-54853] [Manager.IntroscopeWebServices] Setting Cookie CA-APM=null
    9/12/18 10:05:03.557 AM CEST [DEBUG] [btpool0-54859] [Manager.IntroscopeWebServices] Setting Cookie CA-APM=null
    9/12/18 10:05:03.557 AM CEST [DEBUG] [btpool0-54859] [Manager.IntroscopeWebServices] Setting Cookie CA-APM=null

     

    9/12/18 10:06:10.449 AM CEST [DEBUG] [PO:client_main Mailman 1] [Manager.Cluster] Message sent to unregistered mailbox [Server.main:149789 -> Workstation_0.client_main:425791] com.wily.isengard.messageprimitives.MethodCallMessage: {com.wily.introscope.spec.server.beans.metricdata.IChunkedMetricDataNotification.queryComplete, v1, [], source: Server.main:149789, remoteHost: Node=Server, Address=ip_here:5001, Type=socket}

     

    Seems that when i get the error I have the MOM sending out this message ChunkedMetricDataNotification.queryComplete



  • 10.  Re: MetricsDataService SOAP Service returning partial data

    Posted Sep 12, 2018 04:33 AM

    Sorry but I have to correct what i said.

    Now I have taken the error also without the message of 

    IChunkedMetricDataNotification.queryComplete.

    Hence this is not related to the error

     

    Luca



  • 11.  Re: MetricsDataService SOAP Service returning partial data

    Posted Sep 14, 2018 03:40 AM

    Hello community,

    Just to give an end to this story I want to tell you that I have reduced a lot the errors of MOM not returning all the agents GC Heap metrics , cutting a lot of historical metrics for all the collectors.

    I have 5 collectors with 1 year retention of historical metrics. Now the historical are around 1 million for each EM while they were around 2 millions.

     

    Anyway the problem still shows up sometimes.



  • 12.  Re: MetricsDataService SOAP Service returning partial data

    Broadcom Employee
    Posted Sep 14, 2018 03:47 AM

    Thanks Luca for updating the status

    It sounds like you have improved the cluster performance per mike_cook 's suggestion.

    So does the problem now occur with less frequency

    Thanks

    Lynn



  • 13.  Re: MetricsDataService SOAP Service returning partial data

    Broadcom Employee
    Posted Nov 20, 2018 04:56 AM

    Good discussion, thank you.



  • 14.  Re: MetricsDataService SOAP Service returning partial data

    Posted Nov 26, 2018 11:50 AM

    Hello community.

    I can tell you that the problem is the SOAP call to the MOM server.

    Calling the single collector (for all the agents) with the WebService API I get everytime the result and also in a very short time (2 seconds)

    Calling the MOM for all the agents every n minutes I get partial data (also in 15-20 seconds) once on 10 times

    Also I debugged that calling the MOM with the SOAP call this call arrives to every have collector and I also found the right answer from every collector.

    The MOM sometimes is not able to rebuild the answer from all the collectors.

     

    Can you tell me what can be the problem?

     

    Thanks

     

    Luca