Webservice calls make APM slow

[Anon Anon]

Hi all!

 

Here at my company, we are trying to integrate Introscope with other tools and to do it we are calling some APM WS at each 30s. We noticed that, when it reaches approximately 8 simultaneous calls, APM goes very slow.

 

I think that there should be some tuning procedures to makes the responses faster, because every call took more than 8s to query the last 30s of a specific metric and agent. It's only one value retuned... I suppose the answer might be very quick.

 

We are calling http://mti-introscope.sicredi.net/introscope-web-services/services/MetricsDataService, method getMetricData

 

XML:

<soapenv:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:met="http://metricsdata.webservicesimpl.server.introscope.wily.com">
   <soapenv:Header/>
   <soapenv:Body>
      <met:getMetricData soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
         <agentRegex xsi:type="xsd:string">(.*)\|CEM\|TESS Agent</agentRegex>
         <metricRegex xsi:type="xsd:string">Business Service\|REDE COLABORATIVA\|Business Transactions:Total Bussiness Transactions Per Interval</metricRegex>
         <startTime xsi:type="xsd:dateTime">2014-12-01T13:10:00.000Z</startTime>
         <endTime xsi:type="xsd:dateTime">2014-12-01T13:10:30.000Z</endTime>
         <dataFrequency xsi:type="xsd:int">30</dataFrequency>
      </met:getMetricData>
   </soapenv:Body>
</soapenv:Envelope>

Any ideas?

Thanks!

[Haruhiko Davis]

What version of APM are you using?

[Anon Anon]

Hi Haruhiko!

I'm using APM 9.5.3.0.

Thanks.

[Haruhiko Davis]

Thanks for replying.

 

I would also like to ask:

 

• Is your environment clustered (APM cluster, not high availability)
• What language are you using
• Is it possible for you to share the code with us on the APM-Dev group (haven't signed up/not heard? We're located here CA Developer APM). It may be a better location to discuss this with you in detail and help troubleshoot.

 

Please take a look at the materials posted. I know it may seem like a lot of reading, but it really is not much and should only take a few minutes. Discussions on the Dev subcommunity will be more focused on these types of discussions versus general APM queries/troubleshooting requests.

[Anon Anon]

1) Yes. Our environment is composed of two physical machines (IBM Blade, 23CPU, 145GB RAM). There are 4 collectors in each one plus a collector to CEM, in machine 1.

 

2) Actually I'm using an ETL tool called Kettle. It's a component of Pentaho Suite. It makes a webservice call, transform the data and send it to an Oracle DB.

 

3) As it is not a code I don't know if sending you the job will help. I'll join to this community you suggested and see if I find something useful to my case.

 

Thank you very much!

[Haruhiko Davis]

I will take a look at Kettle and let you know.

 

Since you are running your cluster on two physical servers, I now need to ask about your SmartStor configuration.

 

• For each collector, is the SmartStor on a dedicated disk with its own I/O path? If you did not do this, then be prepared for slowness while querying from the MOM.
• Are you running your queries directed at the MOM and not the collectors?
• Can you estimate how much data or how many rows you're returning at each polling?

[Haruhiko Davis]

There is also a note in the guide about the use of these web services.

 

"Note: Keep in mind that the more generic your regular expressions are, the more metric statistics are returned by the Enterprise Manager. That can cause OutOfMemory errors both on your Enterprise Manager and on your web services client. Such large queries can also cause significant slowdown on the Enterprise Manager. To avoid these situations, Enterprise Manager clamps the maximum number of metrics that are matched for one web service query. The default is 10,000, and it can be configured on the Enterprise Manager."

[Anon Anon]

Maybe if I run only one call to APM Webservice using the method getBulkMetricData, instead of several calls, should be better.

Anyone here already used this method? How can I use array in allMetrics field?

 

<soapenv:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:met="http://metricsdata.webservicesimpl.server.introscope.wily.com" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/">
  <soapenv:Header/>
  <soapenv:Body>
      <met:getBulkMetricData soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
        <allMetrics xsi:type="urn:ArrayOfAgentMetricName" soapenc:arrayType="urn:AgentMetricName[]" xmlns:urn="urn:ca.wily.introscope.webservices"/>
        <startTime xsi:type="xsd:dateTime">?</startTime>
        <endTime xsi:type="xsd:dateTime">?</endTime>
        <dataFrequency xsi:type="xsd:int">?</dataFrequency>
      </met:getBulkMetricData>
  </soapenv:Body>
</soapenv:Envelope>

[Haruhiko Davis]

I did not find getBulkMetricData listed in the API guide for 9.5. Can you upload the WSDL?

[Anon Anon]

Yes, for sure!

Follow file attached.

Thanks!

Attachment(s)

MetricsDataService.xml.zip   1 KB 1 version

[Haruhiko Davis]

Werle

 

I haven't had a chance to startup one of my 9.5 VMs yet, but I do see from the bookshelf that this method is not documented.

Here is what I see from the bookshelf.

 

Metrics Data Web Service WSDL Definition

WSDL URL

http://<host>:<port>/introscope-web-services/services/MetricsDataService?wsdl

Operations

The following operations are used by this web service:

WSDL URL

http://<host>:<port>/introscope-web-services/services/MetricsDataService?wsdl

Operations

The following operations are used by this web service:

Operation: getLiveMetricData

Input

getLiveMetricDataRequest

agentRegex type string

metricPrefix type string

Output

getLiveMetricDataResponse

getLiveMetricDataReturn type array of type DTimeslicedResultSetMetricData

• metricData - nullable; type ArrayOfMetricData - array of type MetricData agentName - nullable; type string

  metricName - nullable; type string

  metricType type int

  metricValue - nullable; type string

• timesliceEndTime - nullable; type dateTime
• timesliceStartTime - nullable; type dateTime

Operation: getMetricData

Input

getMetricDataRequest

agentRegex type string

metricRegex type string

startTime type dateTime

endTime type dateTime

dataFrequency type int

Output

getMetricDataResponse

getMetricDataReturn type array of type TimesliceGroupedMetricData

metricData - nullable; type ArrayOfMetricData - array of type MetricData

• agentName - nullable; type string
• metricName - nullable; type string
• metricType type int
• metricValue - nullable; type string

timesliceEndTime - nullable; type dateTime

timesliceStartTime - nullable; type dateTime

Operation: getTopNMetricData

Input

getTopNMetricDataRequest

agentRegex type string

metricRegex type string

startTime type dateTime

endTime type dateTime

dataFrequency type int

topNCount type int

decreasingOrder type boolean

Output

getTopNMetricDataResponse

getTopNMetricDataReturn type array of type TimesliceGroupedMetricData

• metricData - nullable; type ArrayOfMetricData - array of type MetricData

  agentName - nullable; type string

  metricName - nullable; type string

  metricType type int

  metricValue - nullable; type string

• timesliceEndTime - nullable; type dateTime
• timesliceStartTime - nullable; type dateTime

Operation: getExtendedMetricData

Input

getExtendedMetricDataRequest

• agentRegex type string
• metricRegex type string
• startTime type dateTime
• endTime type dateTime
• dataFrequency type int

Output

getExtendedMetricDataResponse

• getExtendedMetricDataReturn array of type

TimesliceGroupedExtendedMetricData

• extendedMetricData - nullable; array of type
• ExtendedMetricData

Fault

IntroscopeWebServicesException

[Guenter DI Grossberger]

Hi,

 

before we talk about the nitty-gritty details can you answer two questions first:

 

• Why do you need to export metrics from CA APM every 30 seconds? What do you do with them? Why poll every 30 seconds?
• How many metrics are you currently exporting every 30 seconds? How many would you like?

 

Ciao,

Guenter

[Haruhiko Davis]

I also asked the same questions previously.

[Anon Anon]

Hi,

I know that it may sound weird, but we want to plot some metrics in real time (almost it). It is not a often practice, but this is an important task to us. I think that APM Webservice is not ready to this job because we tested the same thing with CLWorkstation client and it did the trick.

 

The webservice calls was changed to:

 

java -Xmx64M -Duser=admin -Dpassword=xxxxx -Dhost=mti-introscope.sicredi.net -Dport=5001 -jar /opt/ETLDashboards/lib/CLWorkstation.jar "get historical data from agents matching ($AGENT) and metrics matching ($METRIC) between \"$NOWL30\" and \"$NOW\" with frequency of $FREQ seconds"

 

Thanks to everyone that helped me!!

[Chris_Kline]

Be aware that the CLW does not have the same safeguards built in that the UI has; large historical queries from the CLW can negatively affect the overhead and performance of the EM.  If you're pulling data with great frequency, you might wish to explore the persistent collections in the EM; there's a few different ways to fork the data directly from SmartStor to a CSV or relational DB.