How to get data from ElasticSearch in WIly

[Legacy User]

I want to pull data in Wily APM from ElasticSearch.

How does it work.

 

Here is the flow for ElasticSearch:

I have application data being logged to Kafka and then to ElasticSearch. This data is being pulled in a FrontEnd UI.

I have an API as well but how do I get that data to Wily show up in my dashboard ?

 

Please let me know if you need other questions to answered to help on this.

[Matthew Muskaloon]

ElasticSearch is not supported by APM out of the box, nor do we currently have any field packs for it.

 

However, I am including Hiko_Davis and Guenter_Grossberger to see if they have ran into ElasticSearch before and can offer any insights.

[Hallett German]

I posted a note internally requesting a response. No guarantee that will happen. But at least more people are aware of this query. Converting to a Discussion since this is about custom instrumentation.

[spyderjacks]

To get the data into CA APM, you need to "publish" the data.  Typically, this is via the EPAgent.

Since you have an API, what is it?  There are a variety of mechanisms top bring data to the EPAgent for publishing, everything from log file scraping, command scraping, to RESTful with JSON.  Nothing is out-of-the-box, for sure - but there is a path forward.

[Guenter DI Grossberger]

As Mike wrote we don't have an out-of-the-box solution. You can take GitHub - CA-APM/ca-apm-fieldpack-docker: Restful EPAgent script (python) to monitor Docker via Remote API. as a blueprint to get you data from Elastic (instead of Docker) into APM.

[Jarkko_Linna]

In theory - if I understood your question correctly - you could so something like:

Get EPAgent and configure it's REST API:
https://docops.ca.com/ca-apm/10-5/en/implementing-agents/ep-agent/configure-the-epagent-restful-interface

EPA REST API will accept json data. Since elasticsearch outputs json, you're already near. You need to parse/convert the ELK json into EPA REST accepted format (documented in the above URL). You could use nodejs or something else to do the pull/convert/push.

Then you can push the data into APM EPA REST and have it display as APM metrics. Is this what you're looking for?

[Hallett German]

Dear Vijay:

     There are enough leads here to get you started. Marking this thread as closed/inactive. (Although officially this does not exist. You may post follow-up questions as needed

 

Thanks

Hal German

[Hallett German]

I created  Knowledge Doc TEC1662149  with the above answers. The link is http://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec1662149.html 

[LUKE KABAMBA]

We are using kibana (with Logstash and Elastic Search) extensively for our customized dashboards.

Here is what we did:

1- Deployed the easy integration tool provided for APM.

2- Extract APM data in csv format using a cron script executed every minute.

3- Use Logstash csv to input data in Elsticsearch/Kibana

Thanks, 

[Hallett German]

Thanks Luke for sharing in this area of growing interest.