DX Application Performance Management

Expand all | Collapse all

How to get data from ElasticSearch in WIly

  • 1.  How to get data from ElasticSearch in WIly

    Posted 01-19-2017 12:33 PM

    I want to pull data in Wily APM from ElasticSearch.

    How does it work.

     

    Here is the flow for ElasticSearch:

    I have application data being logged to Kafka and then to ElasticSearch. This data is being pulled in a FrontEnd UI.

    I have an API as well but how do I get that data to Wily show up in my dashboard ?

     

    Please let me know if you need other questions to answered to help on this.



  • 2.  Re: How to get data from ElasticSearch in WIly

    Posted 01-19-2017 12:46 PM

    ElasticSearch is not supported by APM out of the box, nor do we currently have any field packs for it.

     

    However, I am including Hiko_Davis and Guenter_Grossberger to see if they have ran into ElasticSearch before and can offer any insights.



  • 3.  Re: How to get data from ElasticSearch in WIly

    Posted 01-19-2017 01:27 PM

    I posted a note internally requesting a response. No guarantee that will happen. But at least more people are aware of this query. Converting to a Discussion since this is about custom instrumentation.



  • 4.  Re: How to get data from ElasticSearch in WIly

    Posted 01-19-2017 01:27 PM

    To get the data into CA APM, you need to "publish" the data.  Typically, this is via the EPAgent.

    Since you have an API, what is it?  There are a variety of mechanisms top bring data to the EPAgent for publishing, everything from log file scraping, command scraping, to RESTful with JSON.  Nothing is out-of-the-box, for sure - but there is a path forward.



  • 5.  Re: How to get data from ElasticSearch in WIly

    Posted 01-19-2017 07:12 PM

    As Mike wrote we don't have an out-of-the-box solution. You can take GitHub - CA-APM/ca-apm-fieldpack-docker: Restful EPAgent script (python) to monitor Docker via Remote API. as a blueprint to get you data from Elastic (instead of Docker) into APM.



  • 6.  Re: How to get data from ElasticSearch in WIly

    Posted 01-20-2017 05:36 AM

    In theory - if I understood your question correctly - you could so something like:

    Get EPAgent and configure it's REST API:
    https://docops.ca.com/ca-apm/10-5/en/implementing-agents/ep-agent/configure-the-epagent-restful-interface

    EPA REST API will accept json data. Since elasticsearch outputs json, you're already near. You need to parse/convert the ELK json into EPA REST accepted format (documented in the above URL). You could use nodejs or something else to do the pull/convert/push.

    Then you can push the data into APM EPA REST and have it display as APM metrics. Is this what you're looking for?



  • 7.  Re: How to get data from ElasticSearch in WIly

    Posted 01-20-2017 07:50 AM

    Dear Vijay:

         There are enough leads here to get you started. Marking this thread as closed/inactive. (Although officially this does not exist. You may post follow-up questions as needed

     

    Thanks

    Hal German



  • 8.  Re: How to get data from ElasticSearch in WIly

    Posted 01-20-2017 11:37 AM

    I created  Knowledge Doc TEC1662149  with the above answers. The link is http://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec1662149.html 



  • 9.  Re: How to get data from ElasticSearch in WIly

    Posted 01-22-2017 02:03 AM

    We are using kibana (with Logstash and Elastic Search) extensively for our customized dashboards.

    Here is what we did:

    1- Deployed the easy integration tool provided for APM.

    2- Extract APM data in csv format using a cron script executed every minute.

    3- Use Logstash csv to input data in Elsticsearch/Kibana

    Thanks, 



  • 10.  Re: How to get data from ElasticSearch in WIly

    Posted 01-23-2017 08:54 AM

    Thanks Luke for sharing in this area of growing interest.