monitoring KAFKA, ELS & Kubernetes

[Solomon Melamed]

Hello All
Please all ideas to monitoring KAFKA, ELS, MQ & Kubernetes with IM.

Thanks

[Alex Yasuda]

Solomon,

Maybe this Document can help in something about KAFKA, not sure:
https://knowledge.broadcom.com/external/article?articleId=10696

------------------------------
Technical Support Engineer
Broadcom
------------------------------

[Jonathan GENTILHOMME]

Hi,

This topic is a bit old but it stands out in the top google with the keywords: Broadcom DX IM Kafka, Zookeeper, Elastic ...

So I answer, in order to guide those like me, to the sad fact that this technological package has been worked only on the DX APM application side.

/!\ Information established at 2020/11/19, for DX IM 20.3 and below /!\
On the DX IM side, there are some possibilities, below:

• Elastic: is covered by a specific probe in the "Database Monitoring" category. However, the integration is "basic". In particular, it does not cover recent and modern cluster configurations.
  • Probe: elasticsearch (v1.11 au 2020/11/19)
  • MCS Template: elasticsearch_mcs_template (v1.11 au 2020/11/19)

• MQ: (MCS Compatible) If we are talking about IBM technology, this is also covered by a specific probe. We use it, and it responds satisfactorily. It covers a wide version coverage of IBM technologies, and supports Cluster configuration. A few subtleties have to be taken into account in terms of the host Operating System and the JVM used. Nothing insurmountable!
  • Probe: websphere_mq (v2.22 au 2020/11/19)
  • MCS template: websphere_mq_mcs_template (v2.22 au 2020/11/19)

• Kubernetes: Like Open Shift, Kubernetes is not covered by DX IM 20.3. On the other hand, you have a possible monitoring on the "Docker Engine" side. For Kubernetes, if you are on an On-Premise integration, the "simplest" is to install "Kubernetes Dashboard", and to use the "REST API Monitoring" function of DX IM

• KAFKA, Zookeeper, Apache MLlib .... Not covered. You will find only "workarounds", which will allow you to cover minimal needs thanks to the monitoring of processes, logs, and JVMs. But nothing "packaged".
  • Don't forget that Apache KAFKA provides you with different REST APIs, which can/should be used for Monititoring. I have not found a DX IM "Template" for Kafka's REST API via "Webservice REST Monitoring" ... Need to ask Braodcom to create a public API template repository ?

In the case of my company, we have more and more aplications exploiting these technologies, and we are unable to provide "advanced" monitoring solutions with DX IM. The only apparent solution seems to be a migration to DX SaaS, or at least on the DX APM layer (In order to obtain a coverage equivalent to Datadog, App Dynamics ... which are APM solutions).

Best regards,
Jonathan

------------------------------
Lead Monitoring Solution Architect
AXA Investment Managers Paris
------------------------------