DX Infrastructure Manager

Expand all | Collapse all

monitoring KAFKA, ELS & Kubernetes

Jump to Best Answer
  • 1.  monitoring KAFKA, ELS & Kubernetes

    Posted 04-16-2020 07:14 AM
    Edited by Solomon Melamed 04-16-2020 07:15 AM
    Hello All
    Please all ideas to monitoring KAFKA, ELS, MQ & Kubernetes with IM.


  • 2.  RE: monitoring KAFKA, ELS & Kubernetes
    Best Answer

    Broadcom Employee
    Posted 04-16-2020 08:10 AM

    Maybe this Document can help in something about KAFKA, not sure:

    Technical Support Engineer

  • 3.  RE: monitoring KAFKA, ELS & Kubernetes

    Posted 7 days ago
    Edited by Jonathan GENTILHOMME 7 days ago

    This topic is a bit old but it stands out in the top google with the keywords: Broadcom DX IM Kafka, Zookeeper, Elastic ...

    So I answer, in order to guide those like me, to the sad fact that this technological package has been worked only on the DX APM application side.

    /!\ Information established at 2020/11/19, for DX IM 20.3 and below /!\
    On the DX IM side, there are some possibilities, below:

    • Elastic: is covered by a specific probe in the "Database Monitoring" category. However, the integration is "basic". In particular, it does not cover recent and modern cluster configurations.
      • Probe: elasticsearch (v1.11 au 2020/11/19)
      • MCS Template: elasticsearch_mcs_template (v1.11 au 2020/11/19)

    • MQ: (MCS Compatible) If we are talking about IBM technology, this is also covered by a specific probe. We use it, and it responds satisfactorily. It covers a wide version coverage of IBM technologies, and supports Cluster configuration. A few subtleties have to be taken into account in terms of the host Operating System and the JVM used. Nothing insurmountable!
      • Probe: websphere_mq (v2.22 au 2020/11/19)
      • MCS template: websphere_mq_mcs_template (v2.22 au 2020/11/19)

    • Kubernetes: Like Open Shift, Kubernetes is not covered by DX IM 20.3. On the other hand, you have a possible monitoring on the "Docker Engine" side. For Kubernetes, if you are on an On-Premise integration, the "simplest" is to install "Kubernetes Dashboard", and to use the "REST API Monitoring" function of DX IM

    • KAFKA, Zookeeper, Apache MLlib .... Not covered. You will find only "workarounds", which will allow you to cover minimal needs thanks to the monitoring of processes, logs, and JVMs. But nothing "packaged".
      • Don't forget that Apache KAFKA provides you with different REST APIs, which can/should be used for Monititoring. I have not found a DX IM "Template" for Kafka's REST API via "Webservice REST Monitoring" ... Need to ask Braodcom to create a public API template repository ?

    In the case of my company, we have more and more aplications exploiting these technologies, and we are unable to provide "advanced" monitoring solutions with DX IM. The only apparent solution seems to be a migration to DX SaaS, or at least on the DX APM layer (In order to obtain a coverage equivalent to Datadog, App Dynamics ... which are APM solutions).

    Best regards,

    Lead Monitoring Solution Architect
    AXA Investment Managers Paris