CA PM and Kafka integration

[Bruno Sousa]

Hello. I have to integrate CA PM with Kafka streaming. Any advice or best practice that you could share? Thank you.

[Michael Poller]

Hi Bruno,

 

I can confirm from a product support perspective, there is no integration available out of the box for that product.

 

Maybe someone out in the user community has tackled this question in the past?

 

Other than that, I'd suggest:

 

A: Submitting an Idea request for Product Management to consider adding an integration with that product as an out of the box option.

 

B: Engaging your account management team to see if the Services team might be able to assist with a solution.

 

Thanks,

Michael

[Rajani Bayapati]

hi Michael,

we have similar requirement to send DX PM metric data to Kafka Data stream. I have seen the DX PM documentation that it can be achieved from 22.2.5 above .

Can we install DX Netops Kafka software in Aggregator server to capture the metrics? can you through some insights with respect to different metrics that we can capture in Kafka through DX PM

[Rajani Bayapati]

Configure Streaming Metric Export to a Kafka Cluster

Last Updated October 31, 2023 by Broadcom Support Team in Documents

Configure the data aggregator to send, or export, collected performance metrics to Apache Kafka.

The following diagram shows streaming metric export to an Apache Kafka cluster:

The data collectors poll devices for desired metrics and forward these to the data aggregator. The data aggregator determines which devices the metric publisher should export metrics using device collections. The export profile defines which metrics the metric publisher in the data aggregator exports for those devices.

The metric publisher taps into the polled data that the data collectors send to the data aggregator in parallel with the normal polled data loading into the data repository. If the metric publisher encounters polled data that matches the configured export profile, it exports those performance metrics to the Kafka data bus for consumption by one or metric consumers.

The metric publisher exports only non-null raw (as-polled) metrics.

Use the following process to configure streaming metric export:

• Verify the Prerequisites
• Configure the Kafka Producer Config File
• Define the Devices and Metrics to Export 

  If you are setting up metric export for the OI Connector to use, the OI Connector already automatically defines the metrics to export, and you can skip this step.

Verify the Prerequisites

Before you can configure streaming metric export, verify the following items:

• The data aggregator is installed.
• Kafka is installed on a server or cluster of servers that is accessible to the data aggregator, and the data aggregator has permission to access the broker ports on the Kafka servers (by default, port 9092).

  If you do not already have a Kafka deployment, you can use NetOps Kafka, which is provided as a separate download.

  For more information about how to install NetOps Kafka, see Install NetOps Kafka.
• You have configured 

  NetOps Portal

   to collect performance metrics for a given set of devices. You have created a monitoring profile, and applied the monitoring profile polling behavior to device collections.

  For more information, see Manage Monitoring Profiles.

Configure the Kafka Producer Config File

For more information about Kafka configuration information, see the "Kafka Configuration Information" section.

The properties in the Kafka producer config file are prepended with 

producer.

. Limit these properties to what is needed for SSL/TLS authentication.

For more information, see the Kafka documentation.

Follow these steps:

• Edit the 

  <installation_directory>

  /apache-karaf/etc/kafkaexport.producer.cfg

   file.

  Example:

  /opt/IMDataAggregator/apache-karaf/etc/kafkaexport.producer.cfg
  
• installation_directory

  The installation directory of the data aggregator.

  Default:

   

  /opt/IMDataAggregator
• Set the following properties in the file:
• feature.enabled

  Set to 

  feature.enabled=on

  .
• producer.bootstrap.servers

  Displays a list of host/port pairs to use for establishing the initial connection to the Kafka cluster. Change the following:
• Change the 

  changeme

   text to the hostname of a Kafka broker.
• Change 

  9092

   to the broker port.

  For Kafka clusters, you can provide a comma-separated list of 

  broker:port

   values.
• Default:

   

  producer.bootstrap.servers=changeme:9092

  Example:

   

  producer.bootstrap.servers=kafkaservername:9092
• (If you have secured Kafka) Provide a keystore/truststore that includes a trusted certificate that can be used to access Kafka by adding the following properties to the file:

  Prerequisite:

   The truststore and keystore are in place.

  
  producer.security.protocol=SSL
  producer.ssl.keystore.location=
  <path_to_keystore>
  
  producer.ssl.keystore.password=
  <keystore_password>
  
  producer.ssl.key.password=
  <kafka_broker_key_password>
  
  producer.ssl.truststore.location=
  <path_to_truststore>
  
  producer.ssl.truststore.password=
  <truststore_password>
  
  
  

  Example:

  
  producer.security.protocol=SSL
  producer.ssl.keystore.location=/opt/IMDataAggregator/apache-karaf/etc/kafka01.keystore.jks
  producer.ssl.keystore.password=password
  producer.ssl.key.password=password
  producer.ssl.truststore.location=/opt/IMDataAggregator/apache-karaf/etc/kafka.truststore.jks
  producer.ssl.truststore.password=password
  
  
• path_to_keystore

  The absolute path to the keystore file.
• keystore_password

  The password for the keystore file.
• kafka_broker_key_password

  The password for the Kafka broker's key in the keystore. You can make this password different than the keystore password if you choose.
• path_to_truststore

  The location of the truststore file.
• truststore_password

  The password to access the truststore.
• Save your changes.

The Kafka producer config file is configured.

Define the Devices and Metrics to Export

The data aggregator determines which devices the metric publisher should export metrics using device collections. The export profile defines which metrics the metric publisher exports for those devices.

• If you are setting up metric export for the OI Connector to use, the OI Connector already automatically defines the metrics that the metric publisher should export, and you can skip the following procedures.
• The following procedures send cURL commands using a username and prompt for a password. You can send credentials in the cURL commands using the alternate methods.

  For more information about these methods, see 

  NetOps Portal

   REST Web Services.

Use the following process to define the devices and metrics that the metric publisher should export:

• Create the Export Profile
• (If required) Update the Export Profile
• Identify the Tenants that Require Metric Export
• Identify the Device Collections that Require Metric Export
• Associate the Device Collections with the Export Profile

Create the Export Profile

Create only one export profile. If multiple export profiles exist, the data aggregator honors the profile with the lowest item ID, and ignores the remaining export profiles.

Follow these steps:

• Issue the following cURL, with the metric families (and filtered metrics if desired) to be exported:

  
  curl --user <NetOpsPortalUser> --location --request POST 'http://<DA_hostname>:8581/rest/exportprofiles' \
  --header 'Content-Type: application/xml' \
  --data '    <ExportProfile version="1.1.0">
  <!-- The list of metric families whose metrics are to be exported. If any metrics from a metric family are to be exported, the metric family must be listed here. -->        
  <ExportedMetricFamilyList>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedReachabilityInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedCPUInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedPortInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedMemoryInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedAvailabilityInfo</ExportedMetricFamily>
  </ExportedMetricFamilyList>
  <!-- This list filters the specific metrics to be exported if only a subset of metrics should be exported for a given metric family. If any metrics are specified here, then only those metrics are to be exported for the associated metric family. If no metrics for a metric family listed in the ExportedMetricFamily attribute are listed here then all metrics for that metric family are to be exported. -->
  <ExportedMetricsList>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.Availability</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.UtilizationIn</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.UtilizationOut</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsIn</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsPerSecondIn</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsOut</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsPerSecondOut</ExportedMetrics>
  </ExportedMetricsList>
          <Item version="1.0.0">
              <Name>
  <Export Profile Name>
  </Name>
              <Description>Export Profile.</Description>
          </Item>
    </ExportProfile>'
  
  

  Example:

  
  curl --user JaneDoe --location --request POST 'http://myDAHost:8581/rest/exportprofiles' \
  --header 'Content-Type: application/xml' \
  --data '    <ExportProfile version="1.1.0">
  <!-- The list of metric families whose metrics are to be exported. If any metrics from a metric family are to be exported, the metric family must be listed here. -->        
  <ExportedMetricFamilyList>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedReachabilityInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedCPUInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedPortInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedMemoryInfo</ExportedMetricFamily>
  <ExportedMetricFamily>{http://im.ca.com/normalizer}NormalizedAvailabilityInfo</ExportedMetricFamily>
  </ExportedMetricFamilyList>
  <!-- This list filters the specific metrics to be exported if only a subset of metrics should be exported for a given metric family. If any metrics are specified here, then only those metrics are to be exported for the associated metric family. If no metrics for a metric family listed in the ExportedMetricFamily attribute are listed here then all metrics for that metric family are to be exported. -->
  <ExportedMetricsList>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.Availability</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.UtilizationIn</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.UtilizationOut</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsIn</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsPerSecondIn</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsOut</ExportedMetrics>
  <ExportedMetrics>{http://im.ca.com/normalizer}NormalizedPortInfo.BitsPerSecondOut</ExportedMetrics>
  </ExportedMetricsList>
          <Item version="1.0.0">
              <Name>Export Profile Name</Name>
              <Description>Export Profile.</Description>
          </Item>
    </ExportProfile>'
  
  
• NetOpsPortalUser

  The 

  NetOps Portal

   username.
• DA_hostname

  The IP address for the data aggregator.

  Example:

   

  myDAHost
• Export Profile Name

  The name of the export profile.
• When prompted, provide a password.
• Note the item ID of the export profile in the response to the request.

  Sample output:

  
  <success>{Export Profile Item ID}</success>
  
  

The export profile is created.

Update the Export Profile

If required, you can modify the values in the existing export profile.

Follow these steps:

• Issue the following cURL, with the data aggregator server hostname, the 

  NetOps Portal

   username, the export profile ID that needs to be updated and any settings that need to be modified (example below modifies the profile name to 

  New Export Profile Name

  ):

  
  curl --user 
  <NetOpsPortalUser>
   --location --request PUT 'http://
  <DA_hostname>
  :8581/rest/exportprofiles/<ExportProfileID>' \
  --header 'Content-Type: application/xml' \
  --data ' <ExportProfile version="1.1.0"> <Item version="1.0.0"> <Name>
  <New Export Profile Name>
  </Name> </Item> </ExportProfile>'
  
  

  Example:

  
  curl --user JaneDoe --location --request PUT 'http://myDAHost:8581/rest/exportprofiles/<ExportProfileID>' \
  --header 'Content-Type: application/xml' \
  --data ' <ExportProfile version="1.1.0"> <Item version="1.0.0"> <Name>New Export Profile Name></Name> </Item> </ExportProfile>'
  
  
• NetOpsPortalUser

  The 

  NetOps Portal

   username.
• DA_hostname

  The IP address for the data aggregator.

  Example:

   

  myDAHost
• New Export Profile Name

  The updated name for the export profile.
• When prompted, provide a password.

The export profile is updated.

Identify the Tenants that Require Metric Export

Follow these steps:

• List the details of the available tenants in the data aggregator by issuing the following cURL command:

  
  curl --user <NetOpsPortalUser> --request GET 'http://<DA_hostname>:8581/rest/tenants' \
  --header 'Accept: application/xml'
  
  

  Example:

  
  curl --user JaneDoe --request GET 'http://myDAHost:8581/rest/tenants' \
  --header 'Accept: application/xml'
  
  
• NetOpsPortalUser

  The 

  NetOps Portal

   username.
• DA_hostname

  The IP address for the data aggregator.

  Example:

   

  myDAHost
• When prompted, provide a password.
• Locate the tenants that you want to configure metric export by name, and then note the item IDs (the 

  <ID>

  ) for each tenant.

Identify the Device Collections that Require Metric Export

Follow these steps:

• List the details of the available device collections in the data aggregator by issuing the following cURL command:

  
  curl --user <NetOpsPortalUser> --request GET 'http://<DA_hostname>:8581/rest/monitoredgroups' \
  --header 'Accept: application/xml'
  
  

  Example:

  
  curl --user JaneDoe --request GET 'http://myDAHost:8581/rest/monitoredgroups' \
  --header 'Accept: application/xml'
  
  
• NetOpsPortalUser

  The 

  NetOps Portal

   username.
• DA_hostname

  The IP address for the data aggregator.

  Example:

   

  myDAHost
• When prompted, provide a password.
• Locate the device collections that you want to configure metric export by name, and then note item IDs (the 

  <ID>

  ) for each device collection.

Associate the Device Collections with the Export Profile

Associate the device collections in the data aggregator with the export profile on a per-tenant basis.

Complete this procedure for each tenant/collection pair that you want to associate with the export profile.

Follow these steps:

• Issue the following cURL command:

  
  curl --user 
  <NetOpsPortalUser>
   --location --request PUT 'http://
  <DA_hostname>
  :8581/rest/tenant/
  <TenantItemID>
  /exportprofiles/
  <ExportProfileItemId>
  /relatesto/groups/
  <CollectionItemId>
  ' \
  --header 'Accept: application/xml' \
  --header 'Content-Type: application/xml' \
  --data ''
  
  

  Example:

  
  curl --user JaneDoe --location --request PUT 'http://myDAHost:8581/rest/tenant/132614/exportprofiles/1234567/relatesto/groups/56789' \
  --header 'Accept: application/xml' \
  --header 'Content-Type: application/xml' \
  --data ''
  
  
• NetOpsPortalUser

  The 

  NetOps Portal

   username.
• DA_hostname

  The IP address for the data aggregator.

  Example:

   

  myDAHost
• TenantItemID

  The item ID for the tenant.

  Example:

   

  132614
• ExportProfileItemID

  The item ID for the export profile with which you want to associate the device collection.

  Example:

   

  1234567
• CollectionItemID

  The item ID for the device collection that you want to associate with the export profile.

  Example:

   

  56789
• When prompted, provide a password.

The tenant/collection pair is associated with the export profile. The item ID for the export profile is recorded.

If you have not yet issued the cURL command for each tenant/collection pair that you want to associate with the export profile, repeat this procedure for the other tenant/collection pairs.

Kafka Configuration Information

The metric publisher exports metrics to the 

metric-export

 topic and heartbeat messages to the 

metric-heartbeat

 topic. It exports these metrics and messages using the default log retention, partition count, and replication settings that are configured in the Kafka broker.

Kafka export to a Kafka cluster requires that the 

metric-export

 and 

metric-heartbeat

 topics exist, and expects that Kafka will create them when the metric publisher exports metrics and messages to them if they do not exist. If the Kafka administrator has configured the Kafka broker to disallow automatic creation of topics when producers (such as the metric publisher) export metrics and messages to them, then the Kafka administrator must create these topics.

You can change the default log retention, partition count, and replication settings to suit your specific requirements using the Kafka utilities on the Kafka broker.

For more information, see the Kafka documentation.

You can modify the 

metric-export

 topic name in the metric publisher by editing the name in the 

topic

 property in the 

kafkaexport.producer.cfg

 file.

Long-term retention past an hour is not required for the 

metric-heartbeat

 topic.