Problems setting "Alert Name" in the SNMP Connector

[Chris Fitch]

Hi Everyone,

Please bear with me, but I'm very new to writing Policy code in XML.
I'm trying to parse the alert content in the SNMP policy in order to enrich the alert, but am struggling to set the Alert Name.

 

Use case:
varbind-1.3.6.1.4.1.791.4.4.27 contains the detail I want to use e.g. "bank-account-pro" (see below trap extract), but I am unsure which internal SOI field I need to set:

 

<property name="varbind-1.3.6.1.4.1.791.4.4.27" value="bank-account-pro" />

 

I have tried setting various fields under Item and also Alert but none seem to make any difference and the alert gets displayed in the Console with a Name of "Unknown".

 

I suspect something else may also be wrong with the Policy as 5 or 6 different alerts have come through to the SNMP_Connector.log (varbind-1.3.6.1.4.1.791.4.4.9), but I only see the one "Unknown" alert in the Console.

 

I am now stuck and would appreciate any guidance that can be provided.

 

I have attached my SNMP Policy and included the varbinds I'm pulling out below - many thanks in anticipation of any help provided.

 

varbinds:

<property name="varbind-1.3.6.1.4.1.791.4.4.27" value="bank-account-pro" />
<property name="varbind-1.3.6.1.4.1.791.4.4.9" value="MicroserviceCount|bank-account-pro|config-service:active" />
<property name="varbind-1.3.6.1.4.1.791.4.4.24" value="bank-account-pro config-service micro-service Total Active" />
<property name="varbind-1.3.6.1.4.1.791.4.4.11" value="http://srvapmepvlsk00.wt.dcuk01.corp:8080/#console;db=bank-account-pro+Micro-services;dn=SuperDomain%2FProd+PaaS+Bank+Account;mm=bank-account-pro;tr=0" />
<property name="varbind-1.3.6.1.4.1.791.4.4.16" value="3" />

Attachment(s)

snmp_policy.xml.zip   1K 1 version

[Chris Fitch]

Quick update ...the additional alerts did eventually come through, it was just a delay :-)

But they have all come through with a Name of "Unknown".

[Brahmaiah Addepalli]

Hello Chris,

 

If this is IFW based SNMP Connector, please enable DEBUG traces by updating \SOI\resources\log4j.xml for below components

Note: For Catalyst based SNMP Connector, this file is located at \\CA\Catalyst\CatalystConnector\ifw\resources folder

 

FOR RAW ENTITY DEBUG FILES

FOR IFW EMBEDDED EI ENTITY DEBUG FILES

FOR PUB ENTITY DEBUG FILES

FOR REQUEST DEBUG FILES

SET TO DEBUG TO SEE PUBLISHED MESSAGES

 

Generate few traps and track the “AlertedMdrElementID” values in \SOI\log\DebugData\alert_raw, item_raw, alert_pub & CI_pub files, this would give us hints and may adjust the connector policy accordingly.

 

Regards

Brahma

[Chris Fitch]

Hi Brahma,

 

Sadly we're still on the Catalyst version of the SNMP Connector :-(

 

So from your post I assume I should be focussing on the AlertedMdrElementID? - I can see from my test alerts that the value of this field is correct in the alert details section, i.e. "bank-account-pro", but in the Alert itself it is "Unknown"...

 

 

 

Could it be anything to do with the alerts first being generated in this Connector before we had a valid snmp_policy in place and therefore creating malformed CIs? Causing any subsequent alert to be stuck with the same dodgy name - I vaguely remember something about this and think it needed the Connector to be  stopped and removed from the SOI Administration Console, so that the CIs were flushed out and then enabling the Connector again in its configuration file? - I've done this in the past for the APM Connector and wondered if the same process was possible in the Catalyst SNMP one. 

 

Regards

Chris

[Chris Fitch]

I've just tried removing the Connector and then restarting the Service, but sadly it made no difference.

 

Connector Name:                            SNMP Generic Connector

Connector version:                          4.2.00

catalyst version:                              3.4.1.125

Build Number:                                 2.0.0.52

Supported Product versions:          TRAPv1, TRAPv2c, INFORMv2c, TRAPv3, INFORMv3

[Michael Boehm]

Hi Chris,

 

there are two things:

To change the Label of a CI, you have to modify the "Item" class, adding a Format2 block (Label cannot be set directly in the standard Format).

There is no way to define or modify the Label in the Alert class, because an Alert has no knowledge about the CI details it is attached to.  An Alert only knows the ID of that CI (see below).

 

An Alert gets the name Unknown (of type Entity) if the CI it is supposed to be attached to does not exist.

Every CI gets an MDR-triple during the creation process (MdrProduct, MdrProdInstance, MdrElementId).

An Alert gets the information to which CI it is attached to via the Alert-MDR-triple (AlertedMdrProduct, AlertedMdrProdInstance, AlertedMdrElementId).  This triple must match any of the previously created CIs.

 

You have to make sure that the CIs get a triple that the Alert also gets, otherwise you will end up in the situation of getting "Unknown Entity" Alerts.

 

Let me know if you need more help.

 

MichaelBoehm

[Michael Boehm]

I looked at your policy.

The CIs don't get created because you are missing all the mandatory properties.

For RunningSoftware you have to specify a ProductName.

And at least one of the following properties has to be set:

DeviceSysName, DeviceDnsName, DeviceBiosSystemID, DeviceAssetNumber, DevicePhysSerialNumber, DeviceIPV6AddressWithDomain, DeviceIPV4AddressWithDomain, DeviceIPV6Address, DeviceIPV4Address, DeviceMacAddress.

 

For these granular objects (RunningSoftware is a child running on a device) I would also set the InstanceName to a value you know (similar to the Label via Format2), because otherwise the GlobalPolicy will set it based on other information and you cannot make sure the value is what you want it to be.

 

MichaelBoehm

[Chris Fitch]

Thanks very much for your advice Michael. I will have another go at it, armed with this information.

[Chris Fitch]

I've tried to incorporate the changes you've recommended Michael, but now I'm not getting any alerts published. I've been staring at the code for hours and can't see where I've gone wrong. Would you be able to take a look please?

 

<Catalog version="1.0" globalextends="GLOBAL!" >

<!-- 28/02/2019 CF - Initial classification for APM alerting -->
<EventClass name="Item" >
<Classify>
<Field input="varbind-1.3.6.1.4.1.791.4.4.27" pattern=".*-pro$" output="eventtype" outval="RunningSoftware_Micro" />
</Classify>

<Format>
<Field output="MdrProduct" format="CA:00036" input="" />
</Format>
</EventClass>

<EventClass name="RunningSoftware_Micro" extends="Item">
<Parse>
<!-- Strip everything after ":active" as we don't want this for the MdrElementID -->
<Field input="varbind-1.3.6.1.4.1.791.4.4.9" pattern="^(.*?):active(.*)" output="ElementID_1,ElementID_2" />
</Parse>

<Format>
<Field output="ClassName" format="RunningSoftware" input="" />
<Field output="MdrProdInstance" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.27" />
<Field output="DeviceDNSName" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.27" />
<Field output="MdrElementID" format="{0}" input="{replace([ElementID_1,|,_])}" />
</Format>

<Write>
<Field type="file" name="outfile" properties="*" />
<Field type="publishcache" properties="*" />
</Write>
</EventClass>

<EventClass name="Alert" >
<Classify>
<Field input="varbind-1.3.6.1.4.1.791.4.4.27" pattern=".*-pro$" output="eventtype" outval="Alert_Micro" />
</Classify>

<Normalize>
<Field output="Severity" type="map" input="varbind-1.3.6.1.4.1.791.4.4.16">
<mapentry mapin="0" mapout="Normal" />
<mapentry mapin="1" mapout="Normal" />
<mapentry mapin="2" mapout="Minor" />
<mapentry mapin="3" mapout="Critical" />
<mapentry mapin=".*" mapout="Minor" />
</Field>
</Normalize>

<Format>
<!-- These are fixed, default parameters that generally should be set and then overridden if needed -->
<!-- Field output="MdrProduct" format="CA:00036" input="" /-->
<Field output="AlertType" format="Risk-Fault" input="" />
<Field output="AlertedMdrProduct" format="CA:00036" input="" />
<Field output="OccurrenceTimestamp" format="{0}" input='{xsdateTime(now)}' />
<Field output="ReportTimestamp" format="{0}" input='{xsdateTime(now)}' />
</Format>
</EventClass>

<EventClass name="Alert_Micro" extends="Alert">
<Parse>
<!-- Strip everything after ":active" as we don't want this for the MdrElementID -->
<Field input="varbind-1.3.6.1.4.1.791.4.4.9" pattern="^(.*?):active(.*)" output="ElementID_1,ElementID_2" />
</Parse>

<Format>
<Field output="Summary" format="{0} violated threshold" input="{replace([varbind-1.3.6.1.4.1.791.4.4.9,|, ])}" />
<Field output="Message" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.4" />
<Field output="AlertedMdrProdInstance" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.9" />
<Field output="AlertedMdrElementID" format="{0}" input="{replace([ElementID_1,|,_])}" />
<!-- Field output="MdrElementID" format="{0}" input="{replace([ElementID_1,|,_])}" /-->
<!-- Build the Generic helpfile URL for ALM -->
<Field output="userAttribute3" format="http://alerthub.unix.santanderuk.corp/help/wily/APM/{0}" input="{replace([varbind-1.3.6.1.4.1.791.4.4.24, ,%20])}" />
<Field output="userAttribute5" format="APM_SNMP_Alert_RunningSoftware" input="" />
<Field output="userAttribute6" format="Polled" input="" />
<Field output="userAttribute9" format="APM (Introscope)" input="" />
<Field output="userAttribute2" format="{0}" input="{replace([varbind-1.3.6.1.4.1.791.4.4.11, ,%20])}" />
<Field output="Message" format="&lt;b&gt;Hostname: &lt;/b&gt;{0} &lt;b&gt;&lt;br/&gt;Problem: &lt;/b&gt;{1} &lt;br/&gt;{2} &lt;br/&gt;&lt;br/&gt;&#60;a href=&#34;{3}&#34;&#62;&#60;b&#62;&#60;u&#62;Click here for Ops instructions&#60;/b&#62;&#60;/u&#62;&#60;/a&#62; &lt;br/&gt;&lt;br/&gt;&#60;a href=&#34;{4}&#34;&#62;&#60;b&#62;&#60;u&#62;APM Console&#60;/b&#62;&#60;/u&#62;&#60;/a&#62;&lt;br/&gt;" input="AlertedMdrElementID,userAttribute9,Message,userAttribute3,userAttribute2" />
</Format>

<Write>
<Field type="file" name="outfile" properties="*" />
<Field type="publishcache" properties="*" />
</Write>
</EventClass>

 

<EventClass name="USM-Entity" >

</EventClass>

</Catalog>

[Michael Boehm]

Hi Chris,

again, there are some missing attributes, now for the Alert (you had them in the previous policy).

I updated the policy (but did not test it) - I hope it works:

 

<Catalog version="1.0" globalextends="GLOBAL!" >

<!-- 28/02/2019 CF - Initial classification for APM MicroService alerting -->
<EventClass name="Item" >
<Classify>
<Field input="varbind-1.3.6.1.4.1.791.4.4.27" pattern=".*-pro$" output="eventtype" outval="RunningSoftware_Micro" />
</Classify>
<Format>
<Field output="MdrProduct" format="CA:00036" input="" />
<Field output='MdrProdInstance' format='{0}' input='{fqdn(localhost)}' />
</Format>
<Write>
<Field type="file" name="outfile" properties="*" />
<Field type="publishcache" properties="*" />
</Write>
</EventClass>

<EventClass name="RunningSoftware_Micro" extends="Item">
<Format>
<Field output="ClassName" format="RunningSoftware" input="" />
<Field output="MdrElementID" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.27" />
<Field output="ProductName" format="APM-MicroService" input="" />
<Field output="DeviceSysName" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.27" />
<Field output="temp_Label" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.27" />
</Format>
</EventClass>

<EventClass name="Alert" >
<Classify>
<Field input="varbind-1.3.6.1.4.1.791.4.4.27" pattern=".*-pro$" output="eventtype" outval="Alert_Micro" />
</Classify>
<Normalize>
<Field output="Severity" type="map" input="varbind-1.3.6.1.4.1.791.4.4.16">
<mapentry mapin="0" mapout="Normal" />
<mapentry mapin="1" mapout="Normal" />
<mapentry mapin="2" mapout="Minor" />
<mapentry mapin="3" mapout="Minor" />
<mapentry mapin="4" mapout="Major" />
<mapentry mapin="5" mapout="Critical" />
<mapentry mapin=".*" mapout="Minor" />
</Field>
</Normalize>
<Format>
<Field output='ClassName' format='Alert' input='' />
<Field output="AlertType" format="Risk-Fault" input="" />
<Field output="MdrProduct" format="CA:00036" input="" />
<Field output='MdrProdInstance' format='{0}' input='{fqdn(localhost)}' />
<Field output="AlertedMdrProduct" format="CA:00036" input="" />
<Field output='AlertedMdrProdInstance' format='{0}' input='{fqdn(localhost)}' />
<Field output="OccurrenceTimestamp" format="{0}" input='{xsdateTime(now)}'/>
<Field output='ReportTimestamp' format='{0}' input='{xsdateTime(now)}' />
</Format>
<Write>
<Field type="file" name="outfile" properties="*" />
<Field type="publishcache" properties="*" />
</Write>
</EventClass>

<EventClass name="Alert_Micro" extends="Alert">
<Parse>
<Field input="varbind-1.3.6.1.4.1.791.4.4.9" pattern="^(.*?)\|(.*?)\|(.*?):.*$" output="ElementID_1,ElementID_2,ElementID_3" />
</Parse>
<Format>
<Field output="MdrElementID" format="Alert_{0}-{1}-{2}" input="ElementID_1,ElementID_2,ElementID_3" />
<Field output="AlertedMdrElementID" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.27" />
<Field output="Summary" format="{0} violated threshold" input="varbind-1.3.6.1.4.1.791.4.4.24" />
<Field output="Message" format="{0}" input="varbind-1.3.6.1.4.1.791.4.4.4" />
<!-- Build the helpfile URL -->
<Field output="userAttribute2" format="{0}" input="{replace([varbind-1.3.6.1.4.1.791.4.4.11, ,%20])}" />
<Field output="userAttribute3" format="http://alerthub.unix.santanderuk.corp/help/wily/APM/{0}" input="{replace([varbind-1.3.6.1.4.1.791.4.4.24, ,%20])}" />
<Field output="userAttribute5" format="APM_SNMP_Alert_RunningSoftware" input="" />
<Field output="userAttribute6" format="Polled" input="" />
<Field output="userAttribute9" format="APM (Introscope)" input="" />
<Field output="Message" format="&lt;b&gt;Hostname: &lt;/b&gt;{0} &lt;b&gt;&lt;br/&gt;Problem: &lt;/b&gt;{1} &lt;br/&gt;{2} &lt;br/&gt;&lt;br/&gt;&#60;a href=&#34;{3}&#34;&#62;&#60;b&#62;&#60;u&#62;Click here for Ops instructions&#60;/b&#62;&#60;/u&#62;&#60;/a&#62; &lt;br/&gt;&lt;br/&gt;&#60;a href=&#34;{4}&#34;&#62;&#60;b&#62;&#60;u&#62;APM Console&#60;/b&#62;&#60;/u&#62;&#60;/a&#62;&lt;br/&gt;" input="AlertedMdrElementID,userAttribute9,Message,userAttribute3,userAttribute2" />
</Format>
</EventClass>

<EventClass name="USM-Entity" >
<Format2>
<Field conditional='temp_Label' output='Label' format='{0}' input='temp_Label' />
</Format2>
</EventClass>

</Catalog>

 

MichaelBoehm

[Chris Fitch]

Hi Michael,

That's done the job nicely, thank you very much! :-)

One question I have, is that I've noticed you've set AlertedMdrElementID and MdrElementID to different values, for example:

 

AlertedMdrElementID = bank-account-pro
MdrElementID = Alert_MicroserviceCount-bank-account-pro-fileservice-current

 

Is this deliberate? As from how I interpreted your earlier post i.e. the Alert-MDR-triple must match the CI MDR-triple. Or have I misunderstood that?

 

Regards
Chris

[Michael Boehm]

Hi Chris,

CIs which are created have a triple to identify them:  MdrElementId, MdrProduct, MdrProdInstance.

 

Alerts have two triples:

The CI to which the Alert is attached is specified with the Alerted.... triple: AlertedMdrElementId, AlertedMdrProduct, AlertedMdrProdInstance.

But every Alert is also a CI in the DB, e.g. it also has a standard triple: MdrElementId, MdrProduct, MdrProdInstance.

This triple must be unique in the DB as well, e.g. not the same as any CI triple.

Thus, for Alerts I normally specify an MdrElementID with "Alert_......".

 

The MdrElementId of an Alert is taken as unique identifier to decide, if you get an update to an Alert or if it is a separate Alert.  All Alerts with the same MdrElementId are taken as updates.

 

In your case I used CI triples based on the hostname, to only create one CI per host.

But I assume there are different situations for every host that should not overwrite each other but stay on the console as individual Alerts.  Thus, I used a different MdrElementId  for the Alerts, consisting of the three parts MicroserviceCount|bank-account-pro|config-service.

 

If you want different scenarios, you can adopt the MdrElementID of the CI or Alert accordingly.

 

MichaelBoehm 

[Chris Fitch]

Thank you very much for that in-depth explanation Michael - that makes things much clearer for me now. This will give me a good base for going forward.

 

Do you know if there is a reference guide that explains which attributes can be used and when?

[Michael Boehm]

Hi Chris,

 

general information about how to develop connector policy is here:

Creating Connector Policy - CA Service Operations Insight - 4.2 - CA Technologies Documentation 

 

The attributes that can be used for every individual CI class can best be seen in the USM Webview when going into "create CI".   The Schema documentation (available in the SOI Debug page) is also useful.

 

And I shared a document here on the community which attributes are mandatory for every class:

CI Generation: Mandatory and Correlation Properties 

 

I hope that helps.

MichaelBoehm