SNMP Traps from Veritas CommandCentral to Spectrum

[Anon Anon]

Hi All

Hope someone can assist with this, I’m pretty sure it’s something basic that I’m overlooking. Unfortunately it’s the first time I’m trying this.

I’m trying to setup alerting in spectrum for Veritas CommandCentral Storage using SNMP traps.

Steps done so far

I’ve confirmed that the traps are being received on the Spectrum server (used Wireshark)
Attempted to discover source server – it only comes back as pingable
Loaded the MIB files from the Veritas software in Spectrum
Mapped the attribute support and Trap Support in the MIB Tools utility
Confirmed the EventDisp and AlertMap are under the $Spectroot\CsVendor\Veritas folder
Added all SNMP community strings I found mentioned in the Wireshark packet dump to Spectrum (thanks to the Tuesday tech tips)

Problem is I’m not seeing any alerts anywhere in the OneClick console. One of the other threads mentioned they should appear under the VNM ‘s alarm \ event tab?

My questions are;

Does the device sending the traps need to be discovered as a SNMP model in Spectrum?
If not where would I check for the incoming traps \ alerts in Spectrum please?

Thank you
Nicholas

[Anon Anon]

I'm still pretty new at this, but let me take a crack:


"Does the device sending the traps need to be discovered as a SNMP model in Spectrum?"

Any SNMP traffic can be sent to the Spectrum server, doesn't have to be for a specific model. As you know, it's UDP...it goes anywhere and everywhere, the tradeoff is that you don't get any verification it made it there. But if there is no model, then just like you've been told, it's supposed to show up under the VNM. Of course, it arrives as raw data (which you've seen, if you used Wireshark). The raw data is supposed to contain some essential information, such as source, destination, SNMP version, community string, and authentication (if you're using v3)...as well as the OIDs, and supporting varbinds. Spectrum's job is to interpret that into useful information. (I realize you know all this, I'm just thinking out loud.)

When you did your Wireshark, what version of SNMP was being presented? It may be useful to you if you can tell your Veritas device to stick to the basics...v1 and a simple "public" for the community string, at least until you've verified it's successfully sending traps. Also, just because you saw SNMP traffic on a subnet doesn't necessarily mean that it was directed at or received by your Spectrum server; you might want to check the settings on your Veritas device and see if it has a provision to specifically send SNMP traffic to your Spectrum server. Also, some people set up a community string for their NMS, check and make sure that matches.

Is there a firewall between your Veritas device and your Spectrum server? We've had issues with firewalls not passing the 161/162 UDP traffic, though it may be nice enough to pass IMCP traffic.


"If not where would I check for the incoming traps \ alerts in Spectrum please?"

If you're not seeing them under your VNM, my suggestion is to get a copy of an SNMP trap generator program, and manually generate the traps, sending them to your Spectrum box.

TRAPGEN.exe v1 -c public -d (your Spectrum IP address) -i (your Veritas box IP address) -o 1.3.6.1.4.1.1302.3.12.10.2 (Veritas top Enterprise OID .1.3.6.1.4.1.1302) -g 6 -s 4 -v (OID Varbind 1) -v (OID Varbind 2) [SCOM 2007 trapgen]

or

SNMPTRAP -v 1 -c public (your Spectrum ip address) .1.3.6.1.4.1.1302.3.12.10.2 (your Veritas Enterprise OID) 6 4 "" (Varbind OID) s "test" [Net-SNMP tools]

Depending on which OID you choose, you should at least get an event under the VNM...if you did everything else right, though, you might get it under your Pingable model. (For which Varbinds to use, right-click on the trap suppord OID name, then click Information.)


BUT...having said all this, chances are that Spectrum is not actually contacting any SNMP service on your Veritas device...usually, if all you can get is a Pingable Model, that's what it means. Are you sure SNMP is running on the Veritas device?

Please forgive me if any of this is hashing over something you covered, I'm trying to approach it from a different angle.




Afterthought: The SystemEDGE agent is capable of generating traps (not entirely useful, because they are "aggregate" traps...but let's not get in to that). The SystemEDGE agent, when it loads, contains a /bin directory, which has some pretty useful SNMP utilities on it. If you can't find a copy of SCOM 2007 or Net-SNMP, the tools in the /bin directory may suffice. If you don't have SystemEDGE on your Veritas device, it may be useful to have another server with the SystemEDGE agent loaded on the same subnet, sending the sample traps I suggested above.

[Anon Anon]

Thank you

Wireshark seems to have all the 161/162 UDP traffic coming through, I filtered the packets in Wireshark to only show things sent from the Veritas device and all looked in order.

I also did a compare between traps sent from Veritas and traps sent from another server - (Called it Server X).

What I noticed is the traps from Server X are being sent to a different port - the same port that the SpectroServer is using whilst the traps from the Veritas device are being sent to the default SNMP port which 162.

Will change that to send to the same port as Server X and see what happens.

"BUT...having said all this, chances are that Spectrum is not actually contacting any SNMP service on your Veritas device...usually, if all you can get is a Pingable Model, that's what it means. Are you sure SNMP is running on the Veritas device?"

Pretty sure now, it wasn't for the first week - I assumed the Veritas Admin had set it up but he had not, so started from scratch.

"Please forgive me if any of this is hashing over something you covered, I'm trying to approach it from a different angle." Nothing to forgive - approaching it from a different angle is what I need.

Thanks for the tip about the SNMP utilities in the Sysedge folder, will give them ago and try your suggestions out.


Thank you both for your answers :)

[Anon Anon]

Hi All

Have managed to get the snmp traps showing in the VNM have the events in and the alerts mapped to the events.

However the events are still not creating any alarms. I have looked at the Alertmap and confirmed the OIDs in there match the traps coming in, and still nothing.

A couple of things I've noticed is there are 2 EventDisp and 2 Alertmaps;

1 under win32app\Spectrum\custom\Events and the other under win32app\Spectrum\SS\CsVendor\Veritas

My understanding is that if there is a custom one the system will use that.

The other thing is the Alertmap didn't manage the incoming traps initially

Incoming trap example in screenshot

Alert map entry - I've tried removing the .6.1 \ .6.2 \ .6.3 and .6.4 so it matches the incoming trap identifier and no luck either.


# ccCritical alertRecipients
1.3.6.1.4.1.1302.3.12.10.2.6.1 0xfff00006 1.3.6.1.4.1.1302.3.12.10.1.1(1,0)\
# alertSummary
1.3.6.1.4.1.1302.3.12.10.1.2(2,0)\
# alertDescription
1.3.6.1.4.1.1302.3.12.10.1.3(3,0)\
# policyName
1.3.6.1.4.1.1302.3.12.10.1.4(4,0)\
# objectType
1.3.6.1.4.1.1302.3.12.10.1.5(5,0)\
# collectorName
1.3.6.1.4.1.1302.3.12.10.1.6(6,0)\
# ccHost
1.3.6.1.4.1.1302.3.12.10.1.7(7,0)\
# sourceId
1.3.6.1.4.1.1302.3.12.10.1.8(8,0)\
# ccObject
1.3.6.1.4.1.1302.3.12.10.1.9(9,0)\
# sampleData
1.3.6.1.4.1.1302.3.12.10.1.10(10,0)\
# ccAlertSeverity
1.3.6.1.4.1.1302.3.12.10.1.11(11,0)\
# ccAlertTime
1.3.6.1.4.1.1302.3.12.10.1.12(12,0)

# ccError alertRecipients
1.3.6.1.4.1.1302.3.12.10.2.6.2 0xfff00007 1.3.6.1.4.1.1302.3.12.10.1.1(1,0)\
# alertSummary
1.3.6.1.4.1.1302.3.12.10.1.2(2,0)\
# alertDescription
1.3.6.1.4.1.1302.3.12.10.1.3(3,0)\
# policyName
1.3.6.1.4.1.1302.3.12.10.1.4(4,0)\
# objectType
1.3.6.1.4.1.1302.3.12.10.1.5(5,0)\
# collectorName
1.3.6.1.4.1.1302.3.12.10.1.6(6,0)\
# ccHost
1.3.6.1.4.1.1302.3.12.10.1.7(7,0)\
# sourceId
1.3.6.1.4.1.1302.3.12.10.1.8(8,0)\
# ccObject
1.3.6.1.4.1.1302.3.12.10.1.9(9,0)\
# sampleData
1.3.6.1.4.1.1302.3.12.10.1.10(10,0)\
# ccAlertSeverity
1.3.6.1.4.1.1302.3.12.10.1.11(11,0)\
# ccAlertTime
1.3.6.1.4.1.1302.3.12.10.1.12(12,0)

# ccWarning alertRecipients
1.3.6.1.4.1.1302.3.12.10.2.6.3 0xfff00008 1.3.6.1.4.1.1302.3.12.10.1.1(1,0)\
# alertSummary
1.3.6.1.4.1.1302.3.12.10.1.2(2,0)\
# alertDescription
1.3.6.1.4.1.1302.3.12.10.1.3(3,0)\
# policyName
1.3.6.1.4.1.1302.3.12.10.1.4(4,0)\
# objectType
1.3.6.1.4.1.1302.3.12.10.1.5(5,0)\
# collectorName
1.3.6.1.4.1.1302.3.12.10.1.6(6,0)\
# ccHost
1.3.6.1.4.1.1302.3.12.10.1.7(7,0)\
# sourceId
1.3.6.1.4.1.1302.3.12.10.1.8(8,0)\
# ccObject
1.3.6.1.4.1.1302.3.12.10.1.9(9,0)\
# sampleData
1.3.6.1.4.1.1302.3.12.10.1.10(10,0)\
# ccAlertSeverity
1.3.6.1.4.1.1302.3.12.10.1.11(11,0)\
# ccAlertTime
1.3.6.1.4.1.1302.3.12.10.1.12(12,0)

# ccInformational alertRecipients
1.3.6.1.4.1.1302.3.12.10.2.6.4 0xfff00009 1.3.6.1.4.1.1302.3.12.10.1.1(1,0)\
# alertSummary
1.3.6.1.4.1.1302.3.12.10.1.2(2,0)\
# alertDescription
1.3.6.1.4.1.1302.3.12.10.1.3(3,0)\
# policyName
1.3.6.1.4.1.1302.3.12.10.1.4(4,0)\
# objectType
1.3.6.1.4.1.1302.3.12.10.1.5(5,0)\
# collectorName
1.3.6.1.4.1.1302.3.12.10.1.6(6,0)\
# ccHost
1.3.6.1.4.1.1302.3.12.10.1.7(7,0)\
# sourceId
1.3.6.1.4.1.1302.3.12.10.1.8(8,0)\
# ccObject
1.3.6.1.4.1.1302.3.12.10.1.9(9,0)\
# sampleData
1.3.6.1.4.1.1302.3.12.10.1.10(10,0)\
# ccAlertSeverity
1.3.6.1.4.1.1302.3.12.10.1.11(11,0)\
# ccAlertTime
1.3.6.1.4.1.1302.3.12.10.1.12(12,0)

Any suggestions will be welcome.

[Anon Anon]

Nick:


Based on the image you posted, the trap is arriving, and the event is being created exactly as the AlertMap entry dictates. (All varbinds being parsed and passed.)

Does the very first address listed (the one you redacted) match the IP address of your Veritas pingable model, in your map? Can you go in to the Locator tab, in the Navigation pane, then after expanding Devices and double-clicking on All Devices, put that IP address in the search field in the Contents pane...and locate your Veritas Pingable Device there? Does it match the address in the event?

Regarding the Alert you want to be getting, here is a document you should read. VERY important that if you want to understand how the alerts are created, that you read it. If you can't hit the link, let me know and I'll post it in entirety, here. (Make sure you are signed to Support Online before trying to hit it.)

https://comm.support.ca.com/?legacyid=TEC444071

It explains a LOT about how and where Alerts get created. If you wade through it, you will get some GREAT information. Keep in mind that you don't have to edit the files it mentions, you can use also the Event Configuration tool. I had to read it many times before I completely understood it.


Last thoughts: I noticed that you put that you had removed the 6.1, 6.2, 6.3 parts, and didn't see that it helped any. May I ask what your thinking was behind removing them?

[Anon Anon]

Hi

Thanks for the link, it turns out it was mapping to the pingable model properly. Ended up creating an Event Admin container and linking them to that, then the alerts started.


My thinking behind removing the 6.3 6.4 .. .parts was this; I read in another post in the forums that it tries to match the OID to the one present in the AlarmMap file. If it can't find the exact match it tries the next closest one. With the OID ID in trap stopping at .2. and all of them having that I removed the .6.3 and .6.4 and then deleted all but the one to see what would happen when it match precisely.

[macad02]

Hello Nicholas,

By default, Spectrum will process every trap that it receives.

The first thing it does when it receives a trap is check the payload (PDU) for the IP address of the agent who sent the trap.
It will then look for a model in the database that has that IP address and will assert the event on that model.

So it doesn't matter if the device is modeled as and SNMP device or a Pingable the trap event should be visible from the events tab on the model.

If the IP address of the agent is not found in the SSdb, then an "unknown SNMP device" event will been asserted on the VNM model (click on the events tab of the VNM )showing you the IP address, TRAP OID and varbind data from the trap.

Adam