Using Wireshark

[Anon Anon]

[font=Verdana][size=6][size][font]
Hi,

My client is from Banking field. They use Spectrum 9.2. They are facing issues with BGP alarms not getting cleared.

CA Support informed that a trap is required to clear the alarm. Router logs show that the trap was sent but it seems that it didnt reach the Spectroserver.

Hence, CA Support suggested using Wireshark tool to find out whether the trap was generated and why it was not able to reach.

However, due to the security policies, my client is not allowing to use Wireshark and asked if there is any alternate option or whether any other client was asked to Wireshark and they did it and collected the logs.

Bank's one more concern is the size of the wireshark logs which would be iin GBs.

Please suggest what to do.

Thanks,

Sachin
Principal Customer Success Executive

[Anon Anon]

Wireshark can only help you so much though.

If the first event came through there is no network I know of that would block the clear statement for an event. They either all come through or do not in my experience of Cisco kit.

You could of course capture in front of Spectrum and by filtering only on the source IP can reduce the size of it so you can investigate what is there. If all other set and clear events are coming through does not sound a network type of issue there.

could there be a bug in the router though that causes this ?

[Frank Tonjes]

smb76 wrote:

[font=Verdana][size=6][size][font]
Hi,

My client is from Banking field. They use Spectrum 9.2. They are facing issues with BGP alarms not getting cleared.

CA Support informed that a trap is required to clear the alarm. Router logs show that the trap was sent but it seems that it didnt reach the Spectroserver.

Hence, CA Support suggested using Wireshark tool to find out whether the trap was generated and why it was not able to reach.

However, due to the security policies, my client is not allowing to use Wireshark and asked if there is any alternate option or whether any other client was asked to Wireshark and they did it and collected the logs.

Bank's one more concern is the size of the wireshark logs which would be iin GBs.

Please suggest what to do.

Thanks,

Sachin
Principal Customer Success Executive

Hi,

I'm not sure if you have installed Spectrum on Windows or Linux, but if you have used Linux, you can use tcpdump or snoop. Some of these might already be available by the OS build.

Regarding Windows, I'm not to sure how easy this will be, especially since the customer has a problem with Wireshark. Wireshark logs don't have to be huge. It depends on your filter.
If you filter on specific traps - like show me SNMP traps going from x to y, then your logs can be made a lot smaller. The more specific your filters are, the shorter the logs (based on what you are actually sniffing ofc).

Regards,

Frank

[Anon Anon]

franktonjes wrote:

smb76 wrote:

[font=Verdana][size=6][size][font]
Hi,

My client is from Banking field. They use Spectrum 9.2. They are facing issues with BGP alarms not getting cleared.
CA Support informed that a trap is required to clear the alarm. Router logs show that the trap was sent but it seems that it didnt reach the Spectroserver.
Hence, CA Support suggested using Wireshark tool to find out whether the trap was generated and why it was not able to reach.
However, due to the security policies, my client is not allowing to use Wireshark and asked if there is any alternate option or whether any other client was asked to Wireshark and they did it and collected the logs.
Bank's one more concern is the size of the wireshark logs which would be iin GBs.
Please suggest what to do.

Thanks,
Sachin
Principal Customer Success Executive

Hi,

I'm not sure if you have installed Spectrum on Windows or Linux, but if you have used Linux, you can use tcpdump or snoop. Some of these might already be available by the OS build.
Regarding Windows, I'm not to sure how easy this will be, especially since the customer has a problem with Wireshark. Wireshark logs don't have to be huge. It depends on your filter.
If you filter on specific traps - like show me SNMP traps going from x to y, then your logs can be made a lot smaller. The more specific your filters are, the shorter the logs (based on what you are actually sniffing ofc).

Regards,
Frank

Hello Sachin,

If your customer's objection to WireShark extends to all such packet-sniffing tools, then you probably are out of luck,
unless they are willing to grant you a temporary exception for this troubleshooting, and/or allow limited use with explicit capture filters, as Frank suggests.
That is, if your WireShark CAPTURE filter (not your DISPLAY filter) includes UDP 161/162 only (e.g. "host 10.22.33.44 and (udp port 161 or udp port 162)" ),
then that limits the data capture to SNMP polls and traps, which should not contain any sensitive customer information.

Under Linux, packet capturing usually requires root access as well, which probably will be another hurdle for you to jump over.

If your customer's objection is to WireShark in particular (which I doubt), and you are on Windows,
be aware that Windows Server 2003 offers its own counterpart to WireShark, called Microsoft Network Monitor (NetMon, version 2).
NetMon appears under the Administrative Tools menu, but ONLY if you have installed it from your Windows source media, using the Add/Remove Windows Components interface.

Apparently, Windows Server 2008 (and 2008 R2) does not ship with NetMon, but there is a download of NetMon version 3.x available from Microsoft for these platforms.

HTH,
--Mark S

Mark Serencha – Inforonics Global Services, LLC – (m) [phone] – Mark.Serencha_AT_inforonics.com

[Anon Anon]

Thanks Ian, Mark and Frank for replying...

I checked with my client for the suggestions regarding using Filters in Wireshark and NetMon utility in Windows 2008..(Spectrrum installed on Windows 2008).

1. For Wireshark, their Internal Security is not ready for any sniffing tool even if the filters would be applied.

2. For Netmon, Bank's Network team would discuss internally and get back to CA.

What else can we check if the alarms are not clearing? Has anyone seen this issue at any other client site? How BGP is set at other clients?

Please guide or provide some documentation for review.

Thanks,

Sachin
[e-mail]

[Michael Poller]

smb76 wrote:

Thanks Ian, Mark and Frank for replying...

I checked with my client for the suggestions regarding using Filters in Wireshark and NetMon utility in Windows 2008..(Spectrrum installed on Windows 2008).

1. For Wireshark, their Internal Security is not ready for any sniffing tool even if the filters would be applied.

2. For Netmon, Bank's Network team would discuss internally and get back to CA.

What else can we check if the alarms are not clearing? Has anyone seen this issue at any other client site? How BGP is set at other clients?

Please guide or provide some documentation for review.

Thanks,

Sachin
[e-mail]

Hello Sachin,

The earlier poster is correct in that if the trap arrives and generates an alarm, then it is highly unlikely that the clear trap, if it is being sent to Spectrum, is not arriving due to network problems. Unless of course the device is not sending the clear trap at all for some reason related to a configuration problem or bug in the devices software related to traps.

If packet capture/sniffing tools are not an option you can try to debug the trap reception and processing in Spectrum by doing the following:

CLI option for trap alarm processing debug:
This writes logging data to the VNM.out file on the SpectroSERVER
1 - Open a command prompt on the SpectroSERVER and log in to the bash shell with the command:
bash -login
2 - Change directories to the $SPECROOT/vnmsh directory with the command:
cd vnmsh
3 - Connect to the CLI with the command:
./connect
4 - Once connected, run this command to enable the debug:
./update action=0x10245 mh=<VNM_ModelHandle>
(Will need to have the Model Handle attribute value for the VNM model handy)
When done with the debugging, to disable it run the following command:
./update action=0x10246 mh=<VNM modelhandle>

Further Tips and Tricks for this debug:
To turn on debug for a specific IP, 10.10.5.1 for example:
./update action=0x10245 mh=<VNM mh> index=0,attr=1,type=0x13,val=10.10.5.1

That should show processing of the alarm for all models, or that specific model based on IP if that version of debug command was run, in the VNM.out found in the $SPECROOT/SS directory.

If we see data showing processing of that raise and clear trap it means the trap made it the SS and was processed for alarm raise or clear in the OC server.
If still no alarm raise or clear check:
- Alarm processing attributes on the model
- Ensure the model is not in Maintenance Mode

Hope that helps you move this problem forward toward resolution.

Thanks,
Michael

[Anon Anon]

Thanks Michael for your detailed information...

I believe trap alarm processing debug would definitely help my client since they are not ready to use Wireshark etc tools.

I will discuss these steps with the Network team and let you know how it goes.

Regards,

Sachin

[Anon Anon]

Sachin,

BGP alerting is not easy. Mostly, because out of the box, Spectrum is not set up to Alert using most of the BGP traps that come out of routers. In fact, given my (admittedly limited) experience with BGP trapping, I'm surprised that you even get an Event, much less an Alert. So, if your system Alerts, I'm guessing that someone has already done some work within Event Configuration to allow this to happen.

You asked, "How is BGP set at other clients?" For us, we had to set up a Custom alarm within Spectrum. This is because there are several levels of traps available from Cisco routers, and several ways they can show up at the Spectrum server as traps. Here are the levels you can set from Cisco:

snmp-server enable traps bgp [state-changes {[all] backward-trans] [limited]}] | [threshold prefix]

From within this router configuration command, you can select five levels of trapping, from [All] getting every BGP message (and there are a LOT), or you can choose [backward-trans], which will only notify you if you have a backwards transition event (when BGP loses it's connection). We are working with [limited], which is about as close as you can get to having "BGP went down" and "BGP is back up" notifications. (The real problem is that the traps for BGP were not designed with monitoring in mind, and there are so many states between "up" and "down" that it is difficult to get a clear reading on your real BGP status.)

But here's the rub: even if the trapping configuration in your routers is correct, and you are actually getting what passes for "up" and "down" messages, even then you are still not getting clear information. So, if you are troubleshooting, here is step one:


How is the trapping level set up in the router? What level did they set?


Here is the next thing to consider: What are the OIDs that your routers are sending to Spectrum? Obviously, you are already aware of the need to know this, because you have asked how to view incoming traffic without compromising network security. Since your network team appears to be sensitive about network sniffers like Wireshark, then my next suggestion will depend largely on what OS your Spectrum server is running on.

What is the OS that your Spectrum server is running on?

If your Spectrum server is running on Linux, then you'd be able to run TCPDUMP to monitor incoming traffic. The following command will help:

tcpdump -vvvnn -XX -s0 port 162

This will show all SNMP traps and notifications, all you have to do is get your network team to "jiggle" the routers and reproduce the BGP failures and recoveries (and therefore the OIDs) that you're looking for.

If your Spectrum server is running on Windows, then my suggestion is to run a SNMP trap capture utility while on your Spectrum server. There are a few utilities out there which you can use to monitor incoming traps, such as Net-SNMP and SCOM2007. (SCOM has a very nice GUI interface.)


Here is something else to consider: Are you sure that Spectrum is configured to allow the traps to reset your Alarms? Out of the box (OOTB), I found that even though these inbound traps would generate Events, these Events were not configured to reset any Alarms (or generate Alarms).

For instance, supposing your router was set up to send "Limited" traps. "Limited" means "Enables support for notifications for standard backward transition and established events". Typcially, you'd get OIDs like:

bgpEstablishedNotification - 1.3.6.1.2.1.15.0.1

...or...

bgpEstablished - 1.3.6.1.2.1.15.7.6.1


Both of which resolve to Event Code 0x00220010, which technically is a "trap"...BUT...in Event Configuration, for Spectrum OOTB, it is merely an Event...not an Alert or an Alarm. (The OIDs that Cisco has from within their BGP4-MIB don't even resolve to any Events, unless you go to MIB Tools and add them.)

Example: Supposing the "BGP Lost" Alarm you were getting was Event Code 0x00220017 "The BGP Peering session from {S 1} to {S 2} has been lost." Looking through Event Configuration, we can see that the way to clear this Alarm is with Event 0x00220018 "The BGP Peering Session Alarm was cleared."

So the question becomes, "How do I get an Event Code 0x00220018 to be generated?" And the answer is to make sure that the OID you're getting back from your router resolves to an Event (perhaps 0x00220011)...this event can be linked to Event 0x00220018, which can then reset your BGP alarm.


On top of all this, I don't know if you were aware, but there is a way to do a "blind" test of your system...it requires that you make some assumptions (which is never good), but who knows, it may be helpful. Basically, you find some software which allows you to create and send traps to your Spectrum server. (Again, Net-SNMP and SCOM2007 have some tools which enable you to generate and send SNMP v1 traps.) Using this method, you can test your logic and see if the Events correctly correspond to the Alerts that you want to happen. Like I said, it is a "blind" method, because even if you could successfully generate and send a Trap...which would generate the Event you wanted, which would then link to and set off the Alert you desired...

...all you would know for certain is that if Spectrum got this specific trap, your logic would work. You still would not know if your router was sending the same trap. (But at least you'd know your logic worked.)


So to sum all this up, you'll need to know what level of Trapping your network guys have set up, then you'll have to have that TCPDUMP or SNMP capture utility runing, then you'll have to have the network guys jiggle the routers so you can see what OIDs (and varbinds) are arriving, then you have to make sure that the OIDS resolve to the correct Event, then you'll have to make sure that the Event is connected to the correct Alarm.

Simple, right?

I hope this helps.

--Dave


P.S. We also have SPM for Threshold Monitoring of Port 179 set up, because that is the port that BGP uses. If the SPM threshold is exceeded, then we know the BGP link went down. If the threshold is not exceeded, then we know that at least, that tunnel is up.

[Anon Anon]

Thanks Dave for sharing the valuable info...

I am going through your reply and would discuss this with the client.

Regards,

Sachin

[Anon Anon]

Sachin,

One more thing here is as the issue is not specific to any particular device and happens intermittently the trap log may increase incase we wait for the issue to happen after enabling the debug.


kalyan