DX NetOps Manager

Tech Tip: Debugging trap processing in a Spectrum Distributed SpectroSERVER environment with Trap Director enabled 

07-10-2015 11:38 AM

Tech Tip: Debugging trap processing in a Spectrum Distributed SpectroSERVER environment with Trap Director enabled

 

CA Spectrum Tech Tip by: Roger Nason, Support Delivery Manager

 

How to debug trap processing in a Distributed SpectroSERVER environment with Trap Director enabled

 

Description: The following tech tip describes how to monitor a trap from receipt on the Trap Director system to its processing on the destination SpectroSERVER.

 

Notes:

  • It is assumed that a packet capture has already been performed and shows the traps being received at the SpectroSERVER’s defined trap port (default is port 162).
  • All commands are run from the $SPECROOT/vnmsh directory.
  • All debug output is written to the $SPECROOT/SS/VNM.OUT file on the individual SpectroSERVERs

 

Solution:

Trap Director SS

On the SpectroSERVER that is receiving the trap (trap director enabled) enable all alert debugging for traps coming from a specific IP address; send the following CLI commands, substituting the VNM model's handle:

  1. Turn on debug
    1. update action=0x00010291 mh=<model handle of VNM>
      1. Run this to capture trap stats prior to processing
      2. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding (most of these counters can also be seen as model attributes on the VNM model)
    2. update action=0x10245 mh=<VNM mh> index=0,attr=1,type=0x13,val=aaa.bbb.ccc.ddd
        1. aaa.bbb.ccc.ddd = the actual IP address of the device sending the trap
        2. This needs to be running when the trap is received
        3. This will enable alert manager debug for all traps received from the device at the IP address specified in the command

 

Destination SS

On the final destination SpectroSERVER (SS where trap recipient is modeled) send the following CLI commands:

  1. Turn on debug
  2. update action=0x00010291 mh=<model handle of VNM>
    1. Run this to capture trap stats prior to processing
    2. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding (most of these counters can also be seen as model attributes on the VNM model)
  3. update action=0x10245 mh=<VNM mh> index=0,attr=1,type=0x13,val=aaa.bbb.ccc.ddd
    1. This needs to be running when the trap is received
    2. This will enable alert manager debug for all traps received from the device at the IP address specified in the command

 

 

** Now send the trap from a device on a remote landscape that you know is modeled in spectrum**

 

 

Trap Director SS

Once the trap has been sent make sure the debug is disabled to prevent the VNM.OUT file from filling up. In addition we also want to capture the contents of the mux cache and dump the Alert Manager statistics again.

 

  1. To turn off the debug by sending action 0x10246 to the VNM model.
    1. update action=0x10246 mh=<VNM mh>
  2. To capture the contents of the mux cache
    1. update action=0x0001011c mh=<VNM mh>
      1. This should be run after the trap has been received
  3. Dump the Alert Manager Statistics
    1. update action=0x00010291 mh=<model handle of VNM>
      1. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding (most of these counters can also be seen as model attributes on the VNM model)

 

Destination SS

Turn off debugging and dump the Alert Manager statistics again

  1. To turn off the debug by sending action 0x10246 to the VNM model.
    1. update action=0x10246 mh=<VNM mh>
  2. Dump the Alert Manager Statistics
    1. update action=0x00010291 mh=<model handle of VNM>
      1. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding

 

 

 

 

Sample Debug output from Trap Director SS:

 

The following output is intended to facilitate debugging, please forward

to CA technical support. The files listed are not part of the customer

installation of SPECTRUM.

**************************************************************************

Output from Alert Manager Stat dump

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 2

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Output from Alert Manager debug

Jun 18 13:07:26 ALERTMGR TRACE at CsAlertMgr.cc(3201): Alert Manager Debugging Enabled

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertMgMT.cc(1066): Alert manager received a trap:

  source address: 123.4.5.6

  alert code: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertMgMT.cc(1263): queueing trap to be forwarded to remote server

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(829): searching remote cache...

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(847): ...done

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(897): CsAlertFwdMgr::forward_trap_to_remote_servers() - forwarding the trap to 1 remote models.

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(922): forwarding trap to remote model: 0x700069

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(947): trap forwarding done :

  forward list count: 1

  ignore list count : 0

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(1149): forwarding processing complete

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertMgMT.cc(1066): Alert manager received a trap:

  source address: 123.4.5.6

  alert code: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertMgMT.cc(1263): queueing trap to be forwarded to remote server

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(829): searching remote cache...

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(847): ...done

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(897): CsAlertFwdMgr::forward_trap_to_remote_servers() - forwarding the trap to 1 remote models.

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(922): forwarding trap to remote model: 0x700069

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(947): trap forwarding done :

  forward list count: 1

  ignore list count : 0

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(1149): forwarding processing complete

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertMgMT.cc(1066): Alert manager received a trap:

  source address: 123.4.5.6

  alert code: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertMgMT.cc(1263): queueing trap to be forwarded to remote server

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(829): searching remote cache...

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(847): ...done

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(897): CsAlertFwdMgr::forward_trap_to_remote_servers() - forwarding the trap to 1 remote models.

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(922): forwarding trap to remote model: 0x700069

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(947): trap forwarding done :

  forward list count: 1

  ignore list count : 0

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(1149): forwarding processing complete

Jun 18 13:09:40 ALERTMGR TRACE at CsAlertMgr.cc(3325): Alert Manager Debugging Disabled

Output from second Alert Manager stat dump

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 2

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Output from dump of Mux cache

Jun 18 13:10:39 : The following landscapes are participating in trap forwarding:

              Landscape 0x100000

              Landscape 0x400000

              Landscape 0x700000

Jun 18 13:10:39 : The remote trap forwarding cache contains 1 entries.

Cache entry(1):

              model handle = 0x700069

              address = 123.4.5.6

              secure domain =

              traps received = 4

              trap storm size = 20

              trap storm length = 5

              trap storm squelch = 0

              map using IP header = FALSE

              registers for alerts = TRUE

              ignore traps = FALSE

              trapStormHistory is 5 long

              Trap count per bucket is:

              bucket1=0

              bucket2=1

              bucket3=0

              bucket4=1

              bucket5=1

              historyUpdateTime = 1434647350

              lastVisitedTime = 1434647350

              storm started = 0

              cache insert time = Thursday, June 18, 2015 1:03:47 PM

              remote landscape unresponsive = FALSE

 

 

Sample Debug output from Destination SS

SPC-SHD-29021:

**************************************************************************

The following output is intended to facilitate debugging, please forward

to CA technical support. The files listed are not part of the customer

installation of SPECTRUM.

**************************************************************************

Output from Alert Manager stat dump

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 1

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 1

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Output from Alert Manager debug

Jun 18 13:03:40 ALERTMGR TRACE at CsAlertMgr.cc(3201): Alert Manager Debugging Enabled

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(2089): alert manager is processing trap: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(3400): alert is handled by default alert handler

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(3605): did not find any alert entries which would handle the alert, generating unknown alert event

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(2163): failed to handle alert

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(2089): alert manager is processing trap: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(3400): alert is handled by default alert handler

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(3605): did not find any alert entries which would handle the alert, generating unknown alert event

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(2163): failed to handle alert

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(2089): alert manager is processing trap: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(3400): alert is handled by default alert handler

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(3605): did not find any alert entries which would handle the alert, generating unknown alert event

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(2163): failed to handle alert

Jun 18 13:04:48 ALERTMGR TRACE at CsAlertMgr.cc(3325): Alert Manager Debugging Disabled

Output from second Alert Manager stat dump

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 4

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 4

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Statistics
0 Favorited
15 Views
0 Files
0 Shares
0 Downloads

Tags and Keywords

Comments

07-13-2015 09:23 AM

In addition to Rogers details - please have a look to the VNM-Model / Component Details / Subview "Dynamic Debugging" (which is in since R9.2.3H11 and higher).

Here find some notes and please be careful when enable the Dynamic Debug options. Consider - the enabled Event or Alert debugging for ALL activities can affect the SpectroSERVER performance:

 

- Alert Manager with option Disable|Enable  (which is a global enable like doing a ./update action=0x10245  -- NOTE: this causes all "alerts being debugged"

  So - in this case the "CLI/command line" gives you much more options.

 

- Alert  Manager Statistics Dump with option "Dump now" (which is the same like ./update action=0x10291 will do - dumping the data to ./SS/VNM.OUT

 

If you would like to check for "numbers" only - i.e. in case of initial tests - then find the VNM-Model / Component Details -> Subview "Trap Management".

Here you can find the "numbers" of inbound TRAPs - and also here you can trigger the "Remote Forwarding Cache" dump.

 

Best practice while testing is here:

- flush / clear the Remote Forwarding Cache

- re-do the inbound TRAP test

- verify if the "print/dumped" Remote Forwarding Cache entry for the IP-address in question is created at the "receiving SpectroSERVER" instance.

Related Entries and Links

No Related Resource entered.