DX NetOps

Expand all | Collapse all

Alarms not displaying in OneClick alarm tab view...

  • 1.  Alarms not displaying in OneClick alarm tab view...

    Posted Oct 24, 2012 04:02 PM
    Hello,

    As the subject line states, I do not get alarms to show in the alarm tab in OneClick. I can view all of the events in the events tab and see the ones which trigger alarms. The icons for devices which have current alarms are red, orange or what not but do not display in alarm tab view.

    The servers that Spectrum is installed on has had Active Directory GPO applied which is DOD Information Assurance/Security related. These systems have been locked down (over 240 individual tweeks applied) and I have opened up a case tthrough support.

    I thought I would post up on the forums and see if anyone has any advice.

    Thank you, Ken


  • 2.  RE: Alarms not displaying in OneClick alarm tab view...

    Broadcom Employee
    Posted Oct 25, 2012 08:40 AM
    Hi Ken,
    Are there any alarm filters being pushed down to your user?
    Regards,
    Bill


  • 3.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 25, 2012 09:51 AM

    wolwi01 wrote:

    Hi Ken,
    Are there any alarm filters being pushed down to your user?
    Regards,
    Bill
    I don't have any filters applied to the alarms.... and I have tried restarting tomcat.. I had a CA Architect here last week and he was scratching his head over this one also.. so we opened up a case... I am supposed to be in contact with support today.

    I'll post up our findings,,, I hope.

    jlbastos wrote:


    Beyond checking the filters, if you can check this in other OC.
    We have a problem here where some OC became out-of-sync in some areas. (alarms, topology, color of elements, etc)
    If you can, make a restart on tomcat and check it again.
    I tried restarting tomcat...

    Thank you for the reply, Ken


  • 4.  RE: Alarms not displaying in OneClick alarm tab view...

    Broadcom Employee
    Posted Oct 25, 2012 04:15 PM
    Hi Ken,
    Have you tried this:

    Open a bash shell (bash -login)
    cd vnmsh
    type: connect
    type: show alarms

    Do the alarms show up in the show alarms output?
    Thank you,
    Bill


  • 5.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 25, 2012 09:04 PM

    wolwi01 wrote:

    Hi Ken,
    Have you tried this:

    Open a bash shell (bash -login)
    cd vnmsh
    type: connect
    type: show alarms

    Do the alarms show up in the show alarms output?
    Thank you,
    Bill
    No I sure haven't but I will first thing in the morning when I'm back in the lab....

    Thank you for the suggestion, I'll post up my findings, Ken


  • 6.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 26, 2012 07:26 AM

    KenJefferson wrote:

    wolwi01 wrote:

    Hi Ken,
    Have you tried this:

    Open a bash shell (bash -login)
    cd vnmsh
    type: connect
    type: show alarms

    Do the alarms show up in the show alarms output?
    Thank you,
    Bill
    No I sure haven't but I will first thing in the morning when I'm back in the lab....

    Thank you for the suggestion, I'll post up my findings, Ken
    I ran the commands and found plenty of alarms... It appears that my device icons are representing their alarm state accurately and I can see all events including the one triggering alarms.. I just have to figure out how to get them to display again..

    Thanks, Ken


  • 7.  RE: Alarms not displaying in OneClick alarm tab view...

    Broadcom Employee
    Posted Oct 26, 2012 07:52 AM
    Hi Ken,
    What version of Spectrum are you running? Assuming it is 9.2.1+, do you know if port 14001 is blocked? We have seen issues in which the alarms will not show in the alarm panel if port 14001 is blocked.
    Hope that helps
    Jason


  • 8.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 26, 2012 11:02 AM
    Apologies for hijacking this thread but its in the same ball park ..

    I have a local SS and can see my alarms fine but the MLS does not show the alarms only the devices in an alarm state, the spectrum landscape in OC says



    hostname
    0x8500000
    Connected to primary server.
    Never. This could indicate network problems, or that this SpectroSERVER is unable to resolve the hostname of this OneClick Server.
    Not Ready
    mscmom01 (0xc00000)
    Connection established.
    9.2.1.006
    9.2.1.006

    I have the hostname in the hosts file, and the .hostrc - port 14001 is established

    So it says its never had a "heartbeat"

    Have tried bouncing Tomcat on both sides


  • 9.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 26, 2012 12:39 PM

    droodle wrote:

    Apologies for hijacking this thread but its in the same ball park ..

    I have a local SS and can see my alarms fine but the MLS does not show the alarms only the devices in an alarm state, the spectrum landscape in OC says



    hostname
    0x8500000
    Connected to primary server.
    Never. This could indicate network problems, or that this SpectroSERVER is unable to resolve the hostname of this OneClick Server.
    Not Ready
    mscmom01 (0xc00000)
    Connection established.
    9.2.1.006
    9.2.1.006

    I have the hostname in the hosts file, and the .hostrc - port 14001 is established

    So it says its never had a "heartbeat"

    Have tried bouncing Tomcat on both sides
    Drood,, actually would be better if you could move that post to a new thread,,, since it is different.

    I would like to keep the info here on the specific topic.... If folks start chimming in on your issue, mine may be overlooked.

    Thanks, Ken


  • 10.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 25, 2012 09:22 AM
    Beyond checking the filters, if you can check this in other OC.
    We have a problem here where some OC became out-of-sync in some areas. (alarms, topology, color of elements, etc)
    If you can, make a restart on tomcat and check it again.


  • 11.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 26, 2012 01:32 PM
    I dont know if this would help you or you have all ready tried it...

    1) open OneClick console
    2) View-Preferences->Alarms Tab
    3) Click on Reset and then on Aply

    You, should be able to vie the alarms generated on the alarm tab.


  • 12.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Oct 26, 2012 03:23 PM

    shila wrote:

    I dont know if this would help you or you have all ready tried it...

    1) open OneClick console
    2) View-Preferences->Alarms Tab
    3) Click on Reset and then on Aply

    You, should be able to vie the alarms generated on the alarm tab.
    Hey Shila.... thank you for the suggestion but it had no effect on contents of my alarms tab in OneClick....

    Thank you for trying to hook me up!


  • 13.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted May 14, 2013 09:55 AM
    Was this ever resolved? We are having the same issue now on our test environment which is currently running 9.2.1 H06.
    We have checked and don't see any connections on 14001, but everything is running on a single host and we have disabled the local firewall.


  • 14.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted May 14, 2013 10:43 PM
    Ken,

    If alarms are not updating in OneClick, here are some general things to check

    1.
    HF10 includes fixes in this area might be good to first move to that release.
    2.
    Check stdout.log / catalina.out for exceptions
    3. Identify models that are generating an excessive amount of events / alarms. These models can then be put into maintenance mode.
    4.
    Check OneClick Administration page > Administration > Debugging > Alarm Table to see if the alarms exists on OneClick
    5.
    Check OneClick Administration page > Administration > Debugging > Alarm Statistics to see if there is an overabundance of alarms (thousands?) If so, alarm processing may be happening but just slow. If this is the case customer will have to work on cleaning up alarms. Running the following mysql commands might help identify models that are generating an excessive amount of events / alarms. These models can then be put into maintenance mode.


    If alarm counts are excessive, get the most commonly generated events and models which are top offenders:
    a.
    Navigate to $SPECROOT/mysql/bin from a bash prompt and run the following command to log into mysql: ./mysql –uroot –proot ddmdb or for Linux / Solaris: ./mysql --defaults-file=<path to my-spectrum.cnf > -uroot -proot ddmdb
    b.
    SELECT HEX(type), COUNT(*) AS cnt FROM event GROUP BY type ORDER BY cnt DESC LIMIT 10;
    c.
    Now, get the top ten models with the most events: Navigate to $SPECROOT/mysql/bin from a bash prompt and run the following command to log into mysql: ./mysql –uroot –proot ddmdb or for Linux / Solaris: ./mysql --defaults-file=<path to my-spectrum.cnf > -uroot -proot ddmdb
    d.
    SELECT HEX(e.model_h), m.model_name, COUNT(*) AS cnt FROM event e, model m WHERE e.model_h=m.model_h GROUP BY e.model_h ORDER BY cnt DESC LIMIT 10;
    e.
    Address the cause of the top events-ex: customer works with IT to fix problematic devices, places models in maintenance in the meanwhile.
    6. On the landscapes page in OC -> Adminsitration -> check if the hear beat is updated for the server having issue. I believe port 14001 is established on the OC server.


    kalyan


  • 15.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted May 16, 2013 06:48 AM
    Hi Kalyan,

    Thanks for the information. We prefer to work the issue on the current version and environment while we have the opportunity to do this on our lab environment. We could try a reinstall or upgrade, but we would like to figure out and document the underlying problem in case we run into similar issues at some point on our production environment.


    We do see one set of warnings in catalina.out about a failure to bind on port 8443 after 10 tries (all within 1 second) when we restart the OC server:
    May 14, 2013 02:46:11 PM (HttpListener-8443-1) (org.apache.axis2.transport.http.server.DefaultConnectionListenerFailureHandler) - (WARN) - Attempt number 10 of 10 to reestalish connection listener org.apache.axis2.transport.http.server.DefaultConnectionListener@c0890f due to failure
    java.net.BindException: Address already in use
            at java.net.PlainSocketImpl.socketBind(Native Method)
            at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
            at java.net.ServerSocket.bind(ServerSocket.java:319)
            at java.net.ServerSocket.<init>(ServerSocket.java:185)
            at java.net.ServerSocket.<init>(ServerSocket.java:97)
            at org.apache.axis2.transport.http.server.DefaultConnectionListener.run(DefaultConnectionListener.java:75)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:619)
    May 14, 2013 02:46:11 PM (HttpListener-8443-1) (org.apache.axis2.transport.http.server.DefaultConnectionListener) - (INFO) - Listening on port 8443
    May 14, 2013 02:46:11 PM (HttpListener-8443-1) (org.apache.axis2.transport.http.server.DefaultConnectionListenerFailureHandler) - (ERROR) - Terminating connection listener org.apache.axis2.transport.http.server.DefaultConnectionListener@c0890f after 10retries in 0 seconds.
    java.net.BindException: Address already in use
            at java.net.PlainSocketImpl.socketBind(Native Method)
            at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
            at java.net.ServerSocket.bind(ServerSocket.java:319)
            at java.net.ServerSocket.<init>(ServerSocket.java:185)
            at java.net.ServerSocket.<init>(ServerSocket.java:97)
            at org.apache.axis2.transport.http.server.DefaultConnectionListener.run(DefaultConnectionListener.java:75)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:619)
    I don't know if this is an actual issue and if it's related, since things seem to be running fine on 8443. We can see with netstat/ps that a java OneClick process is listening on 8443 (and several other ports)

    We don't see excessive alarms on any models.

    Debugging on the OneClick admin page shows 0 alarms while CLI shows 82 alarms and the Alarm information section on the VNM model in OC shows 132 active alarms (including 47 maintenance alarms) and will update when you hit the refresh icon. The restful web services also show 132 alarms.

    We do see another symptom: if we try to delete a model (in this case a Fanout/Rpt_Segment) is will disappear from the Topology view, but remain visible in the Navagation pane and the List view. If we try to delete it again from there, a error popup states the model does not exist. After a OC client restart the Navigation and List view will also be updated.

    Regards,
    Michiel


  • 16.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Dec 16, 2013 06:51 AM
    Hello to All.
     
    We have the same issue here. Spectrum 9.2.2H08.
     
    The alarms are not generated. In Event Tab we see the events with severity, but in alarm tab we see nothing. In best cases alarm are appears with delay 6-40 minuts. Notification from SANM also not generated. Restart of SPECTRO Server are help but after 2-3 houres we have the same problem again...
     
    Does anyone resolve it? Thx for any help.


  • 17.  RE: Alarms not displaying in OneClick alarm tab view...

    Posted Dec 18, 2013 02:03 AM
    Hello again.
     
     
     
    I think we resolve this problem. When we had analysed it we saw to many Suppresed alarms are generated. We has disabled generating this type alarms and restart Spectrum. We not see this problem anymore already 2 days. I hope this a good workaround solution.
     
    But also we have problem with performance our disk subsystem, so I think this is root cause of our problem