Service Operations Insight

 View Only
  • 1.  5 min spectrum alert delay reflected in SOI

    Posted Jun 14, 2019 04:25 PM
    Edited by Daniel Blanco Jun 14, 2019 05:02 PM
    So I was having issues in my SOI lab where things just broke and I decided to wipe clean and re-install. I just installed the Spectrum 259 connector and pointing it to our Spectrum 10.3 version. I'm noticing when alerts appear in Spectrum those alerts then will eventually show up on the SOI side up to 5 min on the SOI console. Same thing when alerts clear. It takes ~5min for the update to hit the SOI console. 
    What setting is there that controls the updates from the connector to the SOI console? Its supposed to be instant but OOTB is there a delay or bulk setting in place here?

    ------------------------------
    Daniel Blanco
    Enterprise Tools Architect
    Alphaserve Technologies
    ------------------------------


  • 2.  RE: 5 min spectrum alert delay reflected in SOI

    Broadcom Employee
    Posted Jun 15, 2019 10:16 AM
    Hi Daniel,

    1) Make sure Spectrum Server, Spectrum connector & SOI Manager all are in the same time zone & network

    2) If so, you may verify \SOI\jsw\logs\SAM-IntegrationServices_wrapper.log, search for "Publishing Alert : " message. Check the date | time stamp of this message
         This should match with Spectrum OC alert time

    3) Check "\SOI\jsw\logs\SAM-IntegrationServices_wrapper.log" on the SOI Manager for a similar message. Check the date | time stamp of this message
          Both time stamps should be identical

    4) Check "\SOI\tomcat\logs\soimgr-debug.log or connmgr.log ('Connector Manager' component should be in DEBUG on the manager debug page) and check the date | time stamp for this alert id.

    You will see where the delay is. If this is at manager side, probably manager is busy and you could check Job queue monitor status for this Spectrum connector, may be the queue is piled up.

    Hope this helps.

    Thanks
    Brahma


  • 3.  RE: 5 min spectrum alert delay reflected in SOI

    Posted Jun 17, 2019 10:41 AM
    Hey Brahmaiah.
    The time and time zones are in sync on all boxes.
    I checked those log files and for some reason the alert in the connector logs doesn't show up till 9:33. I triggered the alert on the Spectrum side at 9:30.

    INFO | jvm 1 | 2019/06/17 09:33:45 | Publishing Alert : [CA:00005:Spectrum-as-nyclab-spec:5d0795da-930a-100a-0338-80000500a565:LASTRECORD]
    INFO | jvm 1 | 2019/06/17 09:33:45 | CA:00005_Spectrum-as-nyclab-spec@as-nyclab-cntr:LaunchEventProcessing Done.Process available alerts in the Subscribe Queue
    INFO | jvm 1 | 2019/06/17 09:33:46 | FIRST|LAST RECORD Processing ...

    I see in the connector logs though that it is loosing the heartbeat to the spectrum box:
    INFO | jvm 1 | 2019/06/17 09:34:58 | handleIFWSiloException: MDR host as-nyclab-spec is unavailable - heartbeat missed.
    INFO | jvm 1 | 2019/06/17 09:34:58 | com.ca.ucf.api.MDRUnavailableException: MDR host as-nyclab-spec is unavailable - heartbeat missed.

    I see the connector is then un-publishing it self and then re-connecting itself over and over every few minutes in the log. 
    I opened case #20009660 for this...
    Thanks.




    ------------------------------
    Daniel Blanco
    Enterprise Tools Architect
    Alphaserve Technologies
    ------------------------------



  • 4.  RE: 5 min spectrum alert delay reflected in SOI

    Posted Jun 17, 2019 12:39 PM
    How does the connector heartbeat check the Spectrum MDR that its up and responsive? These boxes are on the the same ESX host so there is no network issue but this was a fresh Spectrum 10.3 install on CentOS 7.6. Is there a timeout value I can bump up on the connector for it to not do a re-sync every 10min when there is no response from the Spectrum host?

    ------------------------------
    Daniel Blanco
    Enterprise Tools Architect
    Alphaserve Technologies
    ------------------------------



  • 5.  RE: 5 min spectrum alert delay reflected in SOI

    Posted Jun 17, 2019 03:31 PM
    Fixed the connector heartbeat issue by adding hosts entries on both the Spectrum Box and the connector box by adding entries on both pointing to each other. Then I cycled the connector and now the alerts sync up almost immediately.   :-)



    ------------------------------
    Daniel Blanco
    Enterprise Tools Architect
    Alphaserve Technologies
    ------------------------------