DX NetOps

 View Only
Expand all | Collapse all

SpectroServer Crash Frequently

  • 1.  SpectroServer Crash Frequently

    Posted May 27, 2020 04:42 PM
      |   view attached
    Hello everyone,

    We recently upgraded to Spectrum 10.3, since that our main Spectroserver (MLS) is in "Terminated" state frequently, about every 8 days. When trying to start the spectro server the error message of the attached image is displayed and a BD needs to be restored. How can I avoid these .VNMDB.LOCK or find the root cause of this.


    Thanks.


  • 2.  RE: SpectroServer Crash Frequently

    Broadcom Employee
    Posted May 27, 2020 04:48 PM
    Sabastian,

    The lock file is created when an application (like the SpectroSERVER) has a lock on the SpectroSERVER database. That way only one application can lock the database at any one time. The lock file is removed when the application shuts down gracefully. It is not when it terminates. When the application terminates, there is a high risk of database corruption and you need to restore from a previously saved database.

    To find the root cause of the crash, you will need to open a case with Spectrum support so the errors in the $SPECROOT/SS/VNM.OUT file and the dmp and mdmp files from the crash located in the $SPECROOT/SS/support directory can be analyzed.

    Joe


  • 3.  RE: SpectroServer Crash Frequently

    Posted May 27, 2020 05:32 PM
    Hello Joseph

    thanks for the reply. Whenever this happens I look at the VNM.OUT file and nothing related to the error is evident. Before opening a case I would like to know if I can put a debug mode or something that allows to identify something in the log related to the failure?


  • 4.  RE: SpectroServer Crash Frequently

    Broadcom Employee
    Posted May 28, 2020 07:46 AM
    Sebastian,

    There is no debug that I am aware of. An analysis of the crash data is needed.

    Since this is Windows, check for errors in the Windows Event Viewer as well.

    Joe


  • 5.  RE: SpectroServer Crash Frequently

    Posted May 28, 2020 10:35 AM
    Joseph,

    Following your recommendation when reviewing the windows event viewer in the period of the failure, several Warning type events are registered:



    And at the time the backup is executed an Error type event is evident:


    Can you think that the failure occurs due to server performance problems?



  • 6.  RE: SpectroServer Crash Frequently

    Broadcom Employee
    Posted May 28, 2020 10:51 AM
    Sebastian,

    Looks like you are running out of memory. So either some other process is using up memory and there is no memory available for Spectrum or the SpectroSERVER is using up all the memory indicating a possible memory leak.

    Joe


  • 7.  RE: SpectroServer Crash Frequently

    Posted Jun 01, 2020 09:23 AM
    Hi Joseph,

    I still have not solved my case, whenever the problem occurs I show that the Spectrum processes (spectroserver.exe, mysqld.exe, archmgr.exe) consume around 7 GB of server memory and total is 8GB. This is a very strange behavior since in normal situations the same 3 processes do not consume more than 2 GB of memory.

    I do not know if it is recommended to increase the memory of the server since it can continue consuming the total that is assigned.


  • 8.  RE: SpectroServer Crash Frequently
    Best Answer

    Broadcom Employee
    Posted Jun 01, 2020 10:02 AM
    Sebastian,

    Trouble shooting Spectrum crashes are too complicated and involved for the communities.

    Please open a case with Spectrum Support so the proper analysis can be done to find the root cause and resolve the issue.

    Joe



  • 9.  RE: SpectroServer Crash Frequently

    Broadcom Employee
    Posted May 28, 2020 08:50 AM
    Hi Sebastian,

    Besides the information that Joseph asked to attach to the support case, also attach the output of "getSpectrumInfo.sh lite" script.
    https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/spectrum/10-4/administrating/oneclick-administration/troubleshooting-oneclick/using-the-getspectruminfo-script.html

    ------------------------------
    Technical Support Engineer IV
    Broadcom Inc
    ------------------------------



  • 10.  RE: SpectroServer Crash Frequently

    Posted May 29, 2020 08:31 PM
    Hi Sebastian.

    I have a similar situation in differents enviroments with Spectrum 10.4 and 10.3 on Linux RedHat 7.5 and 7.3.

    In my case I can see memory errors like the following at the same time in 4 SSs.

    ./audit/audit.log:type=ANOM_ABEND msg=audit(1589468891.480:208339): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=29884 comm="SpectroSERVER" reason="memory violation" sig=11
    ./messages-20200517:May 14 10:08:11 HOSTNAME-SS-MLS kernel: SpectroSERVER[29884]: segfault at 7fa227b0d000 ip 00007fa26f441f00 sp 00007fa22cfb96a0 error 4 in libGlobl.so.1[7fa26f3f3000+d4000]
    ./messages-20200517:May 14 10:08:11 HOSTNAME-SS-MLS abrt-hook-ccpp: Process 29884 (SpectroSERVER) of user 1000 killed by SIGSEGV - dumping core
    ./messages-20200517:May 14 10:08:16 HOSTNAME-SS-MLS abrt-server: Executable '/home/SPECTRUM/SS/SpectroSERVER' doesn't belong to any package and ProcessUnpackaged is set to 'no'

    But reviewing the performance I do not see any peak or statistic that shows the saturated memory.


    I add the link of my question.

    https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?MessageKey=bfa3dc51-f65d-4d76-ab98-2a2f16e6dbc1&CommunityKey=671164c3-e575-4b08-96ab-edc2e1ceed13&tab=digestviewer#bmbfa3dc51-f65d-4d76-ab98-2a2f16e6dbc1

    Have you solved your case?



    Regards

    ------------------------------
    Consultant
    DST
    ------------------------------