DX NetOps

  • Private Community
 View Only

  • 1.  The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Posted May 15, 2020 11:07 AM
    Hi All, 

    Greetings! Hope you're doing well. 
    I am seeing this alarm and trying to figure out how to handle this. I've seen related threads but with different error message. 

    The alarm is CONTACT LOST TO PRIMARY ARCHIVE MANAGER
    Symptoms: The SpectroSERVER cannot connect to the primary Archive Manager.
    Alarm
    CA Spectrum Version is 10.0.0.000

    ARCHMGR.OUT output:
    Apr 15 11:36:45 : ArchMgr started as user 'spadmin'
Apr 15 11:36:45 : ArchMgr validating database.
SPC-SHD-29021: 
**************************************************************************
The following output is intended to facilitate debugging, please forward
to CA technical support.  The files listed are not part of the customer
installation of SPECTRUM.
**************************************************************************

Apr 15 11:36:46 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
SHOW COLUMNS FROM event LIKE 'node_id' - Table '.\ddmdb\event' is marked as crashed and should be repaired

Database corruption detected:
    ddmdb.event - Record at pos: 440259984 is not remove-marked
    ddmdb.event - record delete-link-chain corrupted
    ddmdb.event - Corrupt

Error opening the DDM database. One or more tables are corrupt.
Recovery options:

  1. Run `ArchMgr -repair` to attempt recovery
  2. Load a valid DDM savefile using `ddm_load <savefile>`
  3. Initialize the DDM database using `ddm_load -i <LANDSCAPE_HANDLE>`

Apr 15 11:36:46 : ArchMgr error opening database.
​



    Thank you in advance for your help. 



    ------------------------------
    Cheers,
    Noc.user
    ------------------------------


  • 2.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Broadcom Employee
    Posted May 15, 2020 11:19 AM
    As the error states, database corruption was detected. You need to either run the repair or load in a previously saved database.

    1. Log into the system as the user that owns the Spectrum installation

    2. If on Windows, start a bash shell by running "bash -login"

    3. cd to the $SPECROOT/SS/DDM directory and run the following command:

    ./ArchMgr -repair

    4. When complete, start the SpectroSERVER as normal

    Joe
    Original Message


  • 3.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Posted May 15, 2020 10:56 PM
    Edited by Noc.user May 15, 2020 11:03 PM
    Hi @Joseph Ackley, thank you for your feedback. 

    Sorry, I have additional questions: 
    • May I ask how to determine if the ArchMgr is running or not? or how to start/stop?
    • May I ask if there are services/components need to be stop/down before I can run the ArchMgr -repair command?
    • Also, I figured that we have a secondary SS, if the database corruption was fixed in the primary SS by the repair command, will the DB of the secondary SS will also be fixed? or should I also run the command on the second SS? 

    Operating System: Windows Server
    CA Spectrum Version is 10.0.0.000

    Secondary SS ARCHMGR.OUT output:
     
    Mar 21 11:32:34 : ArchMgr started as user 'spadmin'
Mar 21 11:32:34 : ArchMgr validating database.
SPC-SHD-29021: 
**************************************************************************
The following output is intended to facilitate debugging, please forward
to CA technical support.  The files listed are not part of the customer
installation of SPECTRUM.
**************************************************************************

Mar 21 11:32:35 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
SHOW COLUMNS FROM event LIKE 'node_id' - Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:36 : ArchMgr successfully connected to MySQL service
Mar 21 11:32:36 : ArchMgr loaded DDM database with landscape handle 0x100000

Mar 21 11:32:37 : ArchMgr has successfully connected to the SpectroSERVER.
Mar 21 11:32:37 : ArchMgr has successfully advertised CORBA Event Service.

Mar 21 11:32:37 : ArchMgr is now ready on port 0xbafe, precedence 20

Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
SHOW COLUMNS FROM event LIKE 'node_id' - Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783157,99506,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0―0\0\0SENSU: CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\n\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783157,99506,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0―0\0\0SENSU: CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\n\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783157,99506,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0―0\0\0SENSU: CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\n\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1688): Fatal Event Log Manager Error.  Scheduling ArchMgr shutdown.
Mar 21 11:32:38 : Closing all client connections...
Mar 21 11:32:38 : Stopping CORBA communication...
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202358,1584783157,99505,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202358,1584783157,99505,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202358,1584783157,99505,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202358,1584783157,99506,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0―0\0\0SENSU: CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202358,1584783157,99506,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0―0\0\0SENSU: CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202358,1584783157,99506,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\0\0\0verticakms28-prod-172-19-131-18e\0\0\0\0\0\0=\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_cpu -w 80 -c 95 -af\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0z\0\0\0CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0―0\0\0SENSU: CPU OK : user=39% system=2% iowait=0% idle=59% | cpu_user=39%;80;95; cpu_sys=2%;80;95; cpu_iowait=0%;80;95; cpu_idle=59%;\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783158,99514,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0j\0\0\0SENSU: DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\n\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783158,99514,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0j\0\0\0SENSU: DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\n\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783158,99514,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0j\0\0\0SENSU: DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\n\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202335,1584783158,99513,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202335,1584783158,99513,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202335,1584783158,99513,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202335,1584783158,99514,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0j\0\0\0SENSU: DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202335,1584783158,99514,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0j\0\0\0SENSU: DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202335,1584783158,99514,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\0\0\0rbexport-prod-172-19-137-161e\0\0\0\0\0\0I\0\0\0PATH=$PATH:/usr/lib64/nagios/plugins check_disk -w 10% -c 5% -p /var/log/f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0c\0\0\0DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0j\0\0\0SENSU: DISK OK - free space: /var/log 28222 MiB (98.82% inode=100%);| /var/log=334MiB;27097;28602;0;30108\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783158,99518,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0)\0\0\0SENSU: MEM OK - system memory usage: 29%\n\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783158,99518,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0)\0\0\0SENSU: MEM OK - system memory usage: 29%\n\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1116698,1584783158,99518,699,1,'',1,4293918971,0,'\n\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0)\0\0\0SENSU: MEM OK - system memory usage: 29%\n\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202305,1584783158,99517,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202305,1584783158,99517,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202305,1584783158,99517,699,1,'',1,4293918970,0,'	\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202305,1584783158,99518,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0)\0\0\0SENSU: MEM OK - system memory usage: 29%\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202305,1584783158,99518,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0)\0\0\0SENSU: MEM OK - system memory usage: 29%\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1202305,1584783158,99518,699,1,'',1,4293918971,0,'\0\0\0\0\0\0\0\0\0\"\0\0\0rbfinalization-prod-172-19-128-236e\0\0\0\0\0\0$\0\0\0check-memory-percent.rb  -w 85 -c 90f\0\0\0\0\0\0\0\0\0\0d\0\0\0\0\0\0\"\0\0\0MEM OK - system memory usage: 29%\ng\0\0\0\0\0\0\0\0\0createh\0\0\0\0\0\0\0\0\0ﾐ#\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0L+\0\0\0\0)\0\0\0SENSU: MEM OK - system memory usage: 29%\n\n,\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:38 ERROR TRACE at ModelArchDBImp.cc(4814): doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1048576,1584783158,99523,699,1,'',1,67329,20,'\n\0\0\0\0\0\0\0\0\0ﾂ・\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\06ﾟu^ﾂ・\0ｻ\0€\0\0	\0\0\0\0\0\0\0\0\0\0\0\06ﾟu^\0\0\0ﾐ\0€\0\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0') - Table '.\ddmdb\event' is marked as crashed and should be repaired
ModelArchDb ERROR at DDMDbIf2.cc(885): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1048576,1584783158,99523,699,1,'',1,67329,20,'\n\0\0\0\0\0\0\0\0\0ﾂ・\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\06ﾟu^ﾂ・\0ｻ\0€\0\0	\0\0\0\0\0\0\0\0\0\0\0\06ﾟu^\0\0\0ﾐ\0€\0\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2901): Error attempting to end transaction.
ModelArchDb ERROR at DDMDbIf.cc(2911): 
Generic Error: 1, commitTransaction/doSqlQuery/mysql_query: Failure executing query:
INSERT DELAYED IGNORE INTO event(model_h, utime, counter, clk_seq, version, node_id, user_key, type, severity, vardata_string) VALUES(1048576,1584783158,99523,699,1,'',1,67329,20,'\n\0\0\0\0\0\0\0\0\0ﾂ・\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\06ﾟu^ﾂ・\0ｻ\0€\0\0	\0\0\0\0\0\0\0\0\0\0\0\06ﾟu^\0\0\0ﾐ\0€\0\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\0\0\0\n,\0\0\0\0\0\0\0')
Native Error: 145, Table '.\ddmdb\event' is marked as crashed and should be repaired
Mar 21 11:32:38 ERROR TRACE at DDMDbIf.cc(2913): Aborting database transaction.
Mar 21 11:32:38 ERROR TRACE at CsEventMgr.cc(1667): Event Log Manager: Error attempting to log event to database.
Mar 21 11:32:39 : Closing database
Mar 21 11:32:39 : ArchMgr is shutting down...
Mar 21 11:32:39 : ArchMgr has successfully shut down.

    Thanks again in advance. 




    ------------------------------
    Cheers, 
    Noc.user
    ------------------------------

    Original Message


  • 4.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager
    Best Answer

    Broadcom Employee
    Posted May 15, 2020 11:08 PM
    Please see my answers below:

    May I ask how to determine if the ArchMgr is running or not? or how to start/stop?
    ANSWER: If running in Windows, look for the ArchMgr.exe process in the Windows Task Manager. Ir running Linux, run "ps -ef | grep ArchMgr" from the command line
    To start/stop the Archive Manager, Launch the Spectrum Control Panel (SCP) and start/stop using the SCP

    May I ask if there are services/components need to be stop/down before I can run the ArchMgr -repair command?
    ANSWER: No there are no services/components that need to be stoped before running the repair

    Also, I figured that we have a secondary SS, if the database corruption was fixed in the primary SS by the repair command, will the DB of the secondary SS will also be fixed? or should I also run the command on the second SS?
    ANSWER: You will need to run the repair on the secondary SS as well.

    Joe
    Original Message


  • 5.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Posted May 15, 2020 11:14 PM
    Awesome! Thank you, @Joseph Ackley. I'll work on this and update you. ​​

    ------------------------------
    Cheers,
    Noc.user
    ------------------------------

    Original Message


  • 6.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Broadcom Employee
    Posted May 18, 2020 02:22 PM
    You may also reference this KB article on Archive Manager Repair: 

    https://knowledge.broadcom.com/external/article?articleId=57378

    ------------------------------
    Senior Support Engineer
    Broadcom
    ------------------------------

    Original Message


  • 7.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Posted May 18, 2020 10:20 PM
    Hi @Jason Volpini, thank you for the KB link.
    I'll take note of this as well. ​

    ------------------------------
    Cheers,
    Noc.user
    ------------------------------

    Original Message


  • 8.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Posted May 18, 2020 10:18 PM
    Hi @Joseph Ackley, 
    The repair was successful. Thank you for your help. ​

    ------------------------------
    Cheers,
    Noc.user
    ------------------------------

    Original Message


  • 9.  RE: The SpectroSERVER cannot connect to or has lost contact with the primary Archive Manager

    Broadcom Employee
    Posted May 19, 2020 06:57 AM
    OUTSTANDING!!!

    Glad I was able to assist.

    Joe
    Original Message