Service Operations Insight

 View Only
  • 1.  ActiveMQ data kahadb 50GB

    Posted Apr 02, 2020 11:26 AM
    Edited by Daniel Blanco Apr 02, 2020 11:29 AM
    Hello,
    So I got an alert this morning that our SOI MGR drive was almost filled up. 
    Investigating I found that the C:\Program Files (x86)\CA\SOI\apache-activemq\data\kahadb directory has 2900+ entries and was taking up 52GB on our C:\ drive. 
    There is no issue with the DB Connectivity between the SOI MGR and the SQL DB Box. The SOI MGR Debug page the DB Check is all 'goog' results.

    What could be causing this issue? What check's can I run to make sure nothing is being blocked or broken that may be causing this. What logs can I check.

    TIA... 
    ​​
    I already reviewed and saw these threads but they were no help:
    https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=741961
    https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=740512

    We only have a few hundred alerts on the console. In total < 1k.

    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------


  • 2.  RE: ActiveMQ data kahadb 50GB

    Broadcom Employee
    Posted Apr 03, 2020 03:22 AM
    Hello Daniel, can you please verify if one or more of the connectors is reinitializing ? Please check in the Queue Monitor in the SOI Manager debug page http://<soimanager>:7090/sam/debug if you have a large amount of job queue processing.  Please see article https://knowledge.broadcom.com/external/article?articleId=95458
    for steps how to recover from this.

    Kind regards,
    Britta



  • 3.  RE: ActiveMQ data kahadb 50GB
    Best Answer

    Posted Apr 03, 2020 11:06 AM
    Hi Britta! So yesterday did a full complete shut down of all SOI services.
    Then cleared out every single log file on all the SOI boxes and also all the db-##.log files under C:\Program Files (x86)\CA\SOI\apache-activemq\data\kahadb

    I jumped onto the SQL DB box to check DB and I noticed that in SQL Studio the DB Connection wasn't opening and found that strange. So then did a SQL Service STOP, and then START and then SQL Studio was back to normal and I was able to expand the DB tree. 

    Then started up all SOI services and now things are back to normal. The kahadb dir is normal with 4 files. 

    BTW there were no jobs in the debug Active Monitor queue. It was empty. All connectors were up and Online. Thanks for the suggestion though.

    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------



  • 4.  RE: ActiveMQ data kahadb 50GB

    Broadcom Employee
    Posted Apr 03, 2020 11:26 AM
    Hello Daniel, good to hear this is fixed. Take good care of yourself and have a nice weekend.

    Kind regards,
    Britta