Automic Workload Automation

 View Only
Expand all | Collapse all

Anyone experiencing Tomcat memory over utilization (memory leak?)

  • 1.  Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 21, 2023 10:23 AM

    Dear Experts,

    We are running Automic V21 on Tomcat 9.0.22.  After every few days ( 3 - 5 days) , the memory consumption for Tomcat is very high. We have configured to have 8GB min and 16GB as max heap size.

    Within couple of days,  the tomcat service is using around 16 GB. 

    Any idea how can I trouble shoot here please ? 

    Thanks and Regards

    Vimalan



  • 2.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 21, 2023 05:47 PM

    Hey there,

    yes, we've experienced the same.
    Initially i thought it was due the Tomcat version. The same day when i've upgraded our Prod to v21.0.5HF4 ive also upgraded both Tomcat to 9.0.76 . Since then every 4-5 days we need to restart at least one of them.
    We have 2 Tomcats behind Apache LB, and they serve both Prod and Dev. 
    After few days the connections is dragging and the memory utilizations of the tomcat process is pretty high (up to 22GB).
    Last Saturday ive upgraded one of the Tomcats to 9.0.79 and the AE to 21.0.7HF1 and maybe its slightly better. 

    Still i am monitoring this behaviour, only 2 days passed. So far both Tomcats are ~10GB memory consumptions. 



    ------------------------------
    ------------------------------
    Automic SME @ DXC.Technology
    ------------------------------
    ------------------------------



  • 3.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Sep 06, 2023 12:45 PM

    Hi -  Were you able to successfully upgrade only the web server to 21.0.7HF1 or did it require an upgrade to the entire environment?




  • 4.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 22, 2023 01:58 AM

    Hi Maria,

    This is a known bug in the AWI. We have an open support case and it will be fixed in an upcoming hotfix.

    You may see the following message in the AWI logs:

    2023-07-22 22:54:21,564 Timer-0                [ERROR] NOSESSION/- NOUI   [c.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed
    java.lang.reflect.UndeclaredThrowableException: null



  • 5.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 22, 2023 03:09 AM
    Edited by Josef Scharl Aug 22, 2023 03:11 AM

    Hi,
    It's the same here …

    We opened a ticket some time ago and got AWI 21.0.7 as fixed version, but it didn't fix the memory leak. Maybe it's a little bit better, but we still need to restart AWI a least once a week.

    There is also an article online for the bug: https://knowledge.broadcom.com/external/article?articleId=263060

    KR, Tscho




  • 6.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 22, 2023 10:52 AM

    I have a case opened for the same issue.  Support indicated that the hotfix is planned for 21.0.8.




  • 7.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 22, 2023 04:52 PM

    I had the same problem, until I set Apache Tomcat Heap size to 8192 

    Web interface hasn't crashed since then. 




  • 8.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 24, 2023 02:25 AM

    Hi Keld, 

    Is this issue cause any issues for AWI search for use option like not working?

    Thanks and Regards, 

    Venkata Yaswanth 




  • 9.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Aug 24, 2023 03:47 PM

    Yes, I have Automic restart Tomcat once a day. It's probably more often than need be, but I'd rather that than have everything crash.




  • 10.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Sep 19, 2023 06:25 AM
    Edited by Leon Carroll Sep 19, 2023 06:25 AM

    I'm using the Jetty web interface on Windows 2022, I tend it finds to grow to around 10GB memory usage in 21.0.7 then slows down the web ui till I restart it and it drops back to around 1GB. I need to do this every two to four weeks on average. 



  • 11.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Sep 19, 2023 01:13 PM

    Well  i uploaded and installed only Automic Web Interface 21.0.8-release-21.0.8-1692830826459-09cebf03e1d and can report the [ERROR] NOLOGIN/- NOUI   [com.uc4.ecc.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed messages are still appearing every 30 seconds running Apache Tomcat 9.0.72 Server

    So I guess this was a fib.

    UI session cleanup fails and causes system downtime. (AE-33133)

    33502770

    A problem has been fixed where the UI session cleanup ("cleanupStaleConnections") did not work causing multiple error messages being logged and eventual system downtime due to memory overload.

    Components:

    ·        Automic.Web.Interface

    Fixed Versions:

    ·        Automation.Engine 21.0.8 (Release date: 2023-09-07)

    ·        Automation.Engine 21.0.7 HF3 (Release date: 2023-08-25)




  • 12.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Sep 19, 2023 02:55 PM

    As Keld Mollnitz stated, I was also getting [ERROR] NOSESSION/- NOUI and not the [ERROR] NOLOGIN/- NOUI.  The end results still is the cleanupStaleConnections Failed and having to restart Tomcat (8.5.57).

    I upgraded our SandBox environment to 21.0.8 on 9/14.  Our testers are in the process of validating and so far I do not see any cleanupStaleConnections Failed messages.  This worries me since we hoped that 21.0.8 will resolve the heap memory issues.




  • 13.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Sep 25, 2023 07:51 AM
    Edited by Michael A. Lowry Sep 25, 2023 07:58 AM

    @Timothy Yanosko:

    Have you seen the symptom too? I mean, do you have to restart the AWI periodically to maintain application responsiveness?



  • 14.  RE: Anyone experiencing Tomcat memory over utilization (memory leak?)

    Posted Sep 25, 2023 04:22 PM

    Hi Michael,  Yes we've scheduled weekly restarts of the tomcat servers.