Anyone experiencing Tomcat memory over utilization (memory leak?)

[Maria Joseph Vimalan]

Dear Experts,

We are running Automic V21 on Tomcat 9.0.22.  After every few days ( 3 - 5 days) , the memory consumption for Tomcat is very high. We have configured to have 8GB min and 16GB as max heap size.

Within couple of days,  the tomcat service is using around 16 GB. 

Any idea how can I trouble shoot here please ? 

Thanks and Regards

Vimalan

[Krum Ganev]

Hey there,

yes, we've experienced the same.
Initially i thought it was due the Tomcat version. The same day when i've upgraded our Prod to v21.0.5HF4 ive also upgraded both Tomcat to 9.0.76 . Since then every 4-5 days we need to restart at least one of them.
We have 2 Tomcats behind Apache LB, and they serve both Prod and Dev. 
After few days the connections is dragging and the memory utilizations of the tomcat process is pretty high (up to 22GB).
Last Saturday ive upgraded one of the Tomcats to 9.0.79 and the AE to 21.0.7HF1 and maybe its slightly better. 

Still i am monitoring this behaviour, only 2 days passed. So far both Tomcats are ~10GB memory consumptions. 

------------------------------
------------------------------
Automic SME @ DXC.Technology
------------------------------
------------------------------

[Timothy Yanosko]

Hi -  Were you able to successfully upgrade only the web server to 21.0.7HF1 or did it require an upgrade to the entire environment?

[Keld Mollnitz]

Hi Maria,

This is a known bug in the AWI. We have an open support case and it will be fixed in an upcoming hotfix.

You may see the following message in the AWI logs:

2023-07-22 22:54:21,564 Timer-0                [ERROR] NOSESSION/- NOUI   [c.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed
java.lang.reflect.UndeclaredThrowableException: null

[Josef Scharl]

Hi,
It's the same here …

We opened a ticket some time ago and got AWI 21.0.7 as fixed version, but it didn't fix the memory leak. Maybe it's a little bit better, but we still need to restart AWI a least once a week.

There is also an article online for the bug: https://knowledge.broadcom.com/external/article?articleId=263060

KR, Tscho

[Lester Chew]

I have a case opened for the same issue.  Support indicated that the hotfix is planned for 21.0.8.

[Marilyn Singley]

I had the same problem, until I set Apache Tomcat Heap size to 8192 

Web interface hasn't crashed since then. 

[Venkata Yaswanth Chinni]

Hi Keld, 

Is this issue cause any issues for AWI search for use option like not working?

Thanks and Regards, 

Venkata Yaswanth 

[Bethel Ademowore]

Yes, I have Automic restart Tomcat once a day. It's probably more often than need be, but I'd rather that than have everything crash.

[Leon Carroll]

I'm using the Jetty web interface on Windows 2022, I tend it finds to grow to around 10GB memory usage in 21.0.7 then slows down the web ui till I restart it and it drops back to around 1GB. I need to do this every two to four weeks on average. 

[Timothy Yanosko]

Well  i uploaded and installed only Automic Web Interface 21.0.8-release-21.0.8-1692830826459-09cebf03e1d and can report the [ERROR] NOLOGIN/- NOUI   [com.uc4.ecc.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed messages are still appearing every 30 seconds running Apache Tomcat 9.0.72 Server<o:p></o:p>

<o:p> </o:p>

So I guess this was a fib.<o:p></o:p>

<o:p> </o:p>

UI session cleanup fails and causes system downtime. (AE-33133)<o:p></o:p>

33502770<o:p></o:p>

A problem has been fixed where the UI session cleanup ("cleanupStaleConnections") did not work causing multiple error messages being logged and eventual system downtime due to memory overload.<o:p></o:p>

Components:<o:p></o:p>

·        Automic.Web.Interface<o:p></o:p>

Fixed Versions:<o:p></o:p>

·        Automation.Engine 21.0.8 (Release date: 2023-09-07)<o:p></o:p>

·        Automation.Engine 21.0.7 HF3 (Release date: 2023-08-25)<o:p></o:p>

[Lester Chew]

As Keld Mollnitz stated, I was also getting [ERROR] NOSESSION/- NOUI and not the [ERROR] NOLOGIN/- NOUI.  The end results still is the cleanupStaleConnections Failed and having to restart Tomcat (8.5.57).

I upgraded our SandBox environment to 21.0.8 on 9/14.  Our testers are in the process of validating and so far I do not see any cleanupStaleConnections Failed messages.  This worries me since we hoped that 21.0.8 will resolve the heap memory issues.

[Michael A. Lowry]

@Timothy Yanosko:

Have you seen the symptom too? I mean, do you have to restart the AWI periodically to maintain application responsiveness?

[Timothy Yanosko]

Hi Michael,