Anyone experiencing Tomcat memory over utilization (memory leak?)

[Maria Joseph Vimalan]

Dear Experts,

We are running Automic V21 on Tomcat 9.0.22.  After every few days ( 3 - 5 days) , the memory consumption for Tomcat is very high. We have configured to have 8GB min and 16GB as max heap size.

Within couple of days,  the tomcat service is using around 16 GB. 

Any idea how can I trouble shoot here please ? 

Thanks and Regards

Vimalan

[Krum Ganev]

Hey there,

yes, we've experienced the same.
Initially i thought it was due the Tomcat version. The same day when i've upgraded our Prod to v21.0.5HF4 ive also upgraded both Tomcat to 9.0.76 . Since then every 4-5 days we need to restart at least one of them.
We have 2 Tomcats behind Apache LB, and they serve both Prod and Dev. 
After few days the connections is dragging and the memory utilizations of the tomcat process is pretty high (up to 22GB).
Last Saturday ive upgraded one of the Tomcats to 9.0.79 and the AE to 21.0.7HF1 and maybe its slightly better. 

Still i am monitoring this behaviour, only 2 days passed. So far both Tomcats are ~10GB memory consumptions. 

------------------------------
------------------------------
Automic SME @ DXC.Technology
------------------------------
------------------------------

[Timothy Yanosko]

Hi -  Were you able to successfully upgrade only the web server to 21.0.7HF1 or did it require an upgrade to the entire environment?

[Keld Mollnitz]

Hi Maria,

This is a known bug in the AWI. We have an open support case and it will be fixed in an upcoming hotfix.

You may see the following message in the AWI logs:

2023-07-22 22:54:21,564 Timer-0                [ERROR] NOSESSION/- NOUI   [c.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed
java.lang.reflect.UndeclaredThrowableException: null

[Josef Scharl]

Hi,
It's the same here …

We opened a ticket some time ago and got AWI 21.0.7 as fixed version, but it didn't fix the memory leak. Maybe it's a little bit better, but we still need to restart AWI a least once a week.

There is also an article online for the bug: https://knowledge.broadcom.com/external/article?articleId=263060

KR, Tscho

[Lester Chew]

I have a case opened for the same issue.  Support indicated that the hotfix is planned for 21.0.8.

[Marilyn Singley]

I had the same problem, until I set Apache Tomcat Heap size to 8192 

Web interface hasn't crashed since then. 

[Venkata Yaswanth Chinni]

Hi Keld, 

Is this issue cause any issues for AWI search for use option like not working?

Thanks and Regards, 

Venkata Yaswanth 

[Bethel Ademowore]

Yes, I have Automic restart Tomcat once a day. It's probably more often than need be, but I'd rather that than have everything crash.

[Leon Carroll]

I'm using the Jetty web interface on Windows 2022, I tend it finds to grow to around 10GB memory usage in 21.0.7 then slows down the web ui till I restart it and it drops back to around 1GB. I need to do this every two to four weeks on average. 

[Timothy Yanosko]

Well  i uploaded and installed only Automic Web Interface 21.0.8-release-21.0.8-1692830826459-09cebf03e1d and can report the [ERROR] NOLOGIN/- NOUI   [com.uc4.ecc.framework.core.inject.BaseScopeManager] - cleanupStaleConnections failed messages are still appearing every 30 seconds running Apache Tomcat 9.0.72 Server

So I guess this was a fib.

UI session cleanup fails and causes system downtime. (AE-33133)

33502770

A problem has been fixed where the UI session cleanup ("cleanupStaleConnections") did not work causing multiple error messages being logged and eventual system downtime due to memory overload.

Components:

·        Automic.Web.Interface

Fixed Versions:

·        Automation.Engine 21.0.8 (Release date: 2023-09-07)

·        Automation.Engine 21.0.7 HF3 (Release date: 2023-08-25)

[Lester Chew]

As Keld Mollnitz stated, I was also getting [ERROR] NOSESSION/- NOUI and not the [ERROR] NOLOGIN/- NOUI.  The end results still is the cleanupStaleConnections Failed and having to restart Tomcat (8.5.57).

I upgraded our SandBox environment to 21.0.8 on 9/14.  Our testers are in the process of validating and so far I do not see any cleanupStaleConnections Failed messages.  This worries me since we hoped that 21.0.8 will resolve the heap memory issues.

[Michael A. Lowry]

@Timothy Yanosko:

Have you seen the symptom too? I mean, do you have to restart the AWI periodically to maintain application responsiveness?

[Timothy Yanosko]

Hi Michael,  Yes we've scheduled weekly restarts of the tomcat servers.

[Michael A. Lowry]

Is anyone not experiencing Tomcat memory over-utilization?

[Maria Joseph Vimalan]

Hi Michael , All,

I tried with 21 .7 HF3 but still didnt work. We still have to restart the tomcat application atleast 5 days once. 

Regards

Vimalan

[Pete Wirfs]

Hello everyone.  I've been retired for over a year, but my employer asked me to participate in their upgrade to v21, and I'm here trolling for known problems.

We are planning to upgrade to 21.0.8 soon, but I gather the 21.0.8 AWI still has a known memory leak that is managed by recycling tomcat periodically.  is that correct? 

EDIT:
I've been informed we are using Jetty.  I presume it has leak problems too?

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Keld Mollnitz]

Hi @Pete Wirfs,

such a pleasant surprise to see you back in our community...:-)

Unrelated to the AWI, but be informed about an issue with AE v21.0.8 HF1 and the z/OS agent (any version). If you are using the z/OS agent and use it for file transfers (JOBF). If you try to restart a failed file transfer, the z/OS agent may crash !

Prior to version 21.0.8 of Automation Engine, the Automation Engine would always choose "Transfer all new" when you restart a file transfer - even if the user chose "From last restart position"!

Starting from version 21.0.8, the Automation Engine now takes the input from the Restart dialog box (!), however the "Restart from last position" feature is not implemented in the z/OS agent code (all other agents should be ok).

The changed behavior is not documented in the Release Notes (!).

/Keld.

[Adrian Fresno Menendez]

Hello all,

Broadcom had taken note of these problems with the high memory utilization of the AWI and proactively launched a new battery of memory leak scans which resulted into finding new memory-leak issues in AWI that are now being fixed into an AWI hotfix 21.0.8 HF3 that should be out by early November.

We are very sorry about the inconvenience and thank you for your feedback.

In the future, if any issue like this is detected, please take a jmap dump of the java process that starts AWI (usually the Tomcat PID) with a command like this when the memory usage of the process is quite high (ie. around 4GB or more):

jmap -dump:format=b,file=dumptomcat.bin <PID_TOMCAT>

Ideally, capture several of them spaced several hours and please create a new case with Technical Support uploading the output files to the case attachments so that we can analyze them.

Thanks to everyone,

Adrian 

[Michael A. Lowry]

I suggest publishing these troubleshooting instructions in a dedicated KB article.

[Leon Carroll]

Hi Pete

I'm using Jetty with v21.0.7HF3.  It tends to build up till around 5GB or so and then the server goes to 100% till Jetty is restarted. I've put in a weekly restart, though it will run for up to a month before this happens.  I tried increasing the server memory headroom but it didn't help.

On the whole V21 is much better than V12.3.   There's a lot of improvements here.

Some things to watch out for - when you generate your CSR, add the server names, the fully qualified server names, the ip addresses, and the load balanced address to the names list for the cert, otherwise the rest url won't work via the load balancer

Agents that start up before the engine and fail to connect shut down again and don't retry, perhaps I'm missing something in the ini file here

Prompsets accept a leading space in v12.3 and trim them automatically, v21 doesn't and will correctly fail to match.  

I have found the RA BO 4.3.0 agent works with BO 4.3 but not RA BO 4.3.2 or 4.3.3

I was advised to run v21 alongside v12.3 and migrate schedules and workloads across, i did this alongside a platform update to Windows 2022, and it took me a year.  

[Pete Wirfs]

Awesome information!

We don't have z/OS so no concerns there.  (Most of my career I was a z/OS developer who administered a z/OS scheduler, but I helped eliminate it in 2012 after which I redirected my attention to the install and support of something called UC4. LOL!)

My staff are now aware they should monitor AWI operations and possibly install automatic restarts of jetty.  They can't wait for HF3.

They had already learned the hard way about the CSR deal.

In addition to the agents failing to retry, they have experienced an orphaned 21.0.5 windows agent consuming memory until it choked the server to death.

We are passing along the promptset behavior change to our application testers.  We are heavy users of promptsets and were unaware!

Pete

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Leon Carroll]

There is this article for Jetty, the java memory size guidelines aren't in the install guide, but perhaps this helps:
How to increase HEAP size when activating AWI via the Jetty Launcher? (broadcom.com)

[Leon Carroll]

I tried this on my development server (21.0.7HF3) with the recommended -Xmx2048M  and it hit 2Gb in a couple of days and then the CPU hit 100%.  I've removed this again.  I couldn't even start the command prompt, Sorry Kai, I haven't got any dumps.

[Kaj Wierda]

I suggest you try the dev build of 21.0.8 HF3 on your development server.

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------

[Leon Carroll]

Hello, after a week v21.0.8 HF3 is currently consuming 2620 MB of memory on my development system and it is running fine.  I'll update again in another week. We do most of our development there so in terms of AWI usage it is probably the most used intensively. 

I'm running Adoptium OpenJDK 11.0.21_9 and I'm not setting the -xmx flag.

[Michael A. Lowry]

There are two ways of running the AWI:

• Tomcat
• Jetty

Is there a difference in memory leak behavior between these two scenarios?

------------------------------

[Kaj Wierda]

We recommend all customers experiencing memory leak issues with either Tomcat or Jetty to provide us with a heap dump via a support ticket.

Disclaimer:  Please keep in mind that a heap-dump contains a full snapshot of the web-server, and it most likely contains sensitive information like your passwords, intellectual property and personal information. Take care to avoid creating a dump on a sensitive/production environment, instead - if possible - reproduce your issue on an isolated test-system

In order to create a heap dump from a running AWI instance, you must run the following commands (on Linux, AWI is running on Jetty):

 

# Find the PID (Process ID) of the AWI running on Jetty

PID=$(jps | grep 'aa-webui-launcher' | awk '{print $1}')

 

# Run java garbage collection first, to ensure the dump only contains potentially leaked memory

jcmd $PID GC.run

 

# Create a Heap Dump

jmap -dump:format=b,file=awi-heapdump-$(hostname)-$(date -Is).bin $PID

For Apache Tomcat, the first command is slightly different:

 

# Find the PID (Process ID) of the AWI running on Tomcat
PID=$(jps -v | grep 'catalina' | awk '{print $1}')

 

For more information, please refer to the Documentation of your Java Runtime Environment and how to use the tools (jps, jmap, jcmd) mentioned here.

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------

[Kaj Wierda]

We now have a development build available of 21.0.8 HF3 that should address the memory consumption issue.
We are looking for customers to confirm that this dev build addresses the issue they are experiencing, before we will release 21.0.8 HF3 

Please open a support ticket (or update your existing support ticket) to request this dev build.

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------

[Maria Joseph Vimalan]

Thank you for your feedback.

We are running AE on 21.0.5+hf.2.. Do you see any problem  updating the AWI to 21.08.HF3 only ? We dont want to run the upgrade and testing for the entire components.  We will plan for the full upgrade may be mid of next year 2024

Kindly suggest.

Thanks and Regards

Vimalan

[Michael Dolinek]

Hi @Maria Joseph Vimalan

be aware that the compatibility between different versions/service packs of AE or AWI are not tested/verified. Please make sure to keep the versions of all involved components in sync to avoid any unexpected side effects.

Michael

------------------------------
Michael K. Dolinek

Engineering Program Manager | Agile Operation Division
Broadcom Software
------------------------------

[Maria Joseph Vimalan]

Hi Michael,

Thank you for your feedback.  Since we are running quite lot of jobs , this would require extensive testing again if we update all the involved components and with the other parallel projects which are dependent on AE as well, we dont have an option to update the other components here.  

We had critical issues which took some SP's to be fixed in the past. 

Kindly guide what would be the expected side effects if AWI is running on Higher version compared to AE. It is easy for us to revert back the AWI version to older version compared to AE components. 

Regards
Vimalan

[Pete Wirfs]

Running a newer AWI works fine until it doesn't.  We have run on newer AWIs on multiple occasions, but we did have a situation once where the newer AWI required the use of a newer Automic database field we did not have, so we couldn't.

Automic will not support or certify using a newer AWI.  But if you are willing to take on the risk and do your own testing, then it is possible.  And as you have said, it is relatively easy to revert back.

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------