Bad performance after upgrade to 24.4.1HF1

[Bartosz Suchecki]

HFHi,

with the issue from title we already opened a Broadcom ticket but no satisfactory resolution was proposed so I have to ask the community.

We were using 21.0.7, and we upgraded to 24.4.1HF1 - after upgrade we saw a major decrease in performance visible in AWI and WP logs.

Issues on AWI:
- search takes long
- loading any objects takes long
- saving takes long
- opening executions of an object takes long
- switching folders in explorer takes long before objects are loaded
in general loading any tab/data takes long, be it process assembly, process monitoring or tabs in administration perspective

with right clicking it also takes some time before options are added (preparing actions...).

And I'm not talking about 0.5s waiting, but even tens of seconds to load anything... (problem varies between users, some have faster responses some slower).

But AWI reacts immediately when you click, just loading/saving data takes a long time

in WP logs we see a performance score of 11 for database, and with db traces active we see many critical operations (EXEC, SLCU and INSR) but their number wouldn't explain persistent problems with loading data in AWI.

we use Tomcat as our web interface. In tomcat config we have ajp 1.3 redirect for apache where we have proxypass configured.

Engine and database use RHEL 8.9, previous database was on the same server.
We don't see the same issue in lower environment, and all steps to create database and upgrade engine were identical.


Any ideas community?

-------------------------------------------

[Christoph Rekers at Tricise]

Hi Bartosch,

With so few details, hardly anyone will be able to help you. It would be interesting to know which database you're using, which collation/NLS_CHARACTERSET the database uses, AE on the new machine, java version database driver version etc.

And I'd say that nobody really feels like dragging those details out of you one by one...

No offense.

Cheers

Christoph 

------------------------------
--------------------------------------------------------
Automic Consultant and Trainer since 2000
--------------------------------------------------------
now Tricise
------------------------------

[Bogdan Mustiata]

Hi Bartosz,

First, sorry to hear that you're experiencing issues with our product's performance.

In my tests, I wasn't able to make it perform too badly, even by manually increasing the latency to 100ms per request. What's even more interesting is that in my tests, the v21 performed worse than v24. So I'm not sure where to go from here.

There's three possible explanations in my mind:

1.  Tomcat's configuration is wrong, so we have huge latency per each request to Tomcat and back.
2. There's latency between Tomcat and the JCP/REST endpoints.
3. There's latency between AE and the DB.

I would ask you to try the following:

1. Do a `ping` from the user's host to where Tomcat is running. Another from Tomcat to JCP/REST hosts. And another from JWP/WP to DB. This is just to find out what latency is in the network, I don't think that's the main reason for the performance degradation, but it's good to have a baseline.
2. Try to run a test AWI instance by just downloading the standalone Automic Web Interface JAR and running it:
   java -jar aa-webui-launcher.jar
   This will use an embedded Jetty. Does the same performance degradation occur as in your setup, too? If yes, we can rule out the Tomcat configuration as being the issue.

Now, assuming Tomcat's the issue, we'd need your Tomcat connector XML to try to replicate the issue. Please provide us with this information, and we'll proceed from there.

-------------------------------------------

[Bartosz Suchecki]

Hi community,

thanks for your input. We have resolved the issue. Core problem was with proxy and reverse proxy configuration in deployed apache that fetched data from tomcat. There's some kind of unknown incompatibility between our apache->tomcat after upgrading to tomcat 10.

Removing the apache from configuration and running tomcat only resolved all the problems.

Thanks

-------------------------------------------

[Thierry Tarrason]

Hi Bartosz,

We have a very similar experience. Fortunately, we were able to detect it before we upgraded our production.

Especially the right-click can take up to 10s before displaying the menu on some workflows ( but not all ).

After a lot of back and forth with the support, they told us that a fix might be available with the next hot fix, but it's not yet confirmed.

As for the technical details:
AE 24.4.1_hf1 runs on AIX with an Oracle DB
AWI runs in Tomcat 9 on a Red Hat 9. Jetty gives us the same performance issues.

Regards,
Thierry

-------------------------------------------

[Nicole Boeger]

Hi all,

we had performance issues when upgrading to V24 from V21, too. 

First of all, you need to keep your database running smoothly, do all the recommended database and index reorganization on the new engine before running in production.

Apache HTTPD redirecting to tomcat was an issue in our case, too. Support told us to switch on WebSockets on Apache HTTPD. This was not possible with our version of Apache HTTPD. Instead, we eliminated the HTTPD, and enabled our AWI Loadbalancer for WebSockets with sticky sessions supported. The AWI uses the JSESSIONID session cookie. You may check the documentation: https://docs.automic.com/documentation/webhelp/english/ALL/components/DOCU/24.4.0/Automic%20Automation%20Guides/Content/Installation_Manual/AWI/ConfiguringAWI.htm?Highlight=websocket#link22

------------------------------
Regards, Nicole
------------------------------

[Krum Ganev]

We also had some issues with the WebSockets but not with HTTPD.

It turns out that while previously WebSocket were only the recomended setting, now its a mandatory setting.

Broadcom did not notified anyone of that change nor did they included it to the release notes or incompatibilities. 

------------------------------
------------------------------
Automic SME @ DXC.Technology
------------------------------
------------------------------

[Thierry Tarrason]

Hi,

A new AWI 24.4.2_hf3 was made available. The first test results look promising. We are still testing.

BR,
Thierry

-------------------------------------------

[Peter Mordinyi]

Hi Thierry,

Do you have some news for me/us because of testing 24.4.2HF3?
Are there any new findings? Is it still running efficiently and without errors?

Thank you ver much
Regards
Peter

-------------------------------------------

[Thierry Tarrason]

Hi Peter,

I can confirm that 24..4.2_HF1 is much faster than any other V24 AWI we tried. We didn't encouter any big issues until now.

Regards,
Thierry

-------------------------------------------

[Peter Mordinyi]

Thank you very much

-------------------------------------------

[Christoph Rekers at Tricise]

Hi @Thierry Tarrason
Did you update the whole system or just the AWI?

Thanks

Christoph 

------------------------------
--------------------------------------------------------
Automic Consultant and Trainer since 2000
--------------------------------------------------------
now Tricise
------------------------------

[Thierry Tarrason]

Hi Christophe,

On our test environment, we updated only AWI. This was enough.

On our productive system, evrything was upgraded to 24.4.2_hf3 to keep the same version accros the board.

-------------------------------------------