V10 Using more RES (Physical) Memory

[Rick Reed]

I have noticed as I setup our new Version 10 API Gateways (Appliance) under CentOS7 that they gateway process is using a TON more RES (physical) memory than our existing Version 9.4 API Gateways (Appliance) under CentOS6.

As you can see below TOP command results.  Our new V10s are using a large 16.8GB of RES (Physical) memory.  The same TOP command against our V94s are only using a fraction of that RES (Physical) at 4.8GB.

Is this a new Normal for the V10s or are there settings that I need to tweak for the process and/or the OS to get the RES value reduced?

We have 32GB / 8 cores but it just seems out of place for the process to be using so much more Physical memory.  That is over half the available memory.

Thoughts?

[Rick Reed]

I noticed that the NODE_OPTS for the Java processes within the /opt/SecureSpan/Gateway/runtime/etc/conf/appliancedefs.sh is quite a bit different from their V94 cousins.

In particular, I see where the -Xms value is set to the same value as the -Xmx value.  Is there a technical reason why the -Xms (minimum java heap memory) and the -Xmx (largest java heap memory) values are set to the same value?  For my configuration, this is setting the -Xms to right around 16.8GB which is what I was seeing in my Top command.

I have removed the -Xms setting from the appliancedefs.sh and restarted the service.  This has resolved the issue with the gateway process pulling over 16GB of RES memory but I wanted to confirm that this is recommended fix or whether there is a valid Technical reason why the V10 gateways need such a high value for the minimum java heap memory.

Thanks.

[Jay Thorne]

It was always supposed to be set with mx and ms the same. It was like that in Gateway versions 1 through 4. It changed for unknown reasons around version 7ish, as far as I remember. The original version of that startup was written by me in 2003, and I had a fair amount of data backing that choice up. This most recent change is part of an ongoing effort to be more overall intentional with what we provide. 

The gateway is known to perform better and more importantly more predictably under load if it has a single continugous allocation of RAM pre-allocated and not fragmented.

Rarely does the gateway process need 16 gigs of RAM. I also see no need to give the VM itself 32gbytes. I have docker deployments running just fine in 2gigs. If your system trims itself to  4.8, I'd suggest you set the entire VM to be a smaller allocation, like 12 gigs which will then allocate 6 gigs to the JVM. It calculates half of allocated RAM, which has worked well for most of the 17 year history of the product; the OS, various other processes, the process controller, mysql, disk IO buffers and cache take the rest. In a normal system you should see 6 gigs allocated to the JVM, around 1 gig allocated to MySQL, and the rest in buffers and cache, with a very small "Free" number, as Linux will use available RAM as cache and buffers and free it as needed. 

Normally I have a look at the gc logs from production before deciding on changing RAM allocation. 

Using a smaller total VM allocation, but reserved works better in VMWare where over-committed memory is a far too common situation. 

In VMWare the balloon driver can cause severe performance problems in our Gateway if the parent hypervisor attempts to take RAM away from the JVM. We recommend using less RAM, but using reserveratoin for the memory rather than doing partial allocation and making the JVM compete for RAM with other VMs on the same ESX host. This has been the cause of many escalations over the years. VMWare's memory documentation doesn't properly cover how the gateway product uses RAM, so our recommendation is to reserve the memory. 

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-2/reference/virtual-appliance-deployment-best-practices/summary-virtual-appliance-best-practices.html

------------------------------
Architect
Broadcom
------------------------------