java.lang.OutOfMemoryError: Java heap space

[David Richards]

Hi

We are experiencing some instability in out 10.7 cluster where some collectors are disconnecting and shutting down with this error being seen 

20200104 165750 GMT [ERROR] [Manager] Uncaught Exception in Enterprise Manager: In thread 'pool-16-thread-1' and the message is: java.lang.OutOfMemoryError: Java heap space

These are set up with 16GB heap, Java 1.8.

./jre/bin/java -Xms16110m -Xmx16110m -verbose:gc -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xss512k -Djava.io.tmpdir=./work com.zerog.lax.LAX /app/wily/Introscope/Introscope_Enterprise_Manager.lax /tmp/env.properties.2395


When we look in the perflog.txt file it looks like there is plenty of available heap so its a bit strange as I dont understand why the EM thinks all heap is gone.

Any ideas as to extra tuning we can do on the java parameters ?

[Amir Mushtaq]

Hi,

The system may be running out of memory. Check how much free memory is available around the time of OutOfMemoryError error. Also, you can configure the EM to get the heap dump on OutOfMemoryError. 
add -XX:+HeapDumpOnOutOfMemoryError Option in lax file. For details: https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html

Thanks, Amir

[David Richards]

Hi

The host has 64GB installed and at least 6GB free memory available 
I have added the DOOM flag to capture dumps going forward

All the affected hosts are showing GC duration > 1 second for extended periods so is there any tuning on the java params for this JVM (version 1.8)

[Amir Mushtaq]

Hi,

You can enable the GC log as follows: Enable GC log. Additional jvm switches will be required for this.
For Sun jvm, add the following jvm switches: -Xloggc:<filename>.log -XX:+PrintGCDetails

This TechDoc talks about JVM parameters for APM 10.7.
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=93572

Here are couple of TechDocs that give you details on APM Cluster Performance Health Check, and Enterprise Manager Troubleshooting and Best Practices.

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=9397
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=93176

I hope this helps.
Thanks, Amir

[Haruhiko Davis]

consider adding the following configuration:
lax.nl.java.option.additional=-Xms16g -Xmx16g -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 -XX:InitiatingHeapOccupancyPercent=45 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -Xss512k

Where:

• ParallelGCThreads equals the number of cores available to the EM
• ConcGCThreads equals 1/4 the number of cores available to the EM
• InitiatingHeapOccupancyPercent equal to 45% (grow or shrink as needed)

------------------------------
Custom Solution Architect
Broadcom
------------------------------

[David Richards]

Thank you Hiko - I will try this and let you know how we get on.

[Haruhiko Davis]

I should also mention that when using G1GC, it seems to ignore max heap (-Xmx) setting. The behavior looks like it will immediately double to the size of min (-Xms).

So depending on your hardware, you may want to consider starting at 8GB min heap. Also, don't forget to leave yourself at least 2GB overhead for the OS.

------------------------------
Custom Solution Architect
Broadcom
------------------------------

[David Richards]

Yeah I think you could be right, the 3 collectors we applied this change to are starting to die. One has disconnected and shut down in the last few minutes and the other 2 are struggling. I did start with min/max both set to 16GB so I have now tweaked that to 8gb min 16gb max. See what happens with that

[Haruhiko Davis]

Please let us know your results.

I have a customer running these configurations on both VMs and physical servers. Don't forget if you're stacking EMs on a large server to limit the number of processors available to each instance in the EM properties file so they are not stepping on each other for resources.

------------------------------
Custom Solution Architect
Broadcom
------------------------------

[David Richards]

Not good I'm afraid, just got another OOM crash 

Got this in the start up log

9904M->9904M(12G), 0.0920097 secs]
[Full GC (Allocation Failure) 9904M->9810M(12G), 9.3297564 secs]
[Full GC (Allocation Failure) 9810M->9809M(12G), 7.2157220 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/java_pid21791.hprof ...
[GC concurrent-mark-abort]
Dump file is incomplete: No space left on device

Got this in the EM log

20200110 122756 GMT [ERROR] [Manager] Uncaught Exception in Enterprise Manager: In thread 'pool-16-thread-1' and the message is: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:404)
at com.fasterxml.jackson.core.io.SegmentedStringWriter.getAndClear(SegmentedStringWriter.java:83)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3225)
at com.wily.introscope.appmap.events.CollectorEvents.<init>(CollectorEvents.java:53)
at com.wily.introscope.appmap.events.CollectorEventPublisherBean.eventsOccured(CollectorEventPublisherBean.java:126)
at com.wily.introscope.appmap.events.EventManagerImpl.relayEventsToConsumersInternal(EventManagerImpl.java:274)
at com.wily.introscope.appmap.events.EventManagerImpl.relayEventsToConsumers(EventManagerImpl.java:251)
at com.wily.introscope.appmap.events.EventManagerImpl.access$4(EventManagerImpl.java:244)
at com.wily.introscope.appmap.events.EventManagerImpl$1.run(EventManagerImpl.java:176)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
20200110 122756 GMT [INFO] [Manager] Beginning orderly shutdown...

Last entry in perflog doesnt show heap issues

20200110 122756 GMT [DEBUG] [Manager.Performance] 0,12288,5215,8,0,1143,960,8,636850,809819,603,0,703123,156,63,0,0,0,294,55852,548061
,0,43,0,0,0,0,

This is the process listing (after its been restarted)

./jre/bin/java -Xms6g -Xmx12g -verbose:gc -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 -XX:InitiatingHeapOccupancyPercent=45 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/ -Xss512k -Djava.io.tmpdir=./work com.zerog.lax.LAX /app/wily/Introscope/Introscope_Enterprise_Manager.lax /tmp/env.properties.8035

EM host has got 32GB installed, only running one EM

Output from top (after the restart)

Tasks: 217 total, 1 running, 216 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.4 us, 0.3 sy, 1.5 ni, 97.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32159468 total, 6774028 free, 12079012 used, 13306428 buff/cache
KiB Swap: 2097148 total, 2011388 free, 85760 used. 19668240 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8035 wilyadm 24 4 18.4g 10.9g 33380 S 13.0 35.4 4:18.21 java

[David Richards]

I have raised support case 20251224 for this issue now

[Haruhiko Davis]

I will have a look

------------------------------
Custom Solution Architect
Broadcom
------------------------------

[Haruhiko Davis]

How many cores are available on MOM and each collector?
How much free memory is available when the EM is not running?
Are you running APM DB and WebView with the MOM?

------------------------------
Custom Solution Architect
Broadcom
------------------------------