Not good I'm afraid, just got another OOM crash
Got this in the start up log
9904M->9904M(12G), 0.0920097 secs]
[Full GC (Allocation Failure) 9904M->9810M(12G), 9.3297564 secs]
[Full GC (Allocation Failure) 9810M->9809M(12G), 7.2157220 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/java_pid21791.hprof ...
[GC concurrent-mark-abort]
Dump file is incomplete: No space left on device
Got this in the EM log
20200110 122756 GMT [ERROR] [Manager] Uncaught Exception in Enterprise Manager: In thread 'pool-16-thread-1' and the message is: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:404)
at com.fasterxml.jackson.core.io.SegmentedStringWriter.getAndClear(SegmentedStringWriter.java:83)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3225)
at com.wily.introscope.appmap.events.CollectorEvents.<init>(CollectorEvents.java:53)
at com.wily.introscope.appmap.events.CollectorEventPublisherBean.eventsOccured(CollectorEventPublisherBean.java:126)
at com.wily.introscope.appmap.events.EventManagerImpl.relayEventsToConsumersInternal(EventManagerImpl.java:274)
at com.wily.introscope.appmap.events.EventManagerImpl.relayEventsToConsumers(EventManagerImpl.java:251)
at com.wily.introscope.appmap.events.EventManagerImpl.access$4(EventManagerImpl.java:244)
at com.wily.introscope.appmap.events.EventManagerImpl$1.run(EventManagerImpl.java:176)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
20200110 122756 GMT [INFO] [Manager] Beginning orderly shutdown...
Last entry in perflog doesnt show heap issues
20200110 122756 GMT [DEBUG] [Manager.Performance] 0,12288,5215,8,0,1143,960,8,636850,809819,603,0,703123,156,63,0,0,0,294,55852,548061
,0,43,0,0,0,0,
This is the process listing (after its been restarted)
./jre/bin/java -Xms6g -Xmx12g -verbose:gc -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 -XX:InitiatingHeapOccupancyPercent=45 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/ -Xss512k -Djava.io.tmpdir=./work com.zerog.lax.LAX /app/wily/Introscope/Introscope_Enterprise_Manager.lax /tmp/env.properties.8035
EM host has got 32GB installed, only running one EM
Output from top (after the restart)
Tasks: 217 total, 1 running, 216 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.4 us, 0.3 sy, 1.5 ni, 97.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32159468 total, 6774028 free, 12079012 used, 13306428 buff/cache
KiB Swap: 2097148 total, 2011388 free, 85760 used. 19668240 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8035 wilyadm 24 4 18.4g 10.9g 33380 S 13.0 35.4 4:18.21 java
Original Message:
Sent: 01-09-2020 04:27 PM
From: Haruhiko Davis
Subject: java.lang.OutOfMemoryError: Java heap space
Please let us know your results.
I have a customer running these configurations on both VMs and physical servers. Don't forget if you're stacking EMs on a large server to limit the number of processors available to each instance in the EM properties file so they are not stepping on each other for resources.
------------------------------
Custom Solution Architect
Broadcom
Original Message:
Sent: 01-09-2020 03:15 PM
From: David Richards
Subject: java.lang.OutOfMemoryError: Java heap space
Yeah I think you could be right, the 3 collectors we applied this change to are starting to die. One has disconnected and shut down in the last few minutes and the other 2 are struggling. I did start with min/max both set to 16GB so I have now tweaked that to 8gb min 16gb max. See what happens with that
Original Message:
Sent: 01-09-2020 02:07 PM
From: Haruhiko Davis
Subject: java.lang.OutOfMemoryError: Java heap space
I should also mention that when using G1GC, it seems to ignore max heap (-Xmx) setting. The behavior looks like it will immediately double to the size of min (-Xms).
So depending on your hardware, you may want to consider starting at 8GB min heap. Also, don't forget to leave yourself at least 2GB overhead for the OS.
------------------------------
Custom Solution Architect
Broadcom
Original Message:
Sent: 01-08-2020 06:34 AM
From: David Richards
Subject: java.lang.OutOfMemoryError: Java heap space
Thank you Hiko - I will try this and let you know how we get on.
Original Message:
Sent: 01-08-2020 05:26 AM
From: Haruhiko Davis
Subject: java.lang.OutOfMemoryError: Java heap space
consider adding the following configuration:
lax.nl.java.option.additional=-Xms16g -Xmx16g -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 -XX:InitiatingHeapOccupancyPercent=45 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -Xss512k
Where:
- ParallelGCThreads equals the number of cores available to the EM
- ConcGCThreads equals 1/4 the number of cores available to the EM
- InitiatingHeapOccupancyPercent equal to 45% (grow or shrink as needed)
------------------------------
Custom Solution Architect
Broadcom
Original Message:
Sent: 01-06-2020 04:46 AM
From: David Richards
Subject: java.lang.OutOfMemoryError: Java heap space
Hi
We are experiencing some instability in out 10.7 cluster where some collectors are disconnecting and shutting down with this error being seen
20200104 165750 GMT [ERROR] [Manager] Uncaught Exception in Enterprise Manager: In thread 'pool-16-thread-1' and the message is: java.lang.OutOfMemoryError: Java heap space
These are set up with 16GB heap, Java 1.8.
./jre/bin/java -Xms16110m -Xmx16110m -verbose:gc -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xss512k -Djava.io.tmpdir=./work com.zerog.lax.LAX /app/wily/Introscope/Introscope_Enterprise_Manager.lax /tmp/env.properties.2395
When we look in the perflog.txt file it looks like there is plenty of available heap so its a bit strange as I dont understand why the EM thinks all heap is gone.
Any ideas as to extra tuning we can do on the java parameters ?