While walking through the recent bookshelf notes to migrate the embedded JBOSS 5.1 GA IG EAR to Wildfly 8.x.x, I noticed that tuning of the JVM was not mentioned.
JBoss WildFly Installation - CA Identity Governance - 12.6.03 - CA Technologies Documentation
JBoss WildFly Installation - CA Identity Governance - 12.6.04 - CA Technologies Documentation
Since we are using Wildfly 8.x.x with Oracle JDK 8.x, we can leverage newer functionality available in JDK 8.
One area of interest is the initial memory (xms) and the maximum memory (xmx) of the JVM heap size. During the mirgration, the min/max is carried over from the default OOTB settings, which are small.
Another area of interest is the GC (aka garbage collection) routines available to later releases of JDK.
With JDK 8, the default GC is the Parallel GC. It is a good all around GC for most applications, but is it good enough to use for CA IG?
Notes about the top three (3) GC used in production environments.
•JVM’s default collector (-XX:+UseParallelGC )
•Will stop application threads when performing either a minor or full GC collection.
•Use for apps that can tolerate application pauses and are trying to optimize for lower CPU overhead caused by the collector
•CMS collector (-XX:+UseConcMarkSweepGC or -XX:+UseConcMarkSweepGC -XX:+UseParNewGC)
•Uses more CPU in order to provide the application with higher levels of continuous throughput, by using multiple threads to perform scanning and collection.
•Enhance for long-running server applications which are adverse to application freezes with less than heap of 4 Gb in size; using an additional flag: -XX:+UseParNewGC
•Garbage first collector (-XX:+UseG1GC or -XX:+UseG1GC -XX:+UseStringDeduplication )
•Supports heaps larger than 4GB
•Compacts the heap on-the-go
•New optimization enables the G1 collector to identify strings which are duplicated more than once across heap
•Avoid multiple copies of the same string from residing inefficiently within the heap.
To determine the max memory & best GC to use for IG, I setup a lab and recorded some basic values for starting up IG and executing some basic loads.
Here are the results:
Test One (1): Which version of Wildfly and JDK will work with IG migration scripts?
Answer: Wildfly 8.x.x and both JDK 7 & 8 using the non-clustered startup batch file.
Test Two (2): Which version of Wildfly and JDK will work with IG migration scripts?
Test Three (3): Of Wildfly 8.x.x and JDK 8, which is the best min/max memory settings?
Answer: See chart.
Test Four (4): Which GC will work best for IG on Wildfly 8.x.x and JDK 8?
Answer: Appears the latest GC is the best: G1GC or G1GC with the extra flag for strings.
The default GC of ParallelGC allowed for memory bloat.
The use of the G1GC with min/max memory settings has a strong value to performance.
Summary: I have added these steps to my dev-ops processes for new deployments of IG.
:: Step 05b - Update/Tune Wildfly 8 configuration files
:: Update three (3) bat/sh files under E:\Programs\CA\JBOSS\wildfly-8.2.1.Final\bin\
:: ca-gm-run-cluster.bat & ca-gm-run.bat & jboss-cli.bat
:: Add the line below to ensure Oracle JDK 8 to be used instead of JDK 6
:: set JAVA_HOME=E:\Programs\jdk\jdk1_8_0_77_x64
:: Update E:\Programs\CA\JBOSS\wildfly-8.2.1.Final\bin\standalone.conf.bat /.sh
:: Update max memory & GC to
:: set "JAVA_OPTS=-Xms1024M -Xmx8192M -XX:+UseG1GC -XX:+UseStringDeduplication"
:: Validate shutdown with jboss-cli.bat --connect command=:shutdown
:: Step 05c - Create NT Services
:: Copy E:\Programs\CA\JBOSS\wildfly-8.2.1.Final\bin\service\services.bat
:: Update three (3) naming fields to:
:: set SHORTNAME=CA_IG
:: set DISPLAYNAME="CA Identity Governance r12.6.5"
:: set DESCRIPTION="CA Identity Governance r12.6.5 on Wildfly r8.2.1 Application Server"
:: Replace standalone.bat with ca-gm-run-cluster.bat
:: Save & execute service.bat install
:: Open NT services, services.msc, and view/test
A view of using the new JVM flags when starting IG. Monitoring via MS Sysinternals Process Explorer for "java.exe"
After startup was finished, the CPU utilization dropped to less than 5%. The CPU heart-beat of the Java.exe was very steady. No more obvious peaks/valleys. Memory utilization would resize to immediate needs.
Curious if anyone else has collected metrics on IG performance they can share.
Have you used other JVM flags with Wildfly?
Loaded data from Active Directory with 120K accounts via the IG Connector with SSL/TLS cert.
Perform load testing and monitor performance.
Based on results, Increase OS memory to 16 GB, increased JVM to 12 GB.
Memory of IG (java): Upon Startup: 1.6 GB; Upon log into IG and select Universe with 120K accounts, memory jumps to 9.2 GB with CPU idle average < 5%. CPU will peak to >90% when selecting data, but drops down again.
Average Page load < 5 seconds.
Current JVM (JDK8) Settings (delta from defaults):
set "JAVA_OPTS=-Xms1G -Xmx12G -XX:+UseG1GC -XX:+UseStringDeduplication"