Atlassian Confluence 6.1.0 - CA APM 10.0 Tomcat Agent

[Billy Cole]

Does anyone see something in the below that I can try, I need to correct, or any additional logging that might help determine what might help work around this issue?

 

 

We are upgrading our Confluence to 6.1.0 and we currently are running CA APM 10.0.0.12.

--------------------------
Server Information
--------------------------
Application Server: Apache Tomcat/8.0.41
Servlet Version: 3.1

jvmVendor = Oracle Corporation
workingDirectory = E:\Atlassian\Confluence610
javaVersion = 1.8.0_112

operatingSystem = Windows Server 2012 R2 6.3

 

We have two errors during startup in the Atlassian logs:

 

2017-06-16 10:25:40,629 ERROR [Caesium-1-4] [atlassian.scheduler.core.JobLauncher] launchAndBuildResponse Scheduled job with ID 'conversionLoader' failed due to binary incompatibilities
java.lang.NoClassDefFoundError: javax/print/PrintService
at asposewobfuscated.zzGC.<init>(Unknown Source)

 

2017-06-16 10:27:24,791 WARN [localhost-startStop-6] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@ff07e58 took 8233 ms (warning threshold is 5000 ms)
2017-06-16 10:27:24,899 ERROR [localhost-startStop-6] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exception
com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)

 

In the IntroscopeAgent log, turned to debug, we receive the following style errors:

6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info

 

and have created a skip file with the following directives:

SkipPackagePrefix: com.atlassian.confluence
SkipPackagePrefix: com.atlassian.plugins
SkipPackagePrefix: com.atlassian.plugin
SkipPackagePrefix: org.hibernate.boot
SkipPackagePrefix: com.sun.research.ws.wadl
SkipPackagePrefix: com.google.common.collect
SkipPackagePrefix: bucket.core.persistence.hibernate.schema
SkipPackagePrefix: com.atlassian.crowd
SkipPackagePrefix: com.atlassian.event
SkipPackagePrefix: com.atlassian.hibernate
SkipPackagePrefix: com.atlassian.vcache
SkipPackagePrefix: org.hibernate.internal
SkipPackagePrefix: org.hibernate.metamodel

 

We have tried to adjust the bootloader sequence with the following within the Windows Java service wrapper:

 

-Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
-Datlassian.org.osgi.framework.bootdelegation=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,com.sun,com.sun.*org.apache.xalan,org.apache.xalan.*
-Dcom.wily.introscope.agentProfile="e:\\atlassian\\confluence610\\wily\\core\\config\\IntroscopeAgent.profile"
-javaagent:"e:\\atlassian\\confluence610\\wily\\Agent.jar"

 

But we are still getting the Atlassian errors and we are still seeing the following invalid class name errors in the introscopeagent.log

6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info
6/16/17 10:25:47 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:com/sun/research/ws/wadl/package-info

 

Even with both were in the skip directives for package prefix.

 

CA Support Case:

Case Comment 00768835 - Atlassian Confluence - 6.1.0 - "Could not initialize class"

 

 

 

Thank you,

 

Billy

[Sergio Morales Correa]

Hi Billy

I see you have implemented the recommendations in this KB

https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1601606.html

 

Additionally, I recommend you to apply this other KB

https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1387785.html

 

Regarding the messages:

 

>> java.lang.NoClassDefFoundError: javax/print/PrintService

 

Try adding javax.print.* to the -Datlassian.org.osgi.framework.bootdelegation

 

>>2017-06-16 10:27:24,899 ERROR [localhost-startStop-6] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exception

 

I found the same stack trace here : https://community.atlassian.com/t5/Confluence-questions/Unable-to-create-a-tracker-when-osgi-is-not-running/qaq-p/263282

I am not sure if this is related to the agent, if possible, disable the agent entirely and check if the problem persists.

If the problem doesn't occur, enable back the agent jvm args but disable the instrumentation introscope.autoprobe.enable=false

 

>> In the IntroscopeAgent log, turned to debug, we receive the following style errors: 6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info

 

Most probably these are dynamically generated classes by JAXB framework and it simply the agent cannot find any valid classes to probe.

These shouldn't impact any functionality and can be ignored. They are not reported as ERROR messages.

To prevent the messages, you can switch from VERBOSE to INFO logging only

 

Check if the appserver and all the confluence plugins are working as expected.

Otherwise, you need to attach to the support case:

- Agent DEBUG logs

- Confluence logs

Make sure to enable verbose class loading, you need to add:  -verbose:class

 

This information will help Sustaining Engineeering identify what additional jvm argument is required for your setup

 

Regards,

Sergio

[Billy Cole]

Thank you Sergio.

 

- We have attempted to reorder the bootloader as prescribed within the JVM properties with no success.

- We are not having an issue with the file stream.  We have #TurnOn: FileSystemTracing in our profile.

- There is no mention within the error stack of the OSGI framework

 

- When we turn the autoprobe off, Confluence starts without issue

 

With the autoprobe turned on Confluence will not start due to the 

 

"Scheduled job with ID 'conversionLoader' failed due to binary incompatibilities
java.lang.NoClassDefFoundError: javax/print/PrintService"

 

From this, it appears that there is a serialized object, maybe JAXB, that is being reanimated and the class "PrintService" has been altered at the byte code level, more than likely by the autoprobe, which makes the class incompatible with the serialized object form.

 

We have attempted to skip the /javax/print/PrintService class, but that had no impact.  

 

On review of the autoprobe log, have not been able to locate the actual parent, deep inheritance or dynamic instrumented class that might be causing the binary incompatibilities.

 

Thank you,

 

Billy

[Sergio Morales Correa]

FYI --

From case 00768835, issue was resolved by implementing the below changes:

 

1. Java boot delegation parameter:
bootdelegation=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,javax.print.*,com.sun,com.sun.*,org.apache.xalan,org.apache.xalan.*,com.sun.jndi.*,com.icl.saxon,com.icl.saxon.*,javax.servlet,javax.servlet.*,com.sun.xml.bind.* -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl

 

2. Disabled SmartInstrumentation

introscope.agent.deep.instrumentation.enabled=false
introscope.agent.deep.trace.enabled=false
introscope.agent.deep.entrypoint.enabled=false

 

3. Created a custom pbd with the lines to prevent messages related to the invalid classnames : [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class ...

SkipPackagePrefix: com.atlassian.confluence
SkipPackagePrefix: com.atlassian.plugins
SkipPackagePrefix: com.atlassian.plugin
SkipPackagePrefix: org.hibernate.boot
SkipPackagePrefix: com.sun.research.ws.wadl
SkipPackagePrefix: com.google.common.collect
SkipPackagePrefix: bucket.core.persistence.hibernate.schema
SkipPackagePrefix: com.atlassian.crowd
SkipPackagePrefix: com.atlassian.event
SkipPackagePrefix: com.atlassian.hibernate
SkipPackagePrefix: com.atlassian.vcache
SkipPackagePrefix: org.hibernate.internal
SkipPackagePrefix: org.hibernate.metamodel

 

Regards,

Sergio