Does anyone see something in the below that I can try, I need to correct, or any additional logging that might help determine what might help work around this issue?
We are upgrading our Confluence to 6.1.0 and we currently are running CA APM 10.0.0.12.
--------------------------Server Information--------------------------Application Server: Apache Tomcat/8.0.41Servlet Version: 3.1
jvmVendor = Oracle CorporationworkingDirectory = E:\Atlassian\Confluence610javaVersion = 1.8.0_112
operatingSystem = Windows Server 2012 R2 6.3
We have two errors during startup in the Atlassian logs:
2017-06-16 10:25:40,629 ERROR [Caesium-1-4] [atlassian.scheduler.core.JobLauncher] launchAndBuildResponse Scheduled job with ID 'conversionLoader' failed due to binary incompatibilitiesjava.lang.NoClassDefFoundError: javax/print/PrintService at asposewobfuscated.zzGC.<init>(Unknown Source)
2017-06-16 10:27:24,791 WARN [localhost-startStop-6] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@ff07e58 took 8233 ms (warning threshold is 5000 ms)2017-06-16 10:27:24,899 ERROR [localhost-startStop-6] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exceptioncom.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149) at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
In the IntroscopeAgent log, turned to debug, we receive the following style errors:
6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info
and have created a skip file with the following directives:
SkipPackagePrefix: com.atlassian.confluenceSkipPackagePrefix: com.atlassian.pluginsSkipPackagePrefix: com.atlassian.pluginSkipPackagePrefix: org.hibernate.bootSkipPackagePrefix: com.sun.research.ws.wadlSkipPackagePrefix: com.google.common.collectSkipPackagePrefix: bucket.core.persistence.hibernate.schemaSkipPackagePrefix: com.atlassian.crowdSkipPackagePrefix: com.atlassian.eventSkipPackagePrefix: com.atlassian.hibernateSkipPackagePrefix: com.atlassian.vcacheSkipPackagePrefix: org.hibernate.internalSkipPackagePrefix: org.hibernate.metamodel
We have tried to adjust the bootloader sequence with the following within the Windows Java service wrapper:
But we are still getting the Atlassian errors and we are still seeing the following invalid class name errors in the introscopeagent.log
6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info6/16/17 10:25:47 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:com/sun/research/ws/wadl/package-info
Even with both were in the skip directives for package prefix.
CA Support Case:
Case Comment 00768835 - Atlassian Confluence - 6.1.0 - "Could not initialize class"
I see you have implemented the recommendations in this KB
Additionally, I recommend you to apply this other KB
Regarding the messages:
>> java.lang.NoClassDefFoundError: javax/print/PrintService
Try adding javax.print.* to the -Datlassian.org.osgi.framework.bootdelegation
>>2017-06-16 10:27:24,899 ERROR [localhost-startStop-6] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exception
I found the same stack trace here : https://community.atlassian.com/t5/Confluence-questions/Unable-to-create-a-tracker-when-osgi-is-not-running/qaq-p/263282
I am not sure if this is related to the agent, if possible, disable the agent entirely and check if the problem persists.
If the problem doesn't occur, enable back the agent jvm args but disable the instrumentation introscope.autoprobe.enable=false
>> In the IntroscopeAgent log, turned to debug, we receive the following style errors: 6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info
Most probably these are dynamically generated classes by JAXB framework and it simply the agent cannot find any valid classes to probe.
These shouldn't impact any functionality and can be ignored. They are not reported as ERROR messages.
To prevent the messages, you can switch from VERBOSE to INFO logging only
Check if the appserver and all the confluence plugins are working as expected.
Otherwise, you need to attach to the support case:
- Agent DEBUG logs
- Confluence logs
Make sure to enable verbose class loading, you need to add: -verbose:class
This information will help Sustaining Engineeering identify what additional jvm argument is required for your setup
Thank you Sergio.
- We have attempted to reorder the bootloader as prescribed within the JVM properties with no success.
- We are not having an issue with the file stream. We have #TurnOn: FileSystemTracing in our profile.
- There is no mention within the error stack of the OSGI framework
- When we turn the autoprobe off, Confluence starts without issue
With the autoprobe turned on Confluence will not start due to the
"Scheduled job with ID 'conversionLoader' failed due to binary incompatibilitiesjava.lang.NoClassDefFoundError: javax/print/PrintService"
From this, it appears that there is a serialized object, maybe JAXB, that is being reanimated and the class "PrintService" has been altered at the byte code level, more than likely by the autoprobe, which makes the class incompatible with the serialized object form.
We have attempted to skip the /javax/print/PrintService class, but that had no impact.
On review of the autoprobe log, have not been able to locate the actual parent, deep inheritance or dynamic instrumented class that might be causing the binary incompatibilities.
From case 00768835, issue was resolved by implementing the below changes:
1. Java boot delegation parameter:bootdelegation=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,javax.print.*,com.sun,com.sun.*,org.apache.xalan,org.apache.xalan.*,com.sun.jndi.*,com.icl.saxon,com.icl.saxon.*,javax.servlet,javax.servlet.*,com.sun.xml.bind.* -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
2. Disabled SmartInstrumentation
introscope.agent.deep.instrumentation.enabled=false introscope.agent.deep.trace.enabled=false introscope.agent.deep.entrypoint.enabled=false
3. Created a custom pbd with the lines to prevent messages related to the invalid classnames : [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class ...
SkipPackagePrefix: com.atlassian.confluence SkipPackagePrefix: com.atlassian.plugins SkipPackagePrefix: com.atlassian.plugin SkipPackagePrefix: org.hibernate.boot SkipPackagePrefix: com.sun.research.ws.wadl SkipPackagePrefix: com.google.common.collect SkipPackagePrefix: bucket.core.persistence.hibernate.schema SkipPackagePrefix: com.atlassian.crowd SkipPackagePrefix: com.atlassian.event SkipPackagePrefix: com.atlassian.hibernate SkipPackagePrefix: com.atlassian.vcache SkipPackagePrefix: org.hibernate.internal SkipPackagePrefix: org.hibernate.metamodel