DX Application Performance Management

Expand all | Collapse all

Atlassian Confluence 6.1.0 - CA APM 10.0 Tomcat Agent

  • 1.  Atlassian Confluence 6.1.0 - CA APM 10.0 Tomcat Agent

    Posted 06-16-2017 01:09 PM

    Does anyone see something in the below that I can try, I need to correct, or any additional logging that might help determine what might help work around this issue?

     

     

    We are upgrading our Confluence to 6.1.0 and we currently are running CA APM 10.0.0.12.

    --------------------------
    Server Information
    --------------------------
    Application Server: Apache Tomcat/8.0.41
    Servlet Version: 3.1

    jvmVendor = Oracle Corporation
    workingDirectory = E:\Atlassian\Confluence610
    javaVersion = 1.8.0_112

    operatingSystem = Windows Server 2012 R2 6.3

     

    We have two errors during startup in the Atlassian logs:

     

    2017-06-16 10:25:40,629 ERROR [Caesium-1-4] [atlassian.scheduler.core.JobLauncher] launchAndBuildResponse Scheduled job with ID 'conversionLoader' failed due to binary incompatibilities
    java.lang.NoClassDefFoundError: javax/print/PrintService
    at asposewobfuscated.zzGC.<init>(Unknown Source)

     

    2017-06-16 10:27:24,791 WARN [localhost-startStop-6] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@ff07e58 took 8233 ms (warning threshold is 5000 ms)
    2017-06-16 10:27:24,899 ERROR [localhost-startStop-6] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exception
    com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
    at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
    at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)

     

    In the IntroscopeAgent log, turned to debug, we receive the following style errors:

    6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info

     

    and have created a skip file with the following directives:

    SkipPackagePrefix: com.atlassian.confluence
    SkipPackagePrefix: com.atlassian.plugins
    SkipPackagePrefix: com.atlassian.plugin
    SkipPackagePrefix: org.hibernate.boot
    SkipPackagePrefix: com.sun.research.ws.wadl
    SkipPackagePrefix: com.google.common.collect
    SkipPackagePrefix: bucket.core.persistence.hibernate.schema
    SkipPackagePrefix: com.atlassian.crowd
    SkipPackagePrefix: com.atlassian.event
    SkipPackagePrefix: com.atlassian.hibernate
    SkipPackagePrefix: com.atlassian.vcache
    SkipPackagePrefix: org.hibernate.internal
    SkipPackagePrefix: org.hibernate.metamodel

     

    We have tried to adjust the bootloader sequence with the following within the Windows Java service wrapper:

     

    -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
    -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
    -Datlassian.org.osgi.framework.bootdelegation=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,com.sun,com.sun.*org.apache.xalan,org.apache.xalan.*
    -Dcom.wily.introscope.agentProfile="e:\\atlassian\\confluence610\\wily\\core\\config\\IntroscopeAgent.profile"
    -javaagent:"e:\\atlassian\\confluence610\\wily\\Agent.jar"

     

    But we are still getting the Atlassian errors and we are still seeing the following invalid class name errors in the introscopeagent.log

    6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info
    6/16/17 10:25:47 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:com/sun/research/ws/wadl/package-info

     

    Even with both were in the skip directives for package prefix.

     

    CA Support Case:

    Case Comment 00768835 - Atlassian Confluence - 6.1.0 - "Could not initialize class"

     

     

     

    Thank you,

     

    Billy



  • 2.  Re: Atlassian Confluence 6.1.0 - CA APM 10.0 Tomcat Agent

    Posted 06-20-2017 07:48 AM

    Hi Billy

    I see you have implemented the recommendations in this KB

    https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1601606.html

     

    Additionally, I recommend you to apply this other KB

    https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1387785.html

     

    Regarding the messages:

     

    >> java.lang.NoClassDefFoundError: javax/print/PrintService

     

    Try adding javax.print.* to the -Datlassian.org.osgi.framework.bootdelegation

     

    >>2017-06-16 10:27:24,899 ERROR [localhost-startStop-6] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exception

     

    I found the same stack trace here : https://community.atlassian.com/t5/Confluence-questions/Unable-to-create-a-tracker-when-osgi-is-not-running/qaq-p/263282

    I am not sure if this is related to the agent, if possible, disable the agent entirely and check if the problem persists.

    If the problem doesn't occur, enable back the agent jvm args but disable the instrumentation introscope.autoprobe.enable=false

     

    >> In the IntroscopeAgent log, turned to debug, we receive the following style errors: 6/16/17 10:23:23 AM EDT [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class name:org/hibernate/boot/jaxb/hbm/spi/package-info

     

    Most probably these are dynamically generated classes by JAXB framework and it simply the agent cannot find any valid classes to probe.

    These shouldn't impact any functionality and can be ignored. They are not reported as ERROR messages.

    To prevent the messages, you can switch from VERBOSE to INFO logging only

     

    Check if the appserver and all the confluence plugins are working as expected.

    Otherwise, you need to attach to the support case:

    - Agent DEBUG logs

    - Confluence logs

    Make sure to enable verbose class loading, you need to add:  -verbose:class

     

    This information will help Sustaining Engineeering identify what additional jvm argument is required for your setup

     

    Regards,

    Sergio



  • 3.  Re: Atlassian Confluence 6.1.0 - CA APM 10.0 Tomcat Agent

    Posted 07-05-2017 09:12 AM

    Thank you Sergio.

     

    - We have attempted to reorder the bootloader as prescribed within the JVM properties with no success.

    - We are not having an issue with the file stream.  We have #TurnOn: FileSystemTracing in our profile.

    - There is no mention within the error stack of the OSGI framework

     

    - When we turn the autoprobe off, Confluence starts without issue

     

    With the autoprobe turned on Confluence will not start due to the 

     

    "Scheduled job with ID 'conversionLoader' failed due to binary incompatibilities
    java.lang.NoClassDefFoundError: javax/print/PrintService"

     

    From this, it appears that there is a serialized object, maybe JAXB, that is being reanimated and the class "PrintService" has been altered at the byte code level, more than likely by the autoprobe, which makes the class incompatible with the serialized object form.

     

    We have attempted to skip the /javax/print/PrintService class, but that had no impact.  

     

    On review of the autoprobe log, have not been able to locate the actual parent, deep inheritance or dynamic instrumented class that might be causing the binary incompatibilities.

     

    Thank you,

     

    Billy



  • 4.  Re: Atlassian Confluence 6.1.0 - CA APM 10.0 Tomcat Agent

    Posted 09-05-2017 09:42 AM

    FYI --

    From case 00768835, issue was resolved by implementing the below changes:

     

    1. Java boot delegation parameter:
    bootdelegation=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,javax.print.*,com.sun,com.sun.*,org.apache.xalan,org.apache.xalan.*,com.sun.jndi.*,com.icl.saxon,com.icl.saxon.*,javax.servlet,javax.servlet.*,com.sun.xml.bind.* -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl

     

    2. Disabled SmartInstrumentation

    introscope.agent.deep.instrumentation.enabled=false
    introscope.agent.deep.trace.enabled=false
    introscope.agent.deep.entrypoint.enabled=false

     

    3. Created a custom pbd with the lines to prevent messages related to the invalid classnames : [VERBOSE] [IntroscopeAgent.Agent] Skipping invalid class ...


    SkipPackagePrefix: com.atlassian.confluence
    SkipPackagePrefix: com.atlassian.plugins
    SkipPackagePrefix: com.atlassian.plugin
    SkipPackagePrefix: org.hibernate.boot
    SkipPackagePrefix: com.sun.research.ws.wadl
    SkipPackagePrefix: com.google.common.collect
    SkipPackagePrefix: bucket.core.persistence.hibernate.schema
    SkipPackagePrefix: com.atlassian.crowd
    SkipPackagePrefix: com.atlassian.event
    SkipPackagePrefix: com.atlassian.hibernate
    SkipPackagePrefix: com.atlassian.vcache
    SkipPackagePrefix: org.hibernate.internal
    SkipPackagePrefix: org.hibernate.metamodel

     

    Regards,

    Sergio