DX Application Performance Management

 View Only
  • 1.  Dynamic Instrumentation with Exception data java.lang.IncompatibleClassChangeError

    Posted May 28, 2015 12:26 PM

    Hi all folks,

    I try to play with tracing and additional dynamic Instrumentation (based on Websphere 7.0 and Introscope Agent 9.5.2) and got something like this

     

    [27.05.15 18:29:14:785 CEST] 00000022 LocalExceptio E   CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "onMessage" on bean "BeanId(rops#PCEServer.jar#PCEJmsReceiver, null)". Exception data: java.lang.IncompatibleClassChangeError

            at com.myproclassic.common.tservices.tracelog.aspects.PCEAbstractMethodEntryExitTraceAspect.ajc$before$com_myproclassic_common_tservices_tracelog_aspects_PCEAbstractMethodEntryExitTraceAspect$4$be26a676(PCEAbstractMethodEntryExitTraceAspect.aj:140)

            at com.myproclassic.server.services.jmsevents.beans.PCEJmsReceiverBean.onMessage(PCEJmsReceiverBean.java:58)

            at com.ibm.ejs.container.MessageEndpointHandler.invokeMdbMethod(MessageEndpointHandler.java:1093)

            at com.ibm.ejs.container.MessageEndpointHandler.invoke(MessageEndpointHandler.java:778)

            at com.sun.proxy.$Proxy33.onMessage(Unknown Source)

            at com.ibm.ws.sib.api.jmsra.impl.JmsJcaEndpointInvokerImpl.invokeEndpoint(JmsJcaEndpointInvokerImpl.java:233)

            at com.ibm.ws.sib.ra.inbound.impl.SibRaDispatcher.dispatch(SibRaDispatcher.java:918)

            at com.ibm.ws.sib.ra.inbound.impl.SibRaSingleProcessListener$SibRaWork.run(SibRaSingleProcessListener.java:552)

            at com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:399)

            at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)

     

    a short time later, the Application seems to work, but the exception was written in the log many times a minute.

    My problem is, all Applications are running on the System (AIX Lpar) start to write this exception, not the traced/instrumented Application only.

     

    Any idea what's going wrong,

    Lutz



  • 2.  Re: Dynamic Instrumentation with Exception data java.lang.IncompatibleClassChangeError

    Broadcom Employee
    Posted May 28, 2015 04:57 PM

    Hi Lutz,

     

    Oracle describes this error as "a class or interface has changed":

    IncompatibleClassChangeError (Java Platform SE 7 )

    There might be different versions of the same class in the class path and the wrong one is used.

     

    Does this happen only with the Introscope agent or also without Introscope (remove -javaagent parameter)? I don't see a com.wily.* class in the above stack trace.

     

    For a longer explanation and possible solutions see:

    http://stackoverflow.com/questions/1980452/what-causes-java-lang-incompatibleclasschangeerror

     

    There is also a possible workaround/solution for WAS:

    In this case, the solution was to change the classpath order of the module in my war file. That is, open up the war application in the WAS console under and select the client module(s). In the module configuration, set the class-loading to be "parent last".

    This is found in the WAS console:

    • Applications -> Application Types -> WebSphere Enterprise Applications
    • Click link representing your application (war)
    • Click "Manage Modules" under "Modules" section
    • Click link for the underlying module(s)
    • Change "Class loader order" to be "(parent last)".

    Ciao,

    Guenter



  • 3.  Re: Dynamic Instrumentation with Exception data java.lang.IncompatibleClassChangeError

    Posted May 29, 2015 01:44 PM

    Hi, yes, this is the problem,

    but I got this problem a short time after I start playing with dynamic instrumentation in all applications are running on the system and on other systems connected to the same MoM too.

    Oracle describes this error as "a class or interface has changed":

    There might be different versions of the same class in the class path and the wrong one is used.

    In general we use Introscope 9.5.2 with WAS 7.0 and WAS 8.5 since some month in this environment without any problem.

    And after we restart all applications (several are crashed) the problem was gone.

    My problem is to understand what's going wrong. We try do reproduce the problem in the evening, but every thing works well, we got no problems.

     

    Does someone know how Introscope add new dynamic instrumentations to the JVMs and what's going on if I doesn't select an agent in the trace menu. In my opinon the instrumentation changes are configured via the trace menu are sent to one application only, it doesn't affect all JVMs.

     

    Any idea, suggestion,

    Lutz



  • 4.  Re: Dynamic Instrumentation with Exception data java.lang.IncompatibleClassChangeError

    Broadcom Employee
    Posted May 29, 2015 02:40 PM

    Hi,

     

    most probably you have duplicate classes with different versions in the classpath.

     

    If I interpret the above error message correctly then com.myproclassic.common.tservices.tracelog.aspects.PCEAbstractMethodEntryExitTraceAspect expects a different method signature method "onMessage" of class/interface PCEJmsReceiver in PCEServer.jar.

     

    I don't see a direct connection to Introscope or dynamic instrumentation in particular. It may have surfaced because the classes were loaded in a different order because of Introscope's Smart Instrumentation if you have enabled that.

     

    The best advice I have is taken again from What causes java.lang.IncompatibleClassChangeError? - Stack Overflow:

    To resolve these, try launching the VM with -verbose as an argument, then look at the classes that were being loaded when the exception occurs. You should see some surprising information. For instance, having multiple copies of the same dependency and versions you never expected or would have accepted if you knew they were being included.

    write a script that searches inside all the jar files looking for duplicated classes (search by their full-qualified name, i.e with package name)

    Ciao,

    Guenter



  • 5.  Re: Dynamic Instrumentation with Exception data java.lang.IncompatibleClassChangeError

    Posted May 29, 2015 06:47 PM

    Hi, ok.

    I try to do this next week (at tuesday I'm back in the office).

    Nice suggestion, but the application is running since some month without problems.

    But I talk to the folks.

    Bye, I'll be back next week, Lutz



  • 6.  Re: Dynamic Instrumentation with Exception data java.lang.IncompatibleClassChangeError

    Broadcom Employee
    Posted Aug 03, 2015 03:23 PM

    Hi:

    I am marking this as assumed answered you did not request a follow-up over some time. Please open a new question with updated information if needing additional assistance.

      

    Thanks

    Hal German

    APM Support