DX Application Performance Management

 View Only
Expand all | Collapse all

WebSphere Application Fails to start

Luca Razzi

Luca RazziOct 10, 2016 11:53 AM

  • 1.  WebSphere Application Fails to start

    Posted Oct 10, 2016 08:43 AM

    Hello Community

    With Introscope Agent 10.3 installed I cannot start the WebSphere application server with the following error

    [10/10/16 12:55:37:161 CEST] 00000071 AbstractInjec E   CWNEN0009E: The injection engine failed to process the XML code from the deployment descriptor due to the following error: java.lang.NoClassDefFoundError:

     

    Removing the agent i have a successful start of the was application server

     

    java version is

    java version "1.6.0"
    Java(TM) SE Runtime Environment (build pxa6460_26sr8fp20-20160111_01(SR8 FP20))
    IBM J9 VM (build 2.6, JRE 1.6.0 Linux amd64-64 Compressed References 20151222_283040 (JIT enabled, AOT enabled)
    J9VM - R26_Java626_SR8_20151222_1616_B283040
    JIT - tr.r11_20151209_107111.01
    GC - R26_Java626_SR8_20151222_1616_B283040_CMPRSS
    J9CL - 20151222_283040)
    JCL - 20160104_01

     

    WAS release is 8.5.5.9 

     

    Can you help me?

    On the same node I have 3 other application server that are correctly monitored

    Thank you

     

    Luca Razzi



  • 2.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 09:18 AM

    I made a mistake

     

    java version "1.7.0"
    Java(TM) SE Runtime Environment (build pxa6470_27sr3fp30-20160112_01(SR3 FP30))
    IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20151222_283006 (JIT enabled, AOT enabled)
    J9VM - R27_Java727_SR3_20151222_1037_B283006
    JIT - tr.r13.java.sr3.fp30_20151209_107107.01
    GC - R27_Java727_SR3_20151222_1037_B283006_CMPRSS
    J9CL - 20151222_283006)
    JCL - 20160112_01 based on Oracle jdk7u95-b13

    thanks



  • 3.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 11:24 AM

    Can you post the stack trace after the java.lang.NoClassDefFoundError:?

     

    Also what are the parameters you used for the agent?



  • 4.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 11:32 AM

    [10/10/16 12:55:37:181 CEST] 00000072 AbstractInjec E CWNEN0009E: The injection engine failed to process the XML code from the deployment descriptor due to the following error: java.lang.NoClassDefFoundError: it.finanze.security.UnauthenticatedUserException
    at java.lang.J9VMInternals.prepareClassImpl(Native Method)
    at java.lang.J9VMInternals.prepare(J9VMInternals.java:283)
    at java.lang.Class.getDeclaredMethods(Class.java:994)
    at com.ibm.wsspi.injectionengine.InjectionBinding.getDeclaredSetMethods(InjectionBinding.java:292)
    at com.ibm.wsspi.injectionengine.InjectionBinding.addInjectionTarget(InjectionBinding.java:436)
    at com.ibm.ws.ejb.injection.processor.EJBProcessor.processXML(EJBProcessor.java:202)
    at com.ibm.ws.ejb.injection.processor.EJBProcessor.processXML(EJBProcessor.java:150)
    at com.ibm.ws.injectionengine.AbstractInjectionEngine.processXML(AbstractInjectionEngine.java:770)
    at com.ibm.ws.injectionengine.AbstractInjectionEngine.processInjectionMetaData(AbstractInjectionEngine.java:537)
    at com.ibm.ws.injectionengine.SharedInjectionEngineImpl.processInjectionMetaData(SharedInjectionEngineImpl.java:229)
    at com.ibm.ws.injectionengine.ReferenceContextImpl.processImpl(ReferenceContextImpl.java:789)
    at com.ibm.ws.injectionengine.ReferenceContextImpl.process(ReferenceContextImpl.java:390)
    at com.ibm.ws.runtime.component.EJBContainerImpl.start(EJBContainerImpl.java:3564)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1178)
    at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1383)
    at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
    at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:972)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:777)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2197)
    at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5486)
    at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5612)
    at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2202)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
    at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
    at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
    Caused by: java.lang.ClassNotFoundException: it.finanze.security.UnauthenticatedUserException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:600)
    at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:243)
    at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:786)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:764)
    at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:134)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
    at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:62)
    at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:58)
    at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:585)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
    at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:585)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
    ... 30 more

     

     

     

    You need parameters into the agentprofile??



  • 5.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 11:34 AM

    this is what we monitor with the agent

    introscope.autoprobe.directivesFile=websphere-typical.pbl,hotdeploy



  • 6.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 11:43 AM

    What are the startup parameters with the agent.jar file?

     

    The stack trace indicates it cannot find actual java classes.  

     

    Is this a fresh install?  Or was WAS or the agent upgraded?  If an upgrade, was the WAS cache cleared?



  • 7.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 11:52 AM

    String to startup WAS

     

    -javaagent:/prod/CA/APM10.3/wily/Agent.jar -Dcom.wily.introscope.agentProfile=/prod/CA/APM10.3/wily/core/config/IntroscopeAgent.profile



  • 8.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 11:59 AM

    Did you setup a startup class for WebAppSupport.jar or add this jar to the application classpath?



  • 9.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 11:53 AM

    Yes this is a fresh install



  • 10.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 11:57 AM

    A fresh install of Websphere?  

     

    Ok, so to really narrow down if its our agent causing this, disable instrumentation by turning off autoprobe in the IntroscopeAgent.profile and then restart Websphere.  If the same issue happens, then its a problem within Websphere and the presence of our agent.jar is just exposing the bug.



  • 11.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 12:16 PM

    Setting autoprobe to false we have the agent connected to the MOM

    The custom service with WebAppSupport.jar is correctly configured



  • 12.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 12:27 PM

    Do you have any custom pbd's or custom tracers defined in hotdeploy or in any of the existing pbd's?  



  • 13.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 12:32 PM

    Also, you said this was a fresh install.  Was it a fresh install of Websphere?  Or has Websphere existed on the server and the agent freshly installed?  Was java upgraded recently on this version of Websphere?



  • 14.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 12:33 PM

    No.

    This agent is the one CA provides from the Download site. I just added the MOM IP. No more



  • 15.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 12:36 PM

    I have to ask WAS people if it was upgraded

    WAS is 

    WebSphere Integrated Solutions Console

    Release 8.5.5.9


  • 16.  Re: WebSphere Application Fails to start

    Posted Oct 10, 2016 12:42 PM

    Also the process is to install FIX pack to reach 8.5.5.9.

    So the upgrade of the JDK must be done using the fix pack process



  • 17.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 10, 2016 12:57 PM

    If a fix pack was indeed applied, then the class cache could be affecting us and is brought out by our instrumenting of classes which would then mean that the Websphere cache would have to be cleared by the Websphere admin.  Please see the link below on this.

     

    Knowledge Base Articles 



  • 18.  Re: WebSphere Application Fails to start
    Best Answer

    Posted Oct 11, 2016 06:30 AM

    Hello Community.

    Really the problem was related to the classloader but the Introscope agent was not invloved. WebSphere was misconfigured and the classloader was removed each time we started the websphere due to a wrong configuration of the cluster template in WAS.

    Now the correct configuration was added to the WAS cluster template, the agent has been enabled with the autoprobe and everything is OK.

    I thank you so much for your help and very fast response.

     

    Luca Razzi



  • 19.  Re: WebSphere Application Fails to start

    Broadcom Employee
    Posted Oct 11, 2016 08:03 AM

    Dear Luca:

    Thank you for sharing your good news as well as the solution, It looked like that Matt and Hiko pointed you to the right direction with class loading

     

    Sincerely

    Hal German