View Only
  • 1.  Failing to start app service on new server

    Posted Apr 08, 2016 06:17 AM
      |   view attached

    Hi All,


    I'd like to ask for identifying some errors threw during start of app service at our new server we have just got...

    Do you anyone knows what should it caused by? Many thanks

    Complete app-system.log is attached.


    Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[]]




    zip   3 KB 1 version

  • 2.  Re: Failing to start app service on new server
    Best Answer

    Broadcom Employee
    Posted Apr 08, 2016 01:46 PM

    Hi Matej,


    The error you are encountering is the following:

    2016/04/08 09:15:53.342 | Caused by: java.lang.IllegalStateException: Unable to complete the scan for annotations for web application [/niku] due to a StackOverflowError. Possible root causes include a too low setting for -Xss and illegal cyclic inheritance dependencies. The class hierarchy being processed was [org.jaxen.util.AncestorAxisIterator->org.jaxen.util.AncestorOrSelfAxisIterator->org.jaxen.util.AncestorAxisIterator]

    This happens when there is a library conflict.



    1. Please stop services

    2. Then connect to your $clarity_ppm/lib.


    This is a screenshot for 14.3:

    3. Then remove all jaxen files you don't see in the above list (you can move them elsewhere).


    You would probably have similar to these, old libraries in there:


    -rw-r--r--  1 niku clarity  160967 Apr 26  2002 jaxen-core.jar
    -rw-r--r--  1 niku clarity   13707 Apr 26  2002 jaxen-dom.jar
    -rw-r--r--  1 niku clarity    5949 Apr 26  2002 jaxen-jdom.jar


    Please remove/move them outside PPM directory. Then start back the app server, and see if that helps.


    Kind Regards
    Nika Hadzhikidi

  • 3.  Re: Failing to start app service on new server

    Posted Apr 11, 2016 06:24 AM

    Hi Nika,


    Thanks for your valuable response, it helped a lot but I have 2 questions on you.


    After removed all jaxen libs (I had all those 4), Clarity services failed to deploy, even not started of course...

    So finally, I've removed only those "old" jaxen-....jar libs, so only jaxen.jar remained there, then it worked well.


    Q1: Which purpose are these these jaxen libs for?


    Q2: We have 2 new servers actually, one is running fine, this second was facing this problem. Weird thing is that on this first server,

    which is running fine from beginning, all these jaxen libs (4) are there, without no issues. I've just checked it...


    Do you have any explanations there on my questions above? Many thanks


    Note: We're on Clarity ver. 13.3.



  • 4.  Re: Failing to start app service on new server

    Posted Apr 11, 2016 09:53 AM

    Q1.  jaxen: universal Java XPath engine - FAQ


    Clarity/PPM relies heavily on the parsing and executing of XML.  Every request/response likely goes through multiple XSL Transformations, several XPath queries to connect data and configurations for generating responses, and 'DOM walking' to traverse through the nodes (elements) of XML data and XML code files.  Jaxen is one of the libraries we make use of.


    Q2.  Such is the problem with classpath clashes unfortunately.  When Clarity/PPM starts up, it absorbs all the .jar files in the $NIKU_HOME/lib and $NIKU_HOME/customlib folders.  The order that those files are absorbed can vary from environment to environment, system to system.  Internally in those files, if they were to be extracted to disk, the 'class path' to a given class is identical (e.g. org/jaxen/etc/etc/someclassfile.class) and they'd basically overwrite each other - Java in memory is doing something similar (or actually it's a first-come, first-serve basis, and any duplicate paths/classes that come later get skipped as though they are not allowed to 'overwrite' the existing files).


    It typically means that on a given system, the same behaviour will repeat until it is addressed, and other systems will not have the problem at all - I suspect there is something at the OS level (file header tables in a directory) that just has the files in the directory in a given order, and this leads to the issue sticking or not sticking as it does.


    Q3. Why would you have those older version lib files (unasked but thought I'd take a stab at pre-empting it)?


    Normally these clashes come from a couple of reasons.

    1) Some customers make 'backup copies' of some jar files in the $NIKU_HOME/lib folder, not realising that these backups will also be loaded into memory just the same.  When newer versions then come to update/overwrite the older files, you then have multiple files with multiple versions of the same classes and a race to see which gets loaded first.  Probably NOT the cause in your case though, as I believe those other jaxen jar files are just the legitimate names of older versions of the libraries.


    2) Which brings us to custom interfaces/jobs/addins to Clarity/PPM, that are also able to deploy .jar files where they are using capabilities like the XML parsing themselves.  For the version those interfaces/jobs are initially written for, there's usually no clash (due to the developer testing all works well in that version), but when it comes to upgrades, then they can manifest whether they are carrying old libraries with the same names or new.  It also doesn't matter if those interfaces/jobs install those .jar files to the $NIKU_HOME/lib or $NIKU_HOME/customlib as Java (and Clarity/PPM) gives both locations an equal weighting and place when it comes to the loading of those files and classes.  So you can't have older versions in $NIKU_HOME/customlib and Clarity's own upgraded ones in $NIKU_HOME/lib either; the only reconciliations here are to stop using and uninstall the interfaces/jobs, or have them updated to work with the newer libraries Clarity is carrying, or have them built to use other libraries completely (which could still clash in future if Clarity/PPM chooses to later use those other libraries too - there's no fully upgrade-proof strategy here other than diligence and experience).


    Hope that helps.

  • 5.  Re: Failing to start app service on new server

    Posted Apr 12, 2016 02:19 AM

    Hi Nick,


    Very nice post, thanks for bringing some light to this topic for me


    So if it doesn't mind for Clarity, these "old" jaxen-....jar libs were moved out of "lib" folder so now it's working fine