Clarity

 View Only
  • 1.  v13.2   admin search recreate-index-files

    Posted Nov 11, 2013 07:41 PM
      |   view attached

    Hi All,

    The documents are stored in the database.

    After upgrading from Clarity 12.1.3.5 to version 13.2.0.4, recreating the index  results into error:

    admin search recreate-index-files

    java.lang.NoSuchMethodException: org.openxmlformats.schemas.wordprocessingml.x2006.main.impl.CTBookmarkImpl.<init>(org.apache.xmlbeans.SchemaType, boolean)
     at java.lang.Class.getConstructor0(Class.java:2730)
     at java.lang.Class.getDeclaredConstructor(Class.java:2004)
     at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.getJavaImplConstructor2(SchemaTypeImpl.java:1749)
     at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createUnattachedSubclass(SchemaTypeImpl.java:1886)
     at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createUnattachedNode(SchemaTypeImpl.java:1875)
     at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createElementType(SchemaTypeImpl.java:1021)
     at org.apache.xmlbeans.impl.values.XmlObjectBase.create_element_user(XmlObjectBase.java:893)
     at org.apache.xmlbeans.impl.store.Xobj.getUser(Xobj.java:1657)
     at org.apache.xmlbeans.impl.store.Cur.getUser(Cur.java:2654)
     at org.apache.xmlbeans.impl.store.Cur.getObject(Cur.java:2647)
     at org.apache.xmlbeans.impl.store.Cursor._getObject(Cursor.java:995)
     at org.apache.xmlbeans.impl.store.Cursor.getObject(Cursor.java:2904)
     at org.apache.poi.xwpf.usermodel.XWPFParagraph.buildRunsInOrderFromXml(XWPFParagraph.java:124)
     at org.apache.poi.xwpf.usermodel.XWPFParagraph.<init>(XWPFParagraph.java:79)
     at org.apache.poi.xwpf.usermodel.XWPFTableCell.<init>(XWPFTableCell.java:94)
     at org.apache.poi.xwpf.usermodel.XWPFTableRow.getTableCells(XWPFTableRow.java:133)
     at org.apache.poi.xwpf.usermodel.XWPFTableRow.<init>(XWPFTableRow.java:45)
     at org.apache.poi.xwpf.usermodel.XWPFTable.<init>(XWPFTable.java:121)
     at org.apache.poi.xwpf.usermodel.XWPFDocument.onDocumentRead(XWPFDocument.java:150)
     at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:159)
     at org.apache.poi.xwpf.usermodel.XWPFDocument.<init>(XWPFDocument.java:123)
     at com.niku.search.engine.lucene.parsers.MSWordParser.getParsedReaderForOpenDoc(MSWordParser.java:205)
     at com.niku.search.engine.lucene.ParserUtil.parse(ParserUtil.java:86)
     at com.niku.search.engine.lucene.LuceneIndexEngine.fileTextToIndexAttributes(LuceneIndexEngine.java:1075)
     at com.niku.search.engine.lucene.LuceneIndexEngine.indexFileList(LuceneIndexEngine.java:458)
     at com.niku.search.engine.lucene.LuceneIndexEngine.reindexfiles(LuceneIndexEngine.java:142)
     at com.niku.search.engine.lucene.LuceneIndexEngine.reindexfiles(LuceneIndexEngine.java:91)
     at com.niku.search.nspider.Nspider.main(Nspider.java:43)

    Any suggestion please?

     

    Thank you.

    Mayank

     

     

    Attachment(s)

    txt
    rif_log.txt   605 KB 1 version


  • 2.  RE: v13.2   admin search recreate-index-files

    Posted Nov 11, 2013 11:24 PM

    Hello Mayank,

    Can you please try the following?

    1. Pause the "Index contents and documents for search" scheduled job.

    2. Turn off synchronous indexing in the NSA. Restart services after doing this.

    3. Edit \clarity_home\.setup\search.xml

    4. Add the following line to recreate-index-files section of file: <jvmarg value="-Djava.io.tmpdir=D:\javatmp"/>

    5. Using NSA add "-Djava.io.tmpdir=D:\javatmp" to app service jvm parameters

    6. Using NSA add "-Djava.io.tmpdir=D:\javatmp" to bg service jvm parameters

    7. Create folder D:\javatmp

    8. Search for and delete any files that end in write.lock on your app server(s).

    9. Restart app and bg services.

    10. Delete the nspider and nspiderfile folders from the file server on the filestore searchindex

    11. Log into the app server as someone who has full control of the filestore folders including the ability to create new folders/files like the nspiderfiles folders

    12. Run the "nikuadmin search recreate-index-data" command

    13. Run the "nikuadmin search recreate-index-files" command

    14. When this has completed, turn on synchronous indexing **OR** re-schedule the "Index contents and documements" job. (From The Application)
    +++++++++++++++++++++++++++++++++++++++++++
    IMPORTANT NOTE ABOUT RE-INDEXING:
    If you are getting frequent search errors that cause you to reindex too frequently, consider running only the synchronous indexing or only the "Index contents and documents" job. There have been some instances where the running both at the same time have caused more errors to occur. Both methods accomplish essentially the same thing. All documents will be indexed using either one. The synchronous indexing indexes documents as they are added to the system which allows users to search for their document or its contents almost immediately after the document is added to the Clarity system. The disadvantage to this method is that it can provide extra system overhead in systems where lots of documents are being added and/or changed. The "Index Contents and Documents" job indexes documents only when the job is run. Most customers schedule this job to run anywhere from once every 15 minutes to once every hour. The job also does more optimization of the index than the synchronous indexing does. The disadvantage to this method is that users cannot search successfully for recently added documents until the next the job is run. If you choose to use the synchronous indexing option, I recommend running steps 11 and 12 to recreate your indexes once every month or once every 3 months depending on how heavily used your system is. This will optimize the index and improve the performance of your searches. If you are using the "Index Contents and Documents" job, you do not need to run these
    commands quite so frequently. Once or twice a year is probably sufficient.
    ----------------------------------------

    IMPORTANT NOTE:
    ==============================
    Please make sure to do the following after hours; Hence, this could utilize resources

    Regards

    Chandrani



  • 3.  RE: v13.2   admin search recreate-index-files

    Posted Nov 12, 2013 12:57 AM

    Hi Chandrani,

    Well, I followed the steps precisely as you documented, except, instead of "nikuadmin", I ran "admin".

    Nothing got written into D:\Javatmp

    admin search recreate-index-data gets completed successfully in no-time without any error.

    admin search recreate-index-files generates error in the admin.log, as attached before.

    Please advise.

     



  • 4.  RE: v13.2   admin search recreate-index-files

    Broadcom Employee
    Posted Nov 12, 2013 01:23 AM

    Mayank,

    What server you are using? The above steps are for windows ,you need to tweak it for non windows.

    Regards

    Suman



  • 5.  RE: v13.2   admin search recreate-index-files

    Posted Nov 12, 2013 01:33 AM

    Hello Mayank

    Have you checked this ?

    Filestore and Search Index Location Changes

    Two significant changes to the 13.x upgrade process directly affect the filestore feature. The changes require you to act during the upgrade process for Release 13.2 to ensure that the filestore feature operates correctly after the upgrade.

    Note: The actions are necessary if your files are not stored in the database. If the files are stored in the database, no action is required.

    You can determine if files are stored in the database using the Clarity System Administration (CSA) application. Log in to the application and navigate to the Documents and Search link under the Properties tab, then review the Store Files in Database property.

    Important! If the database is not the filestore repository, action is required. Failure to follow the recommended steps can lead to permanent loss of the filestore contents.

    The following changes were made to the upgrade process in Version 13:

    • The upgrade process always installs to an empty destination folder. During the upgrade, files and folders that exist in the target installation folder are moved to a new location before the new files are installed. If the filestore folder is located in the runtime folder, it is moved with the other content (not recommended). If the filestore is located outside the runtime folder (the recommended configuration), the filestore location is not moved or copied from the backup location.
    • The filestore and search index folder structures were altered to accommodate architectural changes in Version 13. This change requires action for all Version 13 upgrades, whether the filestore was located in the installation folder or not.

    Your action depends on the following factors:

    • Where the filestore was before the upgrade
    • The target installation folder that is selected during the upgrade

    The correction must be made immediately after the upgrade completes, before any files are added to the filestore. Any files added to the filestore before the correction is made must be manually preserved using the following steps:

    1. Use CA Clarity PPM to retrieve the newly checked-in documents and save them to a temporary location on the file system.
    2. Perform the steps for the appropriate use case (A, B, or C) presented in the following Upgrade Action.
    3. To check the saved documents back into the CA Clarity PPM filestore, use CA Clarity PPM.
    4. Execute the following commands from the <runtime>/bin folder:
      ‘admin search recreate-index-files’
      ‘admin search recreate-index-data’
      

    Upgrade Action: Complete the filestore and search index correction procedure.

    Before you start, create a backup of the filestore so that recovery is possible in the event of an unexpected problem. Your document store could be used before this procedure can be completed. If so, the newly included documents must be added back to the filestore after you complete the correction procedure.

    The following use cases illustrate the actions to take. To determine the correct course of action, use the filestore location and target installation folder.

    Use Case A

    • Filestore location: The filestore was located in the CA Clarity PPM runtime folder before the upgrade.
    • Target installation folder: The existing (preupgrade) CA Clarity PPM runtime folder was selected as the target installation folder.

    Before the upgrade process ran, the file or index folder was in the <runtime> folder. The upgrade process moved the file or folder to a folder named <runtime>_prev based on the chosen upgrade or installation option. The move resulted in the file or index folder being located in the <runtime>_prev folder. The file or index folder must be moved from the <runtime>_prev folder to another location because any subsequent upgrade attempt deletes this folder.

    Perform these steps for this use case:

    1. If before the upgrade the filestore location was <runtime>/filestore, copy or move the folders under <runtime>_prev/filestore to a new path:
      <new directory>/filestore/clarity
      

      Where <new directory> is located outside of the upgrade resulting <runtime> directory. Having the location outside of the runtime folder follows best practices. Placing the folder outside the CA Clarity PPM installation directory also ensures that this process is not repeated when CA Clarity PPM is upgraded.

      Log in to the CSA application, navigate to the Documents and Search link under the Properties tab, and update the Filestore Directory location. For example, if the filestore path is <old clarity install directory>/filestore, modify the path to <new directory>/filestore.

      Note: Do not append ‘clarity’ to the path in the CSA. The CA Clarity PPM architecture adds this update internally.

    2. If before the upgrade the search index directory location was <runtime>/searchindex, copy or move the folders under <runtime>_prev/searchindex to a new path:
      <new directory>/ searchindex /clarity 
      

      Where <new directory> is located outside of the upgrade resulting <runtime> directory. Having the location outside of the runtime folder follows best practices.

      Log in to the CSA application and navigate to the Documents and Search link under the Properties tab and update the Index Directory location. For example, if the filestore path is <old clarity install directory>/searchindex, modify the path to <new directory>/searchindex.

      Note: Do not append ‘clarity’ to the path in the CSA. The CA Clarity PPM architecture adds this update internally.

    3. Execute the following command from the <runtime>/bin folder:
      ‘admin search recreate-index-files’
      ‘admin search recreate-index-data’
      

    Use Case B

    • Filestore location: The filestore was located in the runtime folder before the upgrade.
    • Target installation folder: A new, empty folder was selected as the target install folder.

    Before the upgrade process ran, the file or index folder was located in the <runtime> folder. The upgrade process produced a different folder named <new runtime>. This process was based on the upgrade or installation option to select a different folder for the upgrade result than the current one. This choice has resulted in the file or index folder being located in the <runtime> folder.

    Perform these steps for this use case:

    1. If before the upgrade the filestore location was <runtime>/filestore, copy or move the folders under <runtime>/filestore to a new path:
      <new directory>/filestore/clarity
      

      Where <new directory> is located outside of the upgrade resulting <runtime> directory. Having the location outside of the runtime folder follows best practices. Locating the folder outside the installation directory ensures that this process is not repeated in the future when CA Clarity PPM is upgraded.

      Log in to the CSA application and navigate to the Documents and Search link under the Properties tab and update the Filestore Directory location. For example, if the filestore path is <old clarity install directory>/filestore, modify the path to <new directory>/filestore.

      Note: Do not append ‘clarity’ to the path in the CSA. The CA Clarity PPM architecture adds this update internally.

    2. If before the upgrade the search index directory location was <runtime>/searchindex, copy or move the folders under <runtime>/searchindex to a new path:
       <new directory>/ searchindex /clarity 
      

      Where <new directory> is located outside the upgrade resulting <runtime> directory. Having the location outside the runtime folder follows best practices.

      Log in to the CSA application, navigate to the Documents and Search link under the Properties tab, and update the Index Directory location. For example, if the filestore path is <old clarity install directory>/searchindex, modify the path to <new directory>/searchindex.

      Note: Do not append ‘clarity’ to the path in the CSA. The CA Clarity PPM architecture adds this update internally.

    3. Execute the following command from the <runtime>/bin folder:
      ‘admin search recreate-index-files’
      ‘admin search recreate-index-data’
      

    Use Case C

    • Filestore location: The filestore was located outside the runtime folder before the upgrade.
    • Target installation folder: The target installation folder is not relevant to this scenario. These steps must be performed for all upgrades with an external filestore.

    The file or index folder is not located in the <runtime> folder.

    Perform these steps for this use case:

    1. If the previous filestore folder path is now located in <some folder>/filestore, copy or move the folders under <some folder>/filestore to a new path:
      <some folder>/filestore/clarity
      
    2. If the previous search index folder path is now located in <some folder>/searchindex, copy or move the folders under <some folder>/seachindex to a new path:
      <some folder>/seachindex/clarity
      
    3. Execute the following command from the <runtime>/bin folder:
      ‘admin search recreate-index-files’
      ‘admin search recreate-index-data’
      

    No modification to the Filestore Directory or Index Directory properties is needed. You can view these properties from the Properties tab, Documents and Search subtab in the CSA application.

    Note: Do not append ‘clarity’ to the path in the CSA. The CA Clarity PPM architecture adds this update internally.

    ==================================================================================================

    Check with support if the issue still persists.

    NJ



  • 6.  RE: v13.2   admin search recreate-index-files

    Posted Nov 12, 2013 05:02 AM

    Hi Suman, 

    This is *all* windows environment for the Application, Database, Reports etc.

    Hi NJ, 

    Referring to the document, it says:

    Note: The actions are necessary if your files are not stored in the database. If the files are stored in the database, no action is required.

    The documents are stored in the Database as mentioned in my post hence I did not focus more on the cases described.

    The errors are found reported on the Apache Tomcat server as per the google search.

    Please advise if any further logging can help fix it.

    Thanks

    Mayank



  • 7.  RE: v13.2   admin search recreate-index-files

    Broadcom Employee
    Posted Nov 12, 2013 05:11 AM

    Hi Mayank,

    While you perform all the steps i hope you have stopped the clarity app services and you run those commands using the admininstrator rights. If you continue facing the error please raise the case with CA Support that will help.

    Regards

    Suman Pramanik



  • 8.  RE: v13.2   admin search recreate-index-files

    Posted Nov 13, 2013 10:05 PM

    Hi Chandrani,
    nikuadmin commmand runs same way with App and Bg services running or not running.

    May I know how adding following parameters help gather debug messages from nikuadmin?

    5. Using NSA add "-Djava.io.tmpdir=D:\javatmp" to app service jvm parameters
    6. Using NSA add "-Djava.io.tmpdir=D:\javatmp" to bg service jvm parameters

    Thanks
    Mayank