Clarity

 View Only
Expand all | Collapse all

Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

  • 1.  Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Oct 04, 2013 05:45 PM
    Team,

    We have app servers 1, 2 and 3.

    App 1 has the nsa and is the main in the cluster.

    All three have been upgraded to 13.2 and they look good in NSA.

    App 1 URL is working and App 3 is working. App 2 does not.

    It says:
    [color]HTTP Status 500 -

    Message

    Description

    The server encountered an internal error that prevented it from fulfilling this request.[color]

    On reviewing the app ca and system logs, the message it has is as follows (and DB says available in app server 2 on nsa) and I'm at my wit's end :*) Any thoughts?


    SYS 2013-10-04 16:36:40,658 [http-bio-80-exec-2] niku.union (none:none:none:none) Clarity 13.2.0.472 initializing...
    WARN 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none)
    WARN 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Could not verify database connection for Local TenantInstance with id = clarity marking as an invalid configuration.
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) ==================================================================
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) ==================================================================
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) CLARITY

    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Tenants: TenantInstance: id: clarity
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Tenants: TenantInstance: appId: app
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Tenants: TenantInstance: bgId: bg
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Tenants: TenantInstance: fileDirectory: tenantDirectory
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Tenants: TenantInstance: status: invalid
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: id: clarity
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: serviceId: ORAU36
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: driver: com.ca.clarity.jdbc.oracle.OracleDriver
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: url: jdbc:clarity:oracle://ORAU36:1521;SID=ORAU36;BatchPerformanceWorkaround=true;InsensitiveResultSetBufferSize=0;ServerType=dedicated;supportLinks=true
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: username: WARM01
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: schemaName: WARM01
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: serviceName: ORAU36
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: vendor: oracle
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: largeIndex: WARM01_I001
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: smallIndex: WARM01_I001
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: largeTables: WARM01_D001
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Properties/Database: smallTables: WARM01_D001
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) ==================================================================
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Tenants for this Application with instance id = app :
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) ==================================================================
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) Active Tenants:
    SYS 2013-10-04 16:36:40,720 [http-bio-80-exec-2] config.TenantConfigurationManager (none:none:none:none) ==================================================================
    ERROR 2013-10-04 16:36:40,720 [http-bio-80-exec-2] niku.union (none:none:none:none) Clarity 13.2.0.472 failed to initialize.
    com.niku.union.config.ConfigurationException: The application failed to start because of a database configuration problem. Please review the application log for specific error information.


  • 2.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Oct 07, 2013 10:07 AM
    I have asked for a db restore and going to do 13.2 again.

    Please help me if I got this right.

    Servers:
    App 1 - main server with NSA
    App 2 - second in the cluster
    App 3 - third in the cluster

    During the upgrade:
    App servers on app servers down.
    Only DB is up.

    Perform 13.2 upgrade job on command line of app 1.

    My question is, should the beacons of all app servers be up at all? So the upgrade job knows?

    Second queston, once doing it on app 1, would i be able to distribute? Coz last time, i did upgrade job on each server.

    I believe one of my steps may have adversely affected the set up and caused my original problem with the tenant.

    Thanks!!!


  • 3.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Broadcom Employee
    Posted Oct 07, 2013 02:36 PM
    Hi..

    One of the approach i take is upgrade one server and see everything is working fine, then have the beacon running in all other servers and distribute else copy all thef older structure and put into multiple servers and make the necessary changes like java path, clarity home and then start the service.

    Hope this helps.

    Regards
    Suman Pramanik


  • 4.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Oct 16, 2013 04:30 PM
    I actually have a very similar approach to Suman. I simply do the install on a single machine, then copy over the clarity home directory over to the other servers. Different versions of clarity in the past have had occasional problems/upgrades to the Beacon/JMS. If I recall correctly, most of the tenant problems are normally related to wanting to communicate with the beacon service. So make sure the beacon servers can see each other with the tower command. You can also try stopping/removing and re-adding all the services.

    Michael Miller


  • 5.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Nov 15, 2013 12:10 PM
    One of the approach i take is upgrade one server and see everything is working fine, then have the beacon running in all other servers and distribute

     

    Hi Suman,

    If I do distribute all from upgraded server(13.2) to other servers in the clusters with V12.06 will upgrade them to V13.2?

    coz i'm following the below path for upgrade in first server

    V12.06 --> V12.1 and then V12.1-->13.2

    Thanks,

    Shiva

     



  • 6.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue
    Best Answer

    Broadcom Employee
    Posted Nov 18, 2013 08:48 AM

    Hi Shiva,

    In your scenario what i would do it upgrade one server by doing progressive upgrade like 12.0 --> 12.1 --> 13.0 -->13.2 and then copy folder structure to the other servers and make necessary prequisite installed before copying and also you need to make the directory changes in the properties.xml for each server and then start all the service except CSA.


    Regards

    Suman Pramanik



  • 7.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Nov 18, 2013 12:05 PM

    I agree with the approach by Suman. But before that, I would recommend you to check DB connection from the problem server to the database. Try creating a ODBC data source or use a sql session to connect to the clarity database directly. It is possible that your DB library got corrputed on the problem server.



  • 8.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Nov 22, 2013 08:15 AM
    prasu09:

    Hi Shiva,

    In your scenario what i would do it upgrade one server by doing progressive upgrade like 12.0 --> 12.1 --> 13.0 -->13.2 and then copy folder structure to the other servers and make necessary prequisite installed before copying and also you need to make the directory changes in the properties.xml for each server and then start all the service except CSA.


    Regards

    Suman Pramanik

    Thank you Suman,

    Your approach helped and I'm able to upgrade all the servers in the cluster.

    Thanks,
    Shiva
     



  • 9.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Broadcom Employee
    Posted Nov 22, 2013 09:15 AM

    Glad Shiva it worked. Please mark the post as resolved.

     

    Regards

    Suman



  • 10.  RE: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Nov 19, 2013 02:39 AM

    by any chance are these servers virtualised? if so make sure they are on the same physical host.



  • 11.  Re: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Apr 17, 2016 12:05 PM

    Thanks all for this thread with the 'just go copy the folder' approach. Capturing this for the next forum traveler.

     

    We recently upgraded to 14.3 stepping through 14.2. After 14.3 was up I used NSA's Distribute All - NSA told me all systems were good and health checks came back clean.

    One of my Background server's BG service failed to come online (beacon was fine) with the below error in the bg-system.log:

    Could not verify database connection for Local TenantInstance with id = clarity marking as an invalid

    The system was working fine in 14.2 an hour before. I've spent 6 hours diffing all the config files and checking system configurations before I found this thread.

     

    Steps to resolve:

    1) Remove all services on bad BG server.

    2) Go to working BG server, remove all services.

    3) Rename Clarity folder on bad BG server (eg clarity_BAK). Copy in Clarity folder from good BG server.

    4) Delete config folder on bad BG server.

    5) Copy in config folder from Clarity folder renamed above (eg clarity_BAK).

    6) Re-add all services.

     

    Is Fixed!

     

    Thanks all - wish I would have found a match on this error sooner. Moral of the story is Invalid Tennent errors may not be configuration setting related, but file system related no matter what NSA is telling you.

     

    Thanks!



  • 12.  RE: Re: Clarity 13.2 upgrade on multiple servers: LocalTenant instance issue

    Posted Oct 24, 2020 06:40 AM
    Edited by Georgy Oct 29, 2020 05:01 AM
    Hello Robert,

    Thank you so much for those perfect steps :)

    Very recently we did the upgrade to v15.8.1 on the PROD LIVE cluster and used the Distribute All which went all fine. The app services in just one of the servers had this same error line "Could not verify database connection for Local TenantInstance with id = clarity marking as an invalid configuration."  in app-ca.log files and that's how I landed up here in this forum page.

    Followed your exact steps like copying over the NIKU_HOME folder from a "working" server after removing all services there and swapping the config folder with the original config folder in the problematic server --- BOOM ---- the errors from app-ca logs just disappeared and the app services are now accessible on that particular server also.

    Thanks again,
    Georgy