Automic Workload Automation

 View Only
Expand all | Collapse all

Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

  • 1.  Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 09:24 AM
    Hi,
    I am doing an Automation Engine upgrade from 12.1.1 to 12.3.1.
    I have also upgrade my DB to 19c and my Linux server to Redhat 7.7

    I'm getting this error when I run the DbLoad 

    I do have a ticket open with support but just wondering if anyone else out there may have seen this before.

    20200107/014617.413 - U00003545 UCUDB: Opening database ...
    20200107/014617.413 - U00029112 UCUDB - Length of the DB-History area: 100
    20200107/014617.668 - U00029114 UCUDB - ALTER SESSION: NLS_LANGUAGE=AMERICAN
    20200107/014617.669 - U00029114 UCUDB - ALTER SESSION: NLS_TERRITORY=AMERICA
    20200107/014617.670 - U00029114 UCUDB - ALTER SESSION: NLS_LENGTH_SEMANTICS=CHAR
    20200107/014617.670 - U00029109 UCUDB: Local information from client environment
    20200107/014617.670 - U00003535 DB INFO: 'OCI_NLS_LANGUAGE = AMERICAN'
    20200107/014617.670 - U00003535 DB INFO: 'OCI_NLS_TERRITORY = AMERICA'
    20200107/014617.670 - U00003535 DB INFO: 'OCI_NLS_CHARACTER_SET = WE8MSWIN1252'
    20200107/014617.670 - U00029110 UCUDB - Session parameters:
    ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=1] [si_int=0] [si_ptr=(nil)] [si_addr=0x1]
    kpedbg_dmp_stack()+394<-kpeDbgCrash()+204<-kpeDbgSignalHandler()+113<-skgesig_sigactionHandler()+258<-__sighandler()<-_ZN5Trace6LoggenElPcz()+229<-0x0000000000000200
    Segmentation fault (core dumped)

    Thanks.
    JOhn.






  • 2.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 09:51 AM
    Hi.

    Well, not seen before, but it's a segmentation fault. Or in other words, your Oracle client (or a function in its libraries) appears to crash.

    • Does your Oracle client work outside of dbload with the same environment variables, i.e. in the same shell/session?
    • Are you doing anything out of the ordinary, like mixing Oracle shared objects between versions via LD_LIBRARY_PATH?
    • Are you using Oracle libraries from a supported version of the Oracle client?
    • Did you set LD_LIBRARY_PATHS to any Automic component? If so: don't. The only LD_LIBRARY_PATH you should have in your environment when running dbload should be pointing to Oracle libraries and nothing else (I have an article somewhere on these forums explaining this mess).
    • Failing anything else, try a different Oracle client. They are cross-compatible afaik, you can use a V18 client for a V19 db.

    Hth,


  • 3.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 10:43 AM
    Thanks @Carsten Schmitz 
    Let me talk with my DBA team regarding the Oracle Client

    I do know that my LD_LIBRARY_PATH does contain many references to my Automic components.
    I can remove these for starters.

    setenv LD_LIBRARY_PATH $ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib:.:/wla/Automic/Automation.Platform/AutomationEngine/bin:/wla/Automic/Automation.Platform/ServiceManager/bin







  • 4.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 10:57 AM
    Hi,

    /wla/Automic/Automation.Platform/AutomationEngine/bin​
    /wla/Automic/Automation.Platform/ServiceManager/bin

    This might be the problem already, using Server or Service Manager libraries with the utilities produces all sorts of strange issues (though I don't know if the "." beforehand will override that and allow the utilities to use their own, correct shared objects).

    Just a quick "unset LD_LIBRARY_PATH" and setting (or exporting) only the string with the Oracle libs should suffice to test that theory.


  • 5.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 11:03 AM
    So when I reduce my LD_LIBRARY_PATH to just
    setenv LD_LIBRARY_PATH $ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib
    It looks like I'm not getting as far

    Getting this error now...

    20200108/075936.059 - U00038091 Change application directory to '../db/'.
    20200108/075936.059 - U00038090 Warning: It was not possible to change current application directory to ''.
    20200108/075936.059 - U00029120 UCUDB: Function LoadLibrary failed. Library 'ucuoci.so' error 'ucuoci.so: cannot open shared object file: No such file or directory'.
    20200108/075936.060 - U00003531 UCUDB: Error while checking data source (e.g. code conversion or driver problem, see log file).
    20200108/075936.060 - U00003531 UCUDB: Error while checking data source (e.g. code conversion or driver problem, see log file).


    My ORACLE_HOME is set to the correct place according to my DBA

    setenv ORACLE_HOME /mis/XSJLINUX/oracle/Linux3.10prod/19.0.0.0


    I'll talk to my DBA team regarding the Oracle Client tomorrow....


  • 6.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 04:01 AM
    ​Hi John

    I hit that library problem on my last upgrade to 12.1 , I needed to add the utilities directory to my LD_LIBRARY_PATH on the app servers :-

    /home/uc4user:echo $LD_LIBRARY_PATH
    :/home/oracle/app/oracle/product/12.1.0/client_1/lib:/software/Automic/Utility/bin



  • 7.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 04:12 AM
    ​Indeed.

    Either adding the utilities directory to LD_LIBRARY_PATH, or not adding any Automic directories at all and launching dbload from the utilities directory both ensures it will not use the wrong libraries.

    Not adding anything at all is the way an Automic dev recommended to me by phone though.

    This is my original PSA on the matter:

    https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=792168

    Br,


  • 8.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 04:56 AM
    Thanks @JOE MURRAY and @Carsten Schmitz
    Will play around with a couple of different options to see which one works...
    ​​


  • 9.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 06:17 AM
    I think I am close but now getting this...

    Version and status of database: '12.1'/'READY'.
    Version and content of data file: '12.3'/'UPGRADE'.
    Version of database objects: '12.1.1+hf.1.build.1364'
    Loading in mode 'UPGRADE'.
    20200109/030641.748 - U00038140 Action not possible. Different Automation Engine systems were started with this database. Please check your database and try again.




  • 10.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 07:11 AM
    Edited by Carsten Schmitz Jan 09, 2020 07:17 AM
    Yeah, definetly a step forward.

    This now sounds to me like either a Zero Downtime Upgrade is in progress or botched, or you are using the wrong utilities. For updates, you need to use the utilities from the new version, those from the old version won't work.

    Hth,

    ------------------------------
    I will not respond to PM asking for help unless there's an actual reason to keep the discussion off of the public forums.
    ------------------------------



  • 11.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 07:35 AM
    I'm definitely using the new Utilities binaries.
    Regarding the Zero Downtime Upgrade, I haven't touched this. We did look at using for previous upgrade from 12 to 12.1 but didn't use it in the end.

    Is it possible I need to remove some data from one of my tables to allow me to progress?

    I did find this, which mentions a delete statement, but not sure it's the same issue...

    https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=787657






  • 12.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 07:55 AM
    No, sorry, I wouldn't know.

    I remember that we had to delete some WP during ​an upgrade, but that was 10.x to 12.x and we deleted them in client 0. It doesn't ring a bell for me regarding 12.1 to 12.3.

    Sorry, not a clue about this one ...


  • 13.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 09:24 AM
    Hi

    This err message usually appears if the System name in ucsrv.ini does not match all [SYSTEMNAME]#WP$$ and [SYSTEMNAME]#CP$$ Objects in Client 0.

    Usually appears if you did somewhen in the past start the system with another INI file or changed system name....

    => start up AE system, go to client 0 and delete all WP and CP objects that do no match your current system name.

    cheers, Wolfgang

    ------------------------------
    Support Info:
    if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
    Otherwise update/upgrade your system and check if the problem still exists.
    ------------------------------



  • 14.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 09:37 AM
    Not sure I can do this @Frank Muffke as we have upgraded our DB from 11c to 19c and also our OS to RedHat 7.7.
    I'm assuming that I won't be able to restart a V12.1 engine on a 19c database...

    What you say makes sense as I did copy an existing System's DB to this new system to test my upgrade so there may be references to the old system.
    Is the WP and CP data stored in a table I can delete from there?




  • 15.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1
    Best Answer

    Posted Jan 09, 2020 10:01 AM
    Edited by Diane Craddock Jan 17, 2020 09:35 AM

    We ran into the same problem when trying to upgrade our system a while back. Broadcom provided us with a SQL query that removed the offending records from HOST and OH but I no longer have the query on hand (plus we're on MSSQL, anyways, and I'd rather avoid any "My system is on fire because of you"-type comments :) ).

    I think it was pretty straight-forward anyways as the records were named with the old system name, but you may want to get it from Broadcom. At least with us, our consultant advised us not to make any direct changes to the database without CA/Broadcom's blessing or support could be refused. Something to keep in mind if things happen to go sideways.  





  • 16.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 10, 2020 02:35 AM
    Hi

    As Kyle  explained, there exits an SQL in Automic Support - unfortunately I do not have it :-(

    => I am afraid you need to open a support call.
    => Or you give it a try with the "foreign" version... some time Ago I installed Operations manger V8 (from 2011) on Win10 with SQL Server 2016 without issues :-)


    cheers, Wolfgang

    ------------------------------
    Support Info:
    if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
    Otherwise update/upgrade your system and check if the problem still exists.
    ------------------------------



  • 17.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 10, 2020 04:44 AM
    Hi Kyle,
    That for that, very useful.
    To make it a bit easier for support to find the SQL, do you have a ticket number by any chance from your previous problem?

    John.



  • 18.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 20, 2020 11:07 AM
    Thanks to everyone, especially @Philipp Elmer for helping here.

    Eventually I deleted rows from OH and HOST tables for type SERV and Upgrade worked.


    select OH_Client, OH_Name, HOST.*

    from OH

    inner join HOST on OH_IDNR = HOST_OH_IDNR

    where OH_DeleteFlag = 0

    and OH_Otype = 'SERV';