Automic Workload Automation

 View Only
Expand all | Collapse all

Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

  • 1.  Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 09:24 AM
    Hi,
    I am doing an Automation Engine upgrade from 12.1.1 to 12.3.1.
    I have also upgrade my DB to 19c and my Linux server to Redhat 7.7

    I'm getting this error when I run the DbLoad 

    I do have a ticket open with support but just wondering if anyone else out there may have seen this before.

    20200107/014617.413 - U00003545 UCUDB: Opening database ...
    20200107/014617.413 - U00029112 UCUDB - Length of the DB-History area: 100
    20200107/014617.668 - U00029114 UCUDB - ALTER SESSION: NLS_LANGUAGE=AMERICAN
    20200107/014617.669 - U00029114 UCUDB - ALTER SESSION: NLS_TERRITORY=AMERICA
    20200107/014617.670 - U00029114 UCUDB - ALTER SESSION: NLS_LENGTH_SEMANTICS=CHAR
    20200107/014617.670 - U00029109 UCUDB: Local information from client environment
    20200107/014617.670 - U00003535 DB INFO: 'OCI_NLS_LANGUAGE = AMERICAN'
    20200107/014617.670 - U00003535 DB INFO: 'OCI_NLS_TERRITORY = AMERICA'
    20200107/014617.670 - U00003535 DB INFO: 'OCI_NLS_CHARACTER_SET = WE8MSWIN1252'
    20200107/014617.670 - U00029110 UCUDB - Session parameters:
    ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=1] [si_int=0] [si_ptr=(nil)] [si_addr=0x1]
    kpedbg_dmp_stack()+394<-kpeDbgCrash()+204<-kpeDbgSignalHandler()+113<-skgesig_sigactionHandler()+258<-__sighandler()<-_ZN5Trace6LoggenElPcz()+229<-0x0000000000000200
    Segmentation fault (core dumped)

    Thanks.
    JOhn.






  • 2.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 09:51 AM
    Hi.

    Well, not seen before, but it's a segmentation fault. Or in other words, your Oracle client (or a function in its libraries) appears to crash.

    • Does your Oracle client work outside of dbload with the same environment variables, i.e. in the same shell/session?
    • Are you doing anything out of the ordinary, like mixing Oracle shared objects between versions via LD_LIBRARY_PATH?
    • Are you using Oracle libraries from a supported version of the Oracle client?
    • Did you set LD_LIBRARY_PATHS to any Automic component? If so: don't. The only LD_LIBRARY_PATH you should have in your environment when running dbload should be pointing to Oracle libraries and nothing else (I have an article somewhere on these forums explaining this mess).
    • Failing anything else, try a different Oracle client. They are cross-compatible afaik, you can use a V18 client for a V19 db.

    Hth,


  • 3.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 10:43 AM
    Thanks @Carsten Schmitz 
    Let me talk with my DBA team regarding the Oracle Client

    I do know that my LD_LIBRARY_PATH does contain many references to my Automic components.
    I can remove these for starters.

    setenv LD_LIBRARY_PATH $ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib:.:/wla/Automic/Automation.Platform/AutomationEngine/bin:/wla/Automic/Automation.Platform/ServiceManager/bin







  • 4.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 10:57 AM
    Hi,

    /wla/Automic/Automation.Platform/AutomationEngine/bin​
    /wla/Automic/Automation.Platform/ServiceManager/bin

    This might be the problem already, using Server or Service Manager libraries with the utilities produces all sorts of strange issues (though I don't know if the "." beforehand will override that and allow the utilities to use their own, correct shared objects).

    Just a quick "unset LD_LIBRARY_PATH" and setting (or exporting) only the string with the Oracle libs should suffice to test that theory.


  • 5.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 08, 2020 11:03 AM
    So when I reduce my LD_LIBRARY_PATH to just
    setenv LD_LIBRARY_PATH $ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib
    It looks like I'm not getting as far

    Getting this error now...

    20200108/075936.059 - U00038091 Change application directory to '../db/'.
    20200108/075936.059 - U00038090 Warning: It was not possible to change current application directory to ''.
    20200108/075936.059 - U00029120 UCUDB: Function LoadLibrary failed. Library 'ucuoci.so' error 'ucuoci.so: cannot open shared object file: No such file or directory'.
    20200108/075936.060 - U00003531 UCUDB: Error while checking data source (e.g. code conversion or driver problem, see log file).
    20200108/075936.060 - U00003531 UCUDB: Error while checking data source (e.g. code conversion or driver problem, see log file).


    My ORACLE_HOME is set to the correct place according to my DBA

    setenv ORACLE_HOME /mis/XSJLINUX/oracle/Linux3.10prod/19.0.0.0


    I'll talk to my DBA team regarding the Oracle Client tomorrow....


  • 6.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 04:01 AM
    ​Hi John

    I hit that library problem on my last upgrade to 12.1 , I needed to add the utilities directory to my LD_LIBRARY_PATH on the app servers :-

    /home/uc4user:echo $LD_LIBRARY_PATH
    :/home/oracle/app/oracle/product/12.1.0/client_1/lib:/software/Automic/Utility/bin



  • 7.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 04:12 AM
    ​Indeed.

    Either adding the utilities directory to LD_LIBRARY_PATH, or not adding any Automic directories at all and launching dbload from the utilities directory both ensures it will not use the wrong libraries.

    Not adding anything at all is the way an Automic dev recommended to me by phone though.

    This is my original PSA on the matter:

    https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=792168

    Br,


  • 8.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 04:56 AM
    Thanks @JOE MURRAY and @Carsten Schmitz
    Will play around with a couple of different options to see which one works...
    ​​


  • 9.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 06:17 AM
    I think I am close but now getting this...

    Version and status of database: '12.1'/'READY'.
    Version and content of data file: '12.3'/'UPGRADE'.
    Version of database objects: '12.1.1+hf.1.build.1364'
    Loading in mode 'UPGRADE'.
    20200109/030641.748 - U00038140 Action not possible. Different Automation Engine systems were started with this database. Please check your database and try again.




  • 10.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1

    Posted Jan 09, 2020 07:11 AM
    Edited by Carsten Schmitz Jan 09, 2020 07:17 AM
    Yeah, definetly a step forward.

    This now sounds to me like either a Zero Downtime Upgrade is in progress or botched, or you are using the wrong utilities. For updates, you need to use the utilities from the new version, those from the old version won't work.

    Hth,

    ------------------------------
    I will not respond to PM asking for help unless there's an actual reason to keep the discussion off of the public forums.
    ------------------------------



  • 11.  RE: Error on Automation Engine DB Upgrade from 12.1.1 to 12.3.1