Automic Workload Automation

 View Only
Expand all | Collapse all

V24 Upgrade of database to UTF8

  • 1.  V24 Upgrade of database to UTF8

    Posted Jul 31, 2024 01:39 PM

    I am upgrading from 21.0.8 to the latest version of V24.  Automic is installed on Windows as MS SQL Server is on Windows.

    I have already created the utf8 database and executed the migration using the provided Workflow, mode 0 and 1.  I later found out that I did see checking the BETA check box.  I found out that this is at the server level and not at the database level.  I have these questions.  I do have a case opened and waiting for a response.  I'm hoping someone in the community might have the answer to these questions:

    1. Can I have the DBA enable BETA, reboot the server and I can continue with mode 2 or I need to drop the database, and recreate the database?  I believe that the DBA also took a backup of the empty database when BETA was not enabled.  Can I restore from the backup if one was created?  I realized that it would be empty and as I write this, I might be surprised if he did, but he could have.
    2. Since the BETA impacts the database server, what impact is there to other Automic database not yet on UTF8 and other non-Automic database on the same server?

    Thanks for any input.



  • 2.  RE: V24 Upgrade of database to UTF8

    Broadcom Employee
    Posted Aug 01, 2024 11:04 AM

    In order to give more context to @Lester Chew question. He is referring to the statement "You must enable a Windows Beta feature so that the "ODBC driver for SQL server" can work with UTF-8 data."  at https://docs.automic.com/documentation/webhelp/english/AA/24.1/DOCU/24.1/Automic%20Automation%20Guides/Content/Installation_Common/PreparationSteps/PrepareAEDB_MSSQL.htm 



    ------------------------------
    Michael K. Dolinek

    Engineering Program Manager | Agile Operation Division
    Broadcom Software
    ------------------------------



  • 3.  RE: V24 Upgrade of database to UTF8

    Posted Aug 05, 2024 12:58 PM

    Could we get some clarification?  Especially with the following that is highlighted.  From my reading of the statement, the AE server is the one that has to have the Windows Beta feature turned on and not necessarily the database.  Is it the AE server, the DB server or both?

    You must enable a Windows Beta feature so that the "ODBC driver for SQL server" can work with UTF-8 data. For the SQL Server, select local to use UTF-8. The AE using an MSSQL DB must run on a machine with the Windows Beta feature enabled. Otherwise, several utilities and SQLi will malfunction when UTF-8 data occurs within the SQL statements they produce. However, enabling the Beta feature comes with risks because switching the entire operating system to UTF-8 affects every process on your system - from older versions of Automic Automation (lower than 24.0) and other applications - and there is no guarantee that every process can handle running on a fully UTF-8 enabled machine.




  • 4.  RE: V24 Upgrade of database to UTF8

    Posted Aug 02, 2024 03:37 AM

    Hi @Lester Chew,

    Because of BETA check box, you cannot have both two database non UTF8 and UTF8 on the same server. So to upgrade from v21 to latest v24 the new database utf8 must be on a different server. This is only for MSSQL Server.


    Regards.

    Anicet



    ------------------------------
    [Automic Automation Consultant]
    [Paris, France]
    ------------------------------



  • 5.  RE: V24 Upgrade of database to UTF8

    Posted Aug 05, 2024 02:20 PM

    Anicet,

    Thanks for your input.  This subject is a bit confusing now.  I had the following reply from Broadcom regarding the other Automic database on the same DB server.  This is their response. 

    .  There should not be any impact on the other database by setting the Beta setting.  That said, please doublecheck this with your system admins and DBA to confirm that they don't have any concerns on this.

    Can you provide what insight you have experienced that you had that lead you to your conclusion?  I understand this is a sever wide setting for BETA.  However, the pre-utf8 Automic database are still under the original format.  The collation option has not been set to 'Latin1_General_100_CI_AS_SC_UTF8' unlike the new database.

    Thanks



    ------------------------------
    Lester Chew
    ------------------------------



  • 6.  RE: V24 Upgrade of database to UTF8

    Posted Aug 06, 2024 03:30 AM

    @Lester Chew,

    have a look on this part of document: Migrating the AE DB to UTF-8 (automic.com)

    Regards

    Anicet



    ------------------------------
    [Automic Automation Consultant]
    [Paris, France]
    ------------------------------



  • 7.  RE: V24 Upgrade of database to UTF8

    Posted Aug 07, 2024 11:35 AM

    There is a new twist on this database migration.  In a previous post, it was stated that I need to have the utf8 database created on a separate server because the BETA setting will impact the non-utf8 databases.

    That was confirmed by Broadcom.  

    Yesterday, I had the DBA created a temporary server to host the utf8 database.  I deleted the Workflow as recommended by Broadcom and installed the migration pack.  After setting up the two CONN objects and tested the connection, I proceeded to execute the migration Workflow.  I started to fill out the promptset when I realized that there is only one prompt for source/destination.  Now that I am dealing with two database servers, this seems like an impossibility to do as there is not a separate prompt for source and another prompt for destination.

    How is it possible to migrate the database if the promptset has only one prompt for source/destination and I'm told that the new utf8 database has to be on another server. 

    I hope the person who stated that I have to have the new database on a separate server can assist me on how it was done when they performed the upgrade.

    I do have a call with Broadcom later today and hopefully this will become clearer and will post an update.




  • 8.  RE: V24 Upgrade of database to UTF8

    Posted Aug 08, 2024 05:27 AM

    Hi @Lester Chew,

    you don't need to have two promptsets. The action pack will use the SQL Agent specified in the prompt with the two CONN Objet located in the SETUP.OBJECTS to point on the source DB (non-utf8) and the destination DB (utf8)

    Regards



    ------------------------------
    [Automic Automation Consultant]
    [Paris, France]
    ------------------------------



  • 9.  RE: V24 Upgrade of database to UTF8

    Broadcom Employee
    Posted Aug 08, 2024 06:18 AM
    Edited by Martin Uferbach Aug 08, 2024 06:23 AM

    I would like to apologize, this is a wrong info in the user documentation. The source and destination DB don't have to be on a different host.

    We already created a bug to get the user documentation fixed in this regard, but also some other aspects that might got mixed up and/or were unclear so far.

    The basic idea is as follows (MSSQL):

    • You do have a non-UTF-8 (old Automic Automation version) and a new empty AE DB (version-less as no schema/data yet) on one DB host/server
    • You enable the Windows Beta feature on the host where the AE is installed to ensure UTF-8/Unicode works properly from an OS and ODBC perspective  
    • You perform the migration via the AE DB migration action pack
    • You perform the AE upgrade to V24

    As the AE host now has to deal with the changed Windows Beta feature setting, this might have an impact on other applications running on the same host. We recommend to not operate an AE of an older version in parallel to the V24 upgrade one, and other non-Automic applications have to be checked if they still work properly with this enabled Window Beta setting.



    ------------------------------
    Product Owner for Automic Automation
    Broadcom
    Austria
    ------------------------------



  • 10.  RE: V24 Upgrade of database to UTF8

    Posted Aug 09, 2024 01:46 AM

    Hello Martin,

    just to make sure I understood correctly, let me re-phrase the information you just provided:

    Given an existing environment with

    • one MSSQL DB server running the V21 AE database
    • one ore more AE application servers running the V21 AE binaries

    the steps to upgrade to V24 are:

    • setup an additional database on the already existing MSSQL DB server => most likely I'd need to double storage beforehand as the V21 database is basically duplicated - correct?

    • one ALL AE application servers - but not the DB server - enable the Windows beta feature => V21 will run with the beta feature enabled! 
      Can Broadcom guarantee that V21 will operate normally (as it would without this setting) in this situation under all circumstances?
    • perform the migration via the AE DB migration action pack => Did you check on the potential performance impact for a busy, productive environment while the entire database is basically replicated on the same hardware?

    • 'upgrade' to V24 => execute the last phase of the migration pack, exchange the binaries and restart the system - correct?

    Please let me know of I understood correctly, if I missed any (important or unimportant) step AND if you - as the vendor - can guarantee that our production system will work correctly and reliably during the transition phase, given the fact that the beta feature on windows will be active while V21 is still running.

    Let me add some questions at the end to help me understand the process and with the goal to minimize the impact on the production environment:

    • which component exactly, requires the windows beta feature during the phase where the database is converted via copy to the second database?
    • would it be possible to have this very component on a separate server and would it be sufficient to activate the beta feature only there? This way the V21 server would not need to have - a basically unsupported - beta feature enabled.
    • Would Broadcom be willing to support a scenario where one would switch the database vendor during the migration? We are thinking of switching to Oracle and it would be great to save the extra copy step ....

    Many Thanks in advance! 

    Best Regards,

    René

    p.s.: Our AE database is about 1TB in size and we are running about 1 million JOBS/JOBF per day. We really need this transition to be flawless.




    • 11.  RE: V24 Upgrade of database to UTF8

      Broadcom Employee
      Posted Aug 09, 2024 04:50 AM
      Edited by Martin Uferbach Aug 09, 2024 08:12 AM

      Hi Rene, 

      I refined my previous post by the one below, I hope it answers your questions and that of the others:

      We would like to apologize, this info in the user documentation is incorrect/misleading. 

      The source and destination MSSQL AE DB don't have to be on a different host, but can be on different hosts.
      The AE DB Migration actionpack provides an option (promptsets) to support two AE DB hosts for MSSQL.

      The basic process is as follows (MSSQL):

      1. You have a non-UTF-8 (V21) and create a new empty AE DB (version-less as no schema/data loaded yet) on one or two separated DB hosts
      2. You perform the migration via the AE DB migration action pack (mind the promptsets option for two DB hosts), the data migration is completed
      3. You enable the Windows Beta feature on the host where the V24 AE is installed to ensure UTF-8/Unicode works properly from an OS and ODBC perspective  
      4. You perform the AE upgrade to V24

      As the V24 AE host has to deal with the changed Windows Beta feature setting, this might have an impact on other applications running on the same host.
      We recommend to enable the Windows Beta feature once the V21 is already shutdown right before the upgrade to V24, but check of other
      non-Automic applications still work properly. 

      AE DB Migration and Performance
      The migration process has the biggest impact on the destination AE DB (write data), the impact on the source AE DB is lower (read data only). To lower the impact on your old V21 system to be upgraded, you can separate the source and destination AE DB on different hosts (MSSQL).

      Components impacted by the Windows Beta feature
      The impacted components are all V24 AE processes that access the AE DB. To lower/mitigate the impact of the old V21 system to be upgraded, installing the V24 AE on a separate host makes sense.

      DB Vendor change during Migration
      A change of the DB vendor during the AE DB migration process is not supported today.

      We already created a bug to get the user documentation fixed in this regard, and we will also refine it with the info stated above.



      ------------------------------
      Product Owner for Automic Automation
      Broadcom
      Austria
      ------------------------------



    • 12.  RE: V24 Upgrade of database to UTF8

      Posted Aug 09, 2024 05:10 AM

      Hello Martin,

      many thanks for the swift response and clarification. 

      Just to make it crystal clear for me and as a double check I understood it right, could you confirm that activating 'the Windows beta feature' is NOT required on the V21 AE app servers, even though this is the system which will write into the new UTF-8 based database?

      Many thanks in advance,

      René




    • 13.  RE: V24 Upgrade of database to UTF8

      Broadcom Employee
      Posted Aug 09, 2024 08:06 AM

      Yes, the V21 system and/or the AE DB migration does NOT require this Windows Beta feature enabled. The AE DB migration copies technically data via the database capabilities itself.

      Therefore it is required for the V24 system operation/processes only.



      ------------------------------
      Product Owner for Automic Automation
      Broadcom
      Austria
      ------------------------------



    • 14.  RE: V24 Upgrade of database to UTF8

      Posted Aug 09, 2024 05:45 AM
      Hello Martin,
      Thank you for your clarification. 
      However, I have a major concern when you recommend installing AE v24 on another server. This will have a very big impact because agents and awi connect to the AE (connection=<AE_SERVER>:8443). 
      If we change the AE server to v24 we will have to update all the agents and other components that connect to the AE system during the upgrade procedure. Can you imagine the impact? the load? for environments with several hundred agents from different operating systems.
       
      Usually the agents are upgraded later, after the central components have been upgraded.

      Since the upgrade must take place when the AEs are stopped (because of the OffLine phase of the UTF-8 migration of the DB), activating WINDOWS Beta and upgrading the AE to v24 can be done on the same server that had AE v21. 
       
      Please clarify this point.

      Kind Regards,
      Anicet


      ------------------------------
      [Automic Automation Consultant]
      [Paris, France]
      ------------------------------



    • 15.  RE: V24 Upgrade of database to UTF8

      Broadcom Employee
      Posted Aug 09, 2024 08:16 AM

      We recommend to have the target AE DB on a different host, but the AE can stay on the same. As the AE V21 should be already shut down before the V24 upgrade is started, there should be no impact of the enabled Windows Beta feature on the shutdown V21 anymore.

      Yes, with this flow we can assure that we always have a backup of the source AE DB and the old V21 system.



      ------------------------------
      Product Owner for Automic Automation
      Broadcom
      Austria
      ------------------------------



    • 16.  RE: V24 Upgrade of database to UTF8

      Posted Aug 09, 2024 07:46 AM

      In preparation of migrating to v24 eventually, we have stood up new servers in our non-production environment (v21) and currently running the old and new Windows servers in tandem.  We turned on the Beta feature for the new servers and have not noticed an impact.

      We are in the process of building out the new production servers.

      Just wondering if we should turn off the Beta feature on the new servers until upgrading to v24?




    • 17.  RE: V24 Upgrade of database to UTF8

      Broadcom Employee
      Posted Aug 09, 2024 08:17 AM

      We didn't test it for old/non-UTF-8 versions, but it might have no impact.



      ------------------------------
      Product Owner for Automic Automation
      Broadcom
      Austria
      ------------------------------



    • 18.  RE: V24 Upgrade of database to UTF8

      Posted Aug 09, 2024 10:51 PM

      Here is the latest.  I've also read the additional post from Anicet, Thank you.

      This is what I have done since my last posting.  

      • Enable Beta option on a new database server that host no other database(s).
      • reboot the server.
      • created utf8 database on the new server with collation set to Latin1_General_100_CI_AS_SC_UTF8.
      • this is the key to the new database linking to the non-utf8 database: create a linked server called UC4_SOURCE using sp_addlinkedserver on the new database server.
      • as recommended by Broadcom, I deleted the migration package and reinstalled.
      • create the two connection objects and test the connection.
      • execute the migration workflow.

      The Workflow executed for some time creating new tables on the new database.  However, when it got to the XACT table, the workflow failed with this error:

      2024-08-07 13:42:38             SET XACT_ABORT ON

      2024-08-07 13:42:38             update UC4_SOURCE.snd_automic_ae.dbo.uc_migration set transfered='Y' where mig_run=1 and transfered='N'

      2024-08-07 13:45:03             The operation could not be performed because OLE DB provider "MSOLEDBSQL" for linked server "UC4_SOURCE" was unable to begin a distributed transaction.

      Going back to some of the posting, Martin, you stated that the two databases does not have to be on different database servers and the documentation is incorrect. That is what I did this before creating the new database on the host that contained the non-utf8 database and the workflow was running fine.  However, it was pointed out by Broadcom that in a step-by-step documentation that I created for the V24 upgrade, it was pointed out that I did not have the step to enable the Beta option.  That's when some red flags were on the way up.  Before I enabled the Beta option and since this setting impacts API functions: Enabling this setting causes certain API functions of the ODBC driver to change their behavior. 

      Martin, you pointed out that the documentation is incorrect.  We do not need to create the utf8 database on a separate host server that would have Beta option enabled.  If this is the case, then the statement in bold above is not a concern to other the new or current Automic database as well as other non-Automic databases on the same host server?

      The original steps I had planned to do was the create the utf8 database on the same database server that host the current V21 Automic database.  After the mode 2 migration is completed, I would shut down AE processes, stop the UC4.ServiceManager.  I would then rename the automic_ae to automic_ae_v21 and rename automic_ae_utf8 to automic_ae.  automic_ae_v21 would be stopped.  This is so I don't have to update any .ini or DSN settings.  Then execute the DB utility update and the rest of the upgrade steps updating the current installs.  I do not install V24 in it's own directory.  I do back up the AutomationEngine, ServiceManager, ServiceManagerDialog, Utility, AWI before doing any of the upgrade steps.  Start AE processes.

      Regarding Rene's posting, performance should not be an issue because you cannot do a ZDU to V24.  This requires a hard shutdown of Automic.  So, the on performance hit I can see is while doing mode 2 migration.

      Regarding Martin's latest post:

      The promptset option for two DB host, you are referring to the prompt: Are different instances used? correct?

      Your addition recommendation is to now is to enable Beta feature when V21 is shutdown.  This means I run the migration workflow mode 0, 1 and 2 first, stop AE processes, stop the V21 database, enable Beta feature, reboot the database server, rename automic_ae to automic_ae_v21, rename automic_ae_utf to automic_ae, start the DB utility upgrade, AE, SM, SMDi and AWI, correct?

      As for having each of the Automic database on separate host, see my comments regarding an error during mode 0 execution.  As stated, I didn't have this issue when both databases were on the same database host, but didn't enable the Beta feature.

      If the impacted components are only related to V24 AE process, then I should not be concern about the migration workflow needs the Beta option enabled since the workflow is running under V21 even though it is writing to the new utf8 database.  I see you have posted later that this is the case.

      Martin, is this information being provided to all of support so they can update my case with the correct information you posted here.  By the way, thanks for all the information you've provided.  We Guinea pigs are very grateful.  




    • 19.  RE: V24 Upgrade of database to UTF8

      Posted Aug 12, 2024 09:54 PM

      Another question.  

      It was indicated that only the V24 upgrade makes use of the BETA option.  How is it that we run the migration workflow using V21 to create and populate the V24 utf-8 database later to executing the Utility DB Upgrade all using V21 and have no issues when the BETA is enabled?




    • 20.  RE: V24 Upgrade of database to UTF8

      Posted Aug 13, 2024 03:38 AM
      Hi Lester,
      The DB migration workflow only runs native SQL statements and does not use odbc (for which the beta option is required).
      Regards
      Peter


      ------------------------------
      Automic Certified Professional/Expert & Broadcom Knight

      For AUTOMIC trainings please check https://www.qskills.de/qs/workshops/automic/
      ------------------------------



    • 21.  RE: V24 Upgrade of database to UTF8

      Posted Aug 14, 2024 01:38 PM

      I'm glad I brought up this topic.  I've read all of the recent responses.

      I did try to create a new utf8 database on a separate database server.  While running Mode 0 migration, I ran into this error: 

      2024-08-07 14:10:21             The operation could not be performed because OLE DB provider "MSOLEDBSQL" for
      linked server "UC4_SOURCE" was unable to begin a distributed transaction.

      I now understand the BETA feature is supposed to be enabled on the AE server.  I've executed intl.cpl on the AE servers (Windows Server 2016), Administrative/Change System locale and there is no check box for BETA.  Apparently this feature did not come with Windows Server 2016. I understand there is a registry change that can be made to enable this feature, but certainly comes with a risk if not correctly done and wonder if 2016 would recognize this registry setting.

      It's clear that the DB non-utf8 and utf8 can be on the same MSSQL database server.
      run the migration mode 0, 1 and 2
      shutdown V21 processes
      set UC4.ServiceManager to disable
      Enable the BETA feature (via intl.cpl or registry change)
      change ODBC data source unchecking 'Perform translation for character data' setting 
      Reboot the server
      Back up the V21 Utility/DB, AE, SM and SMDi directories
      Upgrade the V21 directories Utility/DB, AE, SM, SMDi and AWI to version 24 (on AE server upgrade directory D:\automic to version 24

      Enable UC4.ServiceManager to automatic
      Start SM

      If I need to backout, 

      stop all AE processes
      rename all directories to V24
      restore V21 all AE/AWI/awi.war file and directories
      restore V21 database
      disable BETA
      change ODBC data source checking 'Perform translation for character data' setting 
      reboot server
      UC4.ServiceManager starts after reboot

      Does that sound right?




    • 22.  RE: V24 Upgrade of database to UTF8

      Posted Aug 20, 2024 12:37 PM

      I was wondering the same. This is from Broadcom support:

      Microsoft 2016 isn't supported.

      https://downloads.automic.com/tools/compatibility_matrix?compatibility-mode=component&offering_version_id=components&lifecycle_entity_id=1409143059253&component_id=1409731986667&version_id=1701097261226

      I somehow missed that along the way. Clear as day, though.




    • 23.  RE: V24 Upgrade of database to UTF8

      Posted Sep 02, 2024 06:42 AM

      Hi all,

      I was trying to follow this thread but saw something that could be still confusing.

      From my understanding, the Beta-feature is needed by the AE and is not related to the DB.

      Usually in a Production system you have:
       - a SERVER_AE, that needs this Beta-feature
       - and another SERVER_DB that is not affected by this (as the ODBC connection is created on SERVER_AE)

      For example:

      "Because of BETA check box, you cannot have both two database non UTF8 and UTF8 on the same server"
      ==> This mentions the DB server, so the Beta-feature is not enable hier. (?)

      Kindly let me know if this is also how you interpret this. 

      LG,
      Roney




    • 24.  RE: V24 Upgrade of database to UTF8

      Posted Sep 03, 2024 06:07 AM

      Hi Roney,

      it's also my understanding, that the computer running the AutomationEngine requires this beta feature.

      The documentation (https://docs.automic.com/documentation/webhelp/english/AA/24.1/DOCU/24.1/Automic%20Automation%20Guides/Content/Installation_Common/PreparationSteps/PrepareAEDB_MSSQL.htm?tocpath=Installing%7CManual%20Installation%7CPreparing%20for%20the%20Manual%20Installation%7CPreparing%20the%20AE%20Database%7CPreparing%20the%20AE%20Database%20-%20MS%20SQL%7C_____0) mentions:

      The AE using an MSSQL DB must run on a machine with the Windows Beta feature enabled.

      regards,
      Peter



      ------------------------------
      Automic Certified Professional/Expert & Broadcom Knight

      For AUTOMIC trainings please check https://www.qskills.de/qs/workshops/automic/
      ------------------------------



    • 25.  RE: V24 Upgrade of database to UTF8

      Posted Sep 04, 2024 04:50 AM

      Great! Thanks a lot Peter and Jared.

      Cheers,
      Roney




    • 26.  RE: V24 Upgrade of database to UTF8

      Posted Sep 03, 2024 06:44 AM

      I can also read that as stating that with the BETA feature turned on, if you set up multiple ODBC connections, it requires that they connect to UTF8 databases and not a mixture of UTF8 and non-UTF8.