TL;DR
Automic R&D has found that the Transport / API Rest Object import produces two Client updates per operation which should not be done. A bug was opened for it: AE-24369
Long version:@Adrian Fresno Menendez from Automic support made some great tests regarding object imports via AE REST API, DBload and XML import (using AWI). Here are his test results:
-------------------------------------------------------------------------------------------------------------------------
Environment:
vviesup09 - WO122: 2 node environment on Windows 12.3.3 HF1 with 10WPs with Oracle 12c database
11 clients, 24K objects on table OH
Test performed:
1. Launch the Deploy-Project.ps1 script that imports 4 objects via REST-API ( one at a time) ten times
a. No impact at all on performances
b. No 3434 messages as the routines do no take on on our side
c. We observe that for each object imported, the Client table is modified twice, not sure yet as to this is normal:
20200826/174014.897 - U00011872 Client table modified.
20200826/174014.897 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
20200826/174014.991 - U00011872 Client table modified.
20200826/174014.991 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
d. We also observe that the Cache is updated several times per a single object (U00021903 appears twice for the same single import on OH-IDNR and OH-NAME), not sure yet if this is normal:
20200826/174015.069 - U00004903 Cache for 'USG' successfully initialized. Length = '20480'
20200826/174015.131 - U01006000 Cache for 'HOST_IDNR' successfully initialized. Length = '585600000'.
20200826/174015.131 - U01006000 Cache for 'HOST_NAME' successfully initialized. Length = '20000000'.
20200826/174015.147 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
20200826/174015.147 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'
20200826/174015.194 - U00021903 Cache for 'CONN/CIT' successfully initialized. Length = '8192'
20200826/174015.334 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
20200826/174015.334 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'
2. Launch two JOBs that do a dbunload / dbload to import the same 4 objects using a single transport case
a. No impact at all on performances
b. No 3434 messages as the routines do no take on on our side
c. Only the client table is modified twice for ALL the jobs included in the transport case ( on this case 4)
d. The cache is updated as well twice for ALL the jobs included in the transport case
20200826/175554.766 - U00011872 Client table modified.
20200826/175554.766 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
20200826/175554.860 - U00011872 Client table modified.
20200826/175554.860 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
20200826/175554.938 - U00004903 Cache for 'USG' successfully initialized. Length = '20480'
20200826/175554.985 - U01006000 Cache for 'HOST_IDNR' successfully initialized. Length = '585600000'.
20200826/175554.985 - U01006000 Cache for 'HOST_NAME' successfully initialized. Length = '20000000'.
20200826/175555.001 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
20200826/175555.001 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'
20200826/175555.063 - U00052008 UCUCM module closed. Total time: '0000,00000' seconds.
20200826/175555.063 - U00021903 Cache for 'CONN/CIT' successfully initialized. Length = '8192'
20200826/175555.219 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
20200826/175555.219 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'
3. Import via AWI an xml with the whole 4 objects above
a. No impact at all on performances
b. No 3434 messages as the routines do no take on on our side
c. No client table modification on WP logs
d. No cache update on any WPs
Conclusions:
1. A transport case SHOULD improve the performances and not impact as much as the REST API import, but should impact the same if done for one single object.
2. A XML import should be the less impacting activity for such import operation as no cache / client table refresh is done at all.
-------------------------------------------------------------------------------------------------------------------------
On September 3rd I received the following update:
-------------------------------------------------------------------------------------------------------------------------
R&D has found that indeed the Transport / API Rest Object import produces two Client updates per operation which should not be done.
Hence, they requested to open a bug for it: AE-24369
At this time, we advise you to use the XML import instead of the API Rest Import, if you were to use the API Rest, then you would need to do all the objects (on this case 4) on a single post instead of doing the 4 posts that you are currently doing with this Powershell script so that the impact would be much lower as you have a system with lots of WPs (70).
The more WPs ,OH count and Clients, the worse the effects are when this API Rest import is performed.
We will let you know once they decide what will be done to address this issue in the bug, thanks!
-------------------------------------------------------------------------------------------------------------------------
I will update this thread when I get a new answer from support.
------------------------------
Automation Evangelist
Fiducia & GAD IT AG
---
Mitglied des deutschsprachigen
Automic-Anwendervereins FOKUS e.V.Member of the German speaking Automic user association FOKUS e.V.
------------------------------
Original Message:
Sent: 08-03-2020 07:09 AM
From: Tim Quakulinsky
Subject: AE REST API: Object import leads to massive performance problems / Community support required
Hi guys,
I need your support: We are experiencing significant problems using the AE REST API, which support unfortunately cannot reproduce.
We have colleagues who use "Automation As Code" to create Automic objects in Visual Studio code and import them into the system using the AE REST API. Now these colleagues do not only create a single object but always a workflow consisting of 5 objects: JOBP, JOBS, PRPT, etc. Only one object can be imported at a time in one call to the AE REST API, so the function is called five times in succession, within a few seconds. So far so good.
But what happens in the Automation Engine? Each individual import ensures that all WPs must update their object cache - even if the previous process has not yet been completed (or even started?)! The WPs must do this in the same way after loading objects using the DBload utility - with the difference that this does not happen after each individual transported object, but only once at the end. In our case, this behavior leads to massive performance and processing problems, since all WPs then have 100% load. Unfortunately, this can be related to a number of factors that support cannot easily adjust: suboptimal DB performance (unfortunately :-( ), many WPs (more than 50), many clients (about 50) with a lot of objects, activated Object Auditing, activated Version Management, and so on.
- Who of you has made similar experiences?
- Who of you has made similar experiences after loading the transport case? If so, you might also experience problems when importing via the AE REST API...
- Who of you also has a larger test or development environment and would test the import of multiple objects via the AE REST API? Who knows when your developers will come up with the idea to use this. I didn't know it before the problems occured...
By the way: Did you know that Automic recommends the following for Transport Cases: "To avoid that database locks caused by the DB Load utility obstruct the Automation Engine, make sure that no activities are running on the target Client when loading the object data to the target system." (Source: Transporting Data). In times of ZDU I find this recommendation a joke. I can't understand why Automic uses exactly the same technique for object imports via the AE REST API afterwards. In contrast to this, with XML import, the WPs do NOT have to rebuild the object cache afterwards...
We currently use the Automic version 12.3 ServicePack 2.
Thanks for your help
Tim
------------------------------
Automation Evangelist
Fiducia & GAD IT AG
---
Mitglied des deutschsprachigen Automic-Anwendervereins FOKUS e.V.
Member of the German speaking Automic user association FOKUS e.V.
------------------------------