Automic Workload Automation

 View Only
  • 1.  AE REST API: Object import leads to massive performance problems / Community support required

    Posted Aug 03, 2020 07:09 AM
    Edited by Tim Quakulinsky Sep 18, 2020 04:13 AM
    [Update 2020-09-18 - 10:13 CEST]

    TL;DR

    The creation of objects using AE Rest API provokes multiple client / cache refreshes per created object. On bigger systems with many WPs, this may lead to a standstill of the system.

    Automic published an KB article for the problem with the double cache refresh: AE REST API for Object creation produces duplicated Clients/Object Cache updates per Object

    -------------------------------

    Hi guys,

    I need your support: We are experiencing significant problems using the AE REST API, which support unfortunately cannot reproduce.

    We have colleagues who use "Automation As Code" to create Automic objects in Visual Studio code and import them into the system using the AE REST API. Now these colleagues do not only create a single object but always a workflow consisting of 5 objects: JOBP, JOBS, PRPT, etc. Only one object can be imported at a time in one call to the AE REST API, so the function is called five times in succession, within a few seconds. So far so good.

    But what happens in the Automation Engine? Each individual import ensures that all WPs must update their object cache - even if the previous process has not yet been completed (or even started?)! The WPs must do this in the same way after loading objects using the DBload utility - with the difference that this does not happen after each individual transported object, but only once at the end. In our case, this behavior leads to massive performance and processing problems, since all WPs then have 100% load. Unfortunately, this can be related to a number of factors that support cannot easily adjust: suboptimal DB performance (unfortunately :-( ), many WPs (more than 50), many clients (about 50) with a lot of objects, activated Object Auditing, activated Version Management, and so on.

    • Who of you has made similar experiences?
    • Who of you has made similar experiences after loading the transport case? If so, you might also experience problems when importing via the AE REST API...
    • Who of you also has a larger test or development environment and would test the import of multiple objects via the AE REST API? Who knows when your developers will come up with the idea to use this. I didn't know it before the problems occured...
    By the way: Did you know that Automic recommends the following for Transport Cases: "To avoid that database locks caused by the DB Load utility obstruct the Automation Engine, make sure that no activities are running on the target Client when loading the object data to the target system." (Source: Transporting Data). In times of ZDU I find this recommendation a joke. I can't understand why Automic uses exactly the same technique for object imports via the AE REST API afterwards. In contrast to this, with XML import, the WPs do NOT have to rebuild the object cache afterwards...

    We currently use the Automic version 12.3 ServicePack 2.

    Thanks for your help
    Tim

    ------------------------------
    Automation Evangelist
    Fiducia & GAD IT AG
    ---
    Mitglied des deutschsprachigen Automic-Anwendervereins FOKUS e.V.
    Member of the German speaking Automic user association FOKUS e.V.
    ------------------------------


  • 2.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Aug 03, 2020 09:33 AM
    Edited by Michael A. Lowry Aug 03, 2020 09:47 AM
    We experienced a very similar problem a few years ago. We were seeing very poor AE server performance following a DB load. The problem was a bug in the AE, and it was particularly bad immediately following multiple simultaneous DB loads. WPs handling ITL table entries related to the creation of version control objects were sending far too many kicks to notify other WPs about the changes.

    We caught the problem happening by enabling tracing in the AWI. In the XML trace we saw an unreasonably high number of kicks. Here is an example:
    2017-11-20 09:46:59,712 ionID=0000000020553975 [TRACE] NOSESSION/- NOUI [.aetracing.AutomationEngineTraceListener] - <uc-env session="">
    <kick client="0001" idnr="0000000000" system="AE_SYS1">OFS </kick>
    </uc-env>
    I do not know how the JCP handles XML imports, but if the ITL table is involved, you can list the current entries using this query:
    select ITL_Idnr, ITL_TType, ITL_TIdnr, OH_Name
    from ITL left join OH on ITL_TIdnr=OH_IDnr
    The entries are cleared out very quickly if any WPs are running, so you will have to be quick to catch them.

    Would someone from Broadcom please provide more detailed information about how XML imports via the REST APIs work behind the scenes?




  • 3.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Sep 14, 2020 05:47 AM

    TL;DR

    Automic R&D has found that the Transport / API Rest Object import produces two Client updates per operation which should not be done.  A bug was opened for it: AE-24369


    Long version:

    @Adrian Fresno Menendez from Automic support made some great tests regarding object imports via AE REST API, DBload and XML import (using AWI). Here are his test results:

    -------------------------------------------------------------------------------------------------------------------------

    Environment:

    vviesup09 - WO122: 2 node environment on Windows 12.3.3 HF1 with 10WPs with Oracle 12c database
    11 clients, 24K objects on table OH


    Test performed:

    1. Launch the Deploy-Project.ps1 script that imports 4 objects via REST-API ( one at a time) ten times

    a. No impact at all on performances
    b. No 3434 messages as the routines do no take on on our side
    c. We observe that for each object imported, the Client table is modified twice, not sure yet as to this is normal:
    20200826/174014.897 - U00011872 Client table modified.
    20200826/174014.897 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
    20200826/174014.991 - U00011872 Client table modified.
    20200826/174014.991 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
    d. We also observe that the Cache is updated several times per a single object (U00021903 appears twice for the same single import on OH-IDNR and OH-NAME), not sure yet if this is normal:
    20200826/174015.069 - U00004903 Cache for 'USG' successfully initialized. Length = '20480'
    20200826/174015.131 - U01006000 Cache for 'HOST_IDNR' successfully initialized. Length = '585600000'.
    20200826/174015.131 - U01006000 Cache for 'HOST_NAME' successfully initialized. Length = '20000000'.
    20200826/174015.147 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
    20200826/174015.147 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'
    20200826/174015.194 - U00021903 Cache for 'CONN/CIT' successfully initialized. Length = '8192'
    20200826/174015.334 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
    20200826/174015.334 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'


    2. Launch two JOBs that do a dbunload / dbload to import the same 4 objects using a single transport case

    a. No impact at all on performances
    b. No 3434 messages as the routines do no take on on our side
    c. Only the client table is modified twice for ALL the jobs included in the transport case ( on this case 4)
    d. The cache is updated as well twice for ALL the jobs included in the transport case
    20200826/175554.766 - U00011872 Client table modified.
    20200826/175554.766 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
    20200826/175554.860 - U00011872 Client table modified.
    20200826/175554.860 - U00011869 Client/Mode/TaskPrio/AH_Idnr/TZ/Prio/EH_Kick/Cale_Ahead/VersionMgmnt/XRO/Object_Audit: 0022/GO /200/0005448221/TZ.CET22/200/0003/0014/N/N/N
    20200826/175554.938 - U00004903 Cache for 'USG' successfully initialized. Length = '20480'
    20200826/175554.985 - U01006000 Cache for 'HOST_IDNR' successfully initialized. Length = '585600000'.
    20200826/175554.985 - U01006000 Cache for 'HOST_NAME' successfully initialized. Length = '20000000'.
    20200826/175555.001 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
    20200826/175555.001 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'
    20200826/175555.063 - U00052008 UCUCM module closed. Total time: '0000,00000' seconds.
    20200826/175555.063 - U00021903 Cache for 'CONN/CIT' successfully initialized. Length = '8192'
    20200826/175555.219 - U00021903 Cache for 'OH-IDNR' successfully initialized. Length = '524288'
    20200826/175555.219 - U00021903 Cache for 'OH-NAME' successfully initialized. Length = '524288'

    3. Import via AWI an xml with the whole 4 objects above

    a. No impact at all on performances
    b. No 3434 messages as the routines do no take on on our side
    c. No client table modification on WP logs
    d. No cache update on any WPs

    Conclusions:

    1. A transport case SHOULD improve the performances and not impact as much as the REST API import, but should impact the same if done for one single object.
    2. A XML import should be the less impacting activity for such import operation as no cache / client table refresh is done at all.
    -------------------------------------------------------------------------------------------------------------------------

    On September 3rd I received the following update:
    -------------------------------------------------------------------------------------------------------------------------
    R&D has found that indeed the Transport / API Rest Object import produces two Client updates per operation which should not be done.

    Hence, they requested to open a bug for it: AE-24369

    At this time, we advise you to use the XML import instead of the API Rest Import, if you were to use the API Rest, then you would need to do all the objects (on this case 4) on a single post instead of doing the 4 posts that you are currently doing with this Powershell script so that the impact would be much lower as you have a system with lots of WPs (70).

    The more WPs ,OH count and Clients, the worse the effects are when this API Rest import is performed.

    We will let you know once they decide what will be done to address this issue in the bug, thanks!
    -------------------------------------------------------------------------------------------------------------------------

    ​I will update this thread when I get a new answer from support.

    ------------------------------
    Automation Evangelist
    Fiducia & GAD IT AG
    ---
    Mitglied des deutschsprachigen Automic-Anwendervereins FOKUS e.V.
    Member of the German speaking Automic user association FOKUS e.V.
    ------------------------------



  • 4.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Sep 14, 2020 08:53 AM
    Edited by Michael A. Lowry Sep 18, 2020 05:06 AM
    This confirms that AE object import works very differently via the REST APIs compared to via the Java APIs. Would someone from Broadom please provide more information about how these two functions work behind the scenes? Thanks in advance.

    Ping @Tobias Stanzel, @David Ainsworth
    ​​​​


  • 5.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Sep 15, 2020 03:01 AM
    Edited by Marcel Friedmann Sep 15, 2020 03:18 AM
    I doubled checked this with the behaviour of the apm tool (Plugin Manager - used to export/upgrade/import action packs).
    apm uses the undocumented REST API https://xxxxx.host.com/ae/api/v1/:clientid:/folderobjects to import objects recursively into a folder structure.
    However same problem here.
    CACHE and Client table are updated twice for every WP.

    Kind of frustrating since this makes the AE REST API json Object import unusable for large system setups.
    This also hits CDA. CDA is heavily based on Action Packs.

    ------------------------------
    Architect
    Robert Bosch GmbH
    ------------------------------



  • 6.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Broadcom Employee
    Posted Sep 18, 2020 04:21 AM
    Edited by Markus Embacher Sep 18, 2020 04:24 AM
    XML import is a XREQ which sends KICK messages to WPs to update the caches based on imported object types (one message per object type and WP after the import is finished).


  • 7.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Sep 18, 2020 04:59 AM
    Edited by Tim Quakulinsky Sep 18, 2020 05:39 AM
    I think we customers are on our own again here... Where do we want to collect the available information?

    Here are the information I know about the topic:

    DBload
    • Data format:Proprietary
    • DBload inserts objects in DB
    • Coherence check during insert: ?
    • After insert all WPs must refresh their client / object caches
    AE REST API
    • Data format: JSON
    • JCP inserts objects in DB
    • Coherence check during insert: No. This means that corrupt objects can be created which can results in stability problems of the system.
      Exceptions: Automic fixed AFAIK two problems: Object names could be created with lowercase characters (20073558) and LOGIN could be created with empty host name. WP crashes at starttime (01272075).
    • After insert all WPs must refresh their client / object caches
    XML Import (AWI/JUI/Java API)
    • Data format: XML
    • DWP inserts objects in DB
    • Coherence check during insert.This means that normally no corrupt objects can be created.
    • WPs update their caches based on imported object types (one message per object type and WP)
      Thank you @Markus Embacher

    ------------------------------
    Automation Evangelist
    Fiducia & GAD IT AG
    ---
    Mitglied des deutschsprachigen Automic-Anwendervereins FOKUS e.V.
    Member of the German speaking Automic user association FOKUS e.V.
    ------------------------------



  • 8.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Sep 18, 2020 05:13 AM
    Edited by Michael A. Lowry Sep 22, 2020 10:29 AM
    I'd change the heading XML Import to AWI/JUI/Java API because all three of these interfaces use the same mechanism behind the scenes, ImportObject.




  • 9.  RE: AE REST API: Object import leads to massive performance problems / Community support required

    Posted Sep 15, 2020 05:24 AM
      |   view attached
    I got an update from support yesterday which seems to come from R&D. I asked for an example regarding the import of multiple objects within one JSON post.

    Please note the last sentence: The mentioned fix AE-24369 will only correct the double cache refresh but will not be a fundamental design change to a higher performance behavior (--> XML import). I think I will create a request in the black hole, uh Ideation Portal, for this. If someone of you is faster, please don't forget to write FOKUS in the title (also you Michael). :-)

    The REST endpoint /folderobjects was also mentioned by @Marcel Friedmann just two hours ago. Thank you for pointing this out!

    -------------------------------------------------------------------------------------------------------------------------
    [...]
    As mentioned within the incident - the rest endpoint must be used with care as the current implementation is working as the transport case and refreshing all caches - the customer can try to use the rest endpoint not with a single object - instead use all objects within one JSON (should not be thousands at the same time,...)

    the customer can try to use the POST https://hostname:port/ae/api/v1/<client>/folderobjects endpoint and can use a single JSON to import folder + multiple objects in one go (see attached example).

    At the moment the REST endpoint is not a replacement for the XML import as this is working with kick's instead of cache refreshes.

    To have the correct expectations this fix will only address the multiple cache refresh for a single object but not a design change to make the rest import as the XML import
    -------------------------------------------------------------------------------------------------------------------------
    ​​

    ------------------------------
    Automation Evangelist
    Fiducia & GAD IT AG
    ---
    Mitglied des deutschsprachigen Automic-Anwendervereins FOKUS e.V.
    Member of the German speaking Automic user association FOKUS e.V.
    ------------------------------

    Attachment(s)

    txt
    example_request.json.txt   16 KB 1 version