Automic Workload Automation

 View Only
  • 1.  SearchObject API class returns incomplete list of objects after run of ucybdbld

    Posted Feb 07, 2018 08:25 AM

    As most of you know, we use an [DEAD LINK https://community.automic.com/discussion/comment/31392/#Comment_31392]automated batch deployment system, to promote batches (folders) of AE objects from one system to another (e.g., from TEST to PROD). One of the steps in batch deployment involves removing objects from the target folder that were not part of the deployment. In broad strokes, it does this in the following way:

    1. Get list of objects in target folder, using SearchObject Java API class.
    2. Load transport case file into target system.
    3. Remove objects from target folder that were not in the transport case file.
    4. Run SearchObject again to identify folders under the target folder that no longer have any child objects.
    5. Remove these empty folders in a bottom-up fashion.

    This is an oversimplified description of the process, but it will serve for the purposes of this discussion.

    We discovered today that on rare occasions, SearchObject will return an incomplete list of objects. In our case, it happens in step 4. (And because the list returned omits some objects, the deployment program sometimes mistakenly thinks that some folders are empty when they are not. It then removes these folders, and the child objects end up in <No Folder>.)

    We suspect that this problem is likely to happen when these circumstances are present:

    • SearchObject is run shortly after a transport case file has been loaded.
    • The objects loaded reside in folders that are in the scope of the SearchObject search.
    • The number of objects loaded is high (several hundred or more).
    • The AE system is busy.

    It seems possible that ucybdbld is completing before all of the database changes (e.g., changes to the OFS table) have been committed. I have reported the problem to Automic in INC00219765. I will update this thread when I have more information.



  • 2.  SearchObject API class returns incomplete list of objects after run of ucybdbld

    Posted Mar 05, 2018 05:48 AM

    Automic Development provided a response. I will paraphrase it here.

    The AE DB Load program (ucybdbld) loads objects in a single database transaction. When the ucybdbld program exits with return code 0, this means all of the objects were successfully loaded into the target AE system. Work processes, including the DWPs that handle Java API requests, maintain an object cache that is used to process SearchObject operations. When the DB Load program loadss objects it notifies the WPs that they need to refresh their caches. It does so by adding entries to the internal task list (ITL) table. If a SearchObject request is processed between the the DB Load and the refresh of the object cache, the results may be incomplete or incorrect.

    Error-prone approach

    1. Run the DB load.ucybdbld loads objects & writes corresponding entries to ITL table.
    2. Run the SearchObject request.


    This approach is error-prone, because the following two steps may execute either before or after the SearchObject request is handled:

    • DWP checks for & finds entries in the ITL table.
    • DWP updates its object cache & removes entries from the ITL table.


    If you’re lucky, it will run like this:

    1. Run the DB load.ucybdbld loads objects & writes corresponding entries to ITL table.
    2. DWP checks for & finds entries in the ITL table.
    3. DWP updates its object cache & removes entries from the ITL table.
    4. Run the SearchObject request.


    However, if you’re unlucky, it will run like this:

    1. Run the DB load.ucybdbld loads objects & writes corresponding entries to ITL table.
    2. Run the SearchObject request.
    3. DWP checks for & finds entries in the ITL table.
    4. DWP updates its object cache & removes entries from the ITL table.


    To ensure that the DWPs have an up-to-date object cache when the SearchObject request is sent, it is necessary to insert an extra step in the process.

    More reliable approach

    1. Run the DB load program, loading objects & writing corresponding entries to ITL table.
    2. Check the ITL table for entries. If the table contains any entries, wait a few seconds and try again. Wait until the table is empty. This ensures that the following steps have been completed:
      • DWP checks for & finds entries in the ITL table.
      • DWP updates its object cache & removes entries from the ITL table.
    3. Once the ITL table is empty, Run the SearchObject request.


    Automic confirmed that because of the design of the AE, there is no way to guarantee correct results from SearchObject using the Java APIs alone. It is necessary to insert this additional step that performs a direct query of the AE DB. In very busy systems, where multiple DB loads may run simultaneously or in quick succession, it is conceivable that another DB load is run between the ITL check and the execution of SearchObject. So really, there is no 100% foolproof way to guarantee correct results from SearchObject.

    However, in most cases, as long as SearchObject is executed quickly after the ITL check, it should return correct results. Preventing multiple simultaneous DB loads, and better yet, adding a delay between DB loads, will also help mitigate the problem. (See also this discussion: Poor AE performance following DB load of transport case.)

    This problem raises the more general topic of API completeness & reliability. To get some clarity on the topic, I have posed these questions to Automic:

    1. What API operations may not always return correct results?
    2. In which circumstances are these APIs likely to return incorrect results?
    3. What steps can be taken to mitigate this?


    I will post an update here when I have answers to these questions.



  • 3.  SearchObject API class returns incomplete list of objects after run of ucybdbld

    Posted Mar 07, 2018 08:50 AM
    Updating the process to perform SQL queries will be a bit involved, so in the meantime I am going to add a delay between the DB load and the object clean-up steps. Does anyone know how often the DWPs update their caches? Is this configurable? Do they do it based on kicks, at specific intervals, or a combination of both?


  • 4.  SearchObject API class returns incomplete list of objects after run of ucybdbld

    Posted Mar 15, 2018 08:07 AM
    Yesterday morning, I updated our [DEAD LINK https://community.automic.com/discussion/comment/31392/#Comment_31392]batch deployment system so that it waits 15 seconds after performing the DB load, but before continuing with the post-deployment cleanup actions (where SearchObject is used a couple of times). Since making this change, the problem has not happened. Prior to making this change, the problem was happening several times per day.

    Note that we already disallow simultaneous DB loads to work around another AE limitation. Our experience suggests that the AE works best when DB loads and related changes are serialized and separated in time.


  • 5.  SearchObject API class returns incomplete list of objects after run of ucybdbld

    Posted Mar 15, 2018 11:43 AM

    Well, apparently 15 seconds was not sufficient. The problem happened again about an hour ago. I have increased the delay to 30 seconds. I have also asked Automic in the incident for answers to the questions in comment 36662, above.

    Oh and by the way, there is a related problem, documented in PRB00220244, that can cause SearchObject to return the following error when listing the contents of a folder that does exist:

    U00011700 Cannot find information for path '&01'.

    I'm guessing the root cause of the two problems is the same.



  • 6.  SearchObject API class returns incomplete list of objects after run of ucybdbld

    Posted Mar 19, 2018 05:45 AM

    I opened an enhancement request about this, asking Automic to improve the Java APIs:

    AE Java APIs — eliminate requirement to run direct DB calls to obtain correct results

    Some AE Java API operations do not reliably return correct and up-to-date results. For example, SearchObject may return incorrect or incomplete results if the DWP handling the  SearchObject request has a stale object cache. To work around this  limitation, Automic currently recommends running an SQL SELECT against  the ITL table prior to running any SearchObject operation: SearchObject  be relied upon to return correct results only if the ITL table is  currently empty.

    The AE Java Application Interface should be able  to provided guaranteed correct results without relying on direct called  to the DB. This applies not only to SearchObject, but also to  FolderTree, FolderList, ActivityList, ObjectStatistics, and all other  API classes. In other words, it should be possible to build an reliable,  production-quality application whose only interface to the Automation  Engine is the Application Interface.

    If you like it, please vote for it.