Automic Workload Automation

Expand all | Collapse all

Transport case unload/load vs. XML export/import

  • 1.  Transport case unload/load vs. XML export/import

    Posted 06-21-2016 10:30 AM

    Two approaches to moving objects between AE systems

    Transport case

    1. Select objects, e.g. via the Transport Case in the User Interface or the TransportObject API class.
    2. Unload objects using AE DB Unload program, ucybdbun.
    3. Change objects using AE DB Change program, ucybchng.
    4. Load objects using AE DB Load program, ucybdbld.

    XML

    1. Select objects, e.g., in the User Interface or using the FolderList API class.
    2. Export objects to XML file, e.g., via the Export function in the User Interface or the ExportObject API class.
    3. Change objects in XML file, e.g., using XSLT.
    4. Import objects from XML file, e.g., via the Import function in the User Interface or the ImportObject API class.

    Pros & Cons of each approach

     

    Advantages

    Disadvantages

     

     

    DB Unload/Load

    • Faster, because it goes straight to the database.
    • Does not put an additional load on the Automation Engine
    • Recommended by Automic for making mass changes.
    • No maximum file size.
    • No simultaneous unloads.
    • Transport case file format not backward-compatible.
    • No official documentation of transport case file format. (Unofficial documentation is available.)
    • Not all object attributes can be changed.
    • Loading objects into a different target folder is not straightforward.
    • ucybchng is not always reliable.
    • Bypassing the AE might be less safe.

     

     

    XML Export/Import

    • Simultaneous exports.
    • XML file format largely backward-compatible.
    • Easy to specify new target folder.
    • Fully documented, human-readable XML schema:
      • Change any object attribute.
      • Validate objects’ compliance with standards & conventions.
      • Present a side-by-side comparison of original & changed XML prior to deployment.
    • Easier to integrate with enterprise applications using Java APIs (e.g.,ExportObject & ImportObject).
    • Going through the AE might be safer.
    • No reliance on ucybchng.
    • Slower, because it goes through the AE server.
    • Puts a higher load on the AE (DWPs).
    • No vendor-supported way of changing exported XML files.
    • Automic no longer publishes documentation of the XML schema.
    • Maximum file size for imports: 30 MB.
    • Not recommended by Automic for making mass changes.

     

    Original post

    Our current system for promoting batches between staging environments is based on the transport case mechanism. For a while now, we have been looking into the possibility of moving away from this approach, and switching to an approach based on XML. Before we put too much effort into this transition, I thought it wise to ask the broader community for input.

    Questions

    1. Has anyone switched from using the transport case to XML for batch deployments? How did it go? What did you learn?
    2. Has anyone performed benchmarks to compare the relative performance of the two approaches, and to measure the load each places on the AE/DB?
    3. How might it be possible to mitigate the disadvantages of the XML approach?


  • 2.  Transport case unload/load vs. XML export/import

    Posted 06-21-2016 10:48 AM

    Hi Michael

    Actually there is a supported XML based transportation & change software (WF2 by System Partners). We're using it for quiet a while and have no big issues so far. The tool exports on a per-object-level, so maximum file sizes are no problem. I regularily export & import around 300 objects - I don't know if you consider this amount as "mass change".

    Mutation is either done using the WF2 tool-features itself or self-written sed / xslt scripts - like removing "modified"-dates for Subversion checkins.

    Regards
    Joel



  • 3.  Transport case unload/load vs. XML export/import

    Posted 06-21-2016 10:52 AM
    Joel Wiesmann wrote:
     

    Actually there is a supported XML based transportation & change software (WF2 by System Partners). We're using it for quiet a while and have no big issues so far. The tool exports on a per-object-level, so maximum file sizes are no problem. I regularily export & import around 300 objects - I don't know if you consider this amount as "mass change".

    Mutation is either done using the WF2 tool-features itself or self-written sed / xslt scripts - like removing "modified"-dates for Subversion checkins.

    Thanks for the reply, Joel. We have worked with System Partners before, for a migration project. We have been meaning to run a PoC based on the idea of using WF2 for this, but have not gotten around to it yet. Exporting one XML file per object is certainly a way around the file size limitation; but I wonder if it does not impose other costs.


  • 4.  Transport case unload/load vs. XML export/import

    Posted 06-22-2016 03:32 AM
    Hi,

    Most often we are exporting/importing objects:

    Select objects.
    1. Export objects to XML file(s).
    2. Change objects in XML.
    3. Import objects from XML.

    Less often we use this approach:
    1. Select objects.
    2. Unload objects using AE DB Unload program, ucybdbun.
    3. Change objects using an Editor like Notepad++
    4. Load objects using AE DB Load program, ucybdbld.



  • 5.  Transport case unload/load vs. XML export/import

    Posted 06-22-2016 09:03 AM
    I can comment on using XML for this type of thing.  Performance WILL be an issue.  Additionally, don't forget that for every upgrade - you now have an added requirement of analyzing the XML files and updating your home grown solution.  This caused delays for us in upgrading.  Upgrade isn't hard, but now you've got this added dependency.  Now our issue was resources - we had built a home grown solution, but then by the time the upgrade happened - we no longer had anyone to support it, so this may not be as big of a deal for you - just something to realize.  Automic can and will change the format of the XML files.  As new features are added, new tags are added, etc.  It's something you have to be aware of.

    I'm also curious as to your statement about ucybchg not being reliable.  What do you mean by this?


  • 6.  Transport case unload/load vs. XML export/import

    Posted 06-22-2016 09:08 AM
    I'm also curious as to your statement about ucybchg not being reliable.  What do you mean by this?
    See this post:
    Problems with AE DB Change program ucybchng in v11.2.1

    Since we started using the Automation Engine, ucybchng has been by far the single component that has caused us the most trouble. We have identified at least 15 bugs in the program since 2012. (The most recent bugs described in the aforementioned post appear to be new in v11. These bugs were not present in v9sp11.) Yes, the bugs are eventually fixed; but each one imposes a non-negligible cost on our operation.

    Moreover, despite the fact that Automic recommends using the transport case mechanism for mass exports & imports, the poor quality of the AE DB Change program in recent years suggests that the company is not seriously committed to this approach.


  • 7.  Transport case unload/load vs. XML export/import

    Posted 06-22-2016 09:25 AM
    Laura Albrecht wrote:

    I can comment on using XML for this type of thing.  Performance WILL be an issue. Additionally, don't forget that for every upgrade - you now have an added requirement of analyzing the XML files and updating your home grown solution.  This caused delays for us in upgrading.  Upgrade isn't hard, but now you've got this added dependency.
    I have already begun to prepare to run a set of benchmarks to compare the relative performance of (and AE/DB load imposed by) each approach. I want to have at least some idea of what the impact will be, before we commit to this transition.

    We are aware that the XML schema will change. We hope that Automic will continue to support importing XML files from previous versions (as it does with transport case files). Thanks for your reply.


  • 8.  Transport case unload/load vs. XML export/import

    Posted 12-12-2016 05:38 AM
    I updated the table of advantages and disadvantages with details about the backward-compatibility of the two file formats.


  • 9.  Transport case unload/load vs. XML export/import

    Posted 12-12-2016 05:42 AM
    Michael Lowry wrote:
    Laura Albrecht wrote:
    I can comment on using XML for this type of thing... don't forget that for every upgrade - you now have an added requirement of analyzing the XML files and updating your home grown solution.  This caused delays for us in upgrading.  Upgrade isn't hard, but now you've got this added dependency.
    ...
    We are aware that the XML schema will change. We hope that Automic will continue to support importing XML files from previous versions (as it does with transport case files). Thanks for your reply.
    LauraAlbrecht608310: I started a new discussion on XML schema changes between versions. Perhaps we can get an official list of changes from Automic. Failing that, we can compile our own list.


  • 10.  Transport case unload/load vs. XML export/import

    Posted 12-12-2016 07:34 AM

    Automation Engine v12 introduces the ability to use aByteBufferduring export/import of XML files.

    Has anyone used this? We would be interested to know whether this capability delivers significant performance gains. If so, it might mitigate the relative performance disadvantage of XML over the transport case, and make XML more suitable for mass exports/imports.


  • 11.  Transport case unload/load vs. XML export/import

    Posted 12-12-2016 08:12 AM

    Hi Michael

    Just curious - are you unhappy with the XML import performance or is it because of the Atomic recommendation of not using it for batch import / exports? I don't know how your system is setup, here we usually deploy during daytime and most objects are scheduled to start in the evening. So importing is not time critical and we also don't notice any extra load on the system.

    Regards
    Joel



  • 12.  Transport case unload/load vs. XML export/import

    Posted 12-12-2016 08:22 AM
    Joel Wiesmann wrote:
    Just curious - are you unhappy with the XML import performance or is it because of the Atomic recommendation of not using it for batch import / exports? I don't know how your system is setup, here we usually deploy during daytime and most objects are scheduled to start in the evening. So importing is not time critical and we also don't notice any extra load on the system.
    First, we notice the warnings in the documentation, including this one in the javadoc page for ImportObject:
    The importing and exporting functionality is not suitable for mass transports! Use the UC4 Transport Case instead for this purpose.
    Second, it seems clear that Automic would not have added the capability to use ByteBuffer if performance were not a concern. Perhaps this is the first step toward supporting mass exports/imports via XML.


  • 13.  Transport case unload/load vs. XML export/import

    Posted 12-12-2016 08:49 AM

    Hi Michael

    I heard that the import / export uses temporary tables in the database. This is possibly the reason why it's not suiteable for "mass" export/import. I could imagine that AWI is the reason for supporting bytebuffer. That way AWI does possibly not need to handle temporary files but could directly read / write to the web client.

    Anyway if the mass export/import is the reason it's even better.



  • 14.  Transport case unload/load vs. XML export/import

    Posted 12-20-2016 10:02 AM
    Hi Everyone,

    Michael_Lowry, I could be wrong, but I think the two methods added to ImportObject and ExportObject classes (addImportedBinary and getBinaryContent) are meant to handle the content of the newish "STORAGE" object during export and import (the binaries within those objects can be potentially numerous / large).

    I'm not personally super familiar with the transport case, but one aspect of the XML export / import we have seen a limitation on is the ability to retrieve dependent objects (even with the getObjectReferences method that was introduced in v11). 

    The Export with dependencies will work to an extent (it will export the JOBS within JOBP, the promptsets.. but it will not export objects referenced in the Pre/Post/Process tabs etc.)

    To alleviate that, we built a utility (see link below) that is able to resolve such "deep" dependencies (or simply the standard dependencies of the v11 export feature).

    it also allows for the export / import of the underlying folder structure of exported objects and to only export objects that different from a particular package of existing xml files.

    more info here:

    https://github.com/brendanSapience/UC4-Automic-AE-CLI-Binary-Repository/wiki/OBJECTS_Export_Import.jar

    i can provide source code and help if needed!

    bren
     


  • 15.  Transport case unload/load vs. XML export/import

    Posted 12-20-2016 12:29 PM
    Speaking of XML, we have just discovered a potential problem with ImportObject or with the way we have implemented it. We are seeing a big jump in DWP usage and database locks when we use import an XML file containing AE object definitions. It happens when we submit the import using the sendRequestAndWait() method of the Connection class. The XML files we are importing are tiny — just single DOCU objects. So I do not think these imports should impact the system so much. We are currently looking into it.

    brendan_sapience_automic: have you observed anything like this before? I would be glad to see the source code of your tool so that I can learn the proper way to use the import and export classes.


  • 16.  Transport case unload/load vs. XML export/import

    Posted 12-21-2016 12:56 PM
    Michael_Lowry: the source code & most of the logic for it is here:

    https://github.com/brendanSapience/UC4-Automic-AE-CLI-OBJECTS-Export-Import

    And it also uses components from the following 2 repos (components shared across all binaries of the CLI):

    https://github.com/brendanSapience/UC4-Automic-AE-CLI-Std-Common

    https://github.com/brendanSapience/UC4-Automic---Java-API-Framework-Simplified

    Feel free to PM me if it seems cryptic, i'd be happy to explain all of it after the holidays :).

    bren



  • 17.  Transport case unload/load vs. XML export/import

    Posted 01-20-2017 03:41 AM
    I updated the table of advantages and disadvantages to note that starting with v12, Automic no longer publishes documentation of the XML schema for AE export files.


  • 18.  Transport case unload/load vs. XML export/import

    Posted 02-01-2018 07:58 AM
    Hello,

    I have just come across this thread and wondered if Automic has a 'recommended' approach of object import automation?

    I have used the API to create a client, attach an agent to that client and create a user.

    In terms of getting the Automic objects into the empty client, any idea what the best method is? The API or the transport case?

    Any update/answers greatly appreciated...

    Cheers,

    Dan


  • 19.  Transport case unload/load vs. XML export/import

    Posted 02-01-2018 08:12 AM
    Hi Dan,

    I hope I got the question right.

    Basically both approaces (Transport Case and XML export) are recommended and supported by Automic. Michael_Lowry in the first beginning has listed all the pros and cons for each approach.

    Before you migrate data three questions come into my mind you should ask yourself before you perform the transport:

    1.) how many objects do I want to export?
    if there are only just a few objects (a specific amount we really can not give, since the objects very strong vary in their size e.g. a windows job containing a simple "dir" command and nothing else is "smaller" than a calendar containing 100s of entries and dependencies to other calendars which you might also have to transport then) you can use the .xml export since it is easy to handle, and with the Java API (if i understood right) with some java programming you can run this also.
    if there is a large amount to objects, better go with the transport case, it will be quicker, and can also be kind of automated using OS jobs on the machine your utilities are installed on.

    2.) do i want to manipulate the objects before I import them to the new client?
    in that case definetly using the transport case since using the changeutility (ucybchng) is the only supported way to perform changes on objects if they are exported. Changing the .xml file is very dangerous and should not be done.

    3.) do I want to export all the objects from a client to import to another?
    in that case, you can use a third possibility, the clientcopy tool. ucybdbcc, this is used for copy the whole content of a client to another empty client without the need of the transport case or the .xml export function. You can also perform a client copy over two separated systems (e.g. from DEV to PROD system), prerequisition for this is, that both systems are on the same major and minor version.

    I hope this helps you with your decisions.


  • 20.  Transport case unload/load vs. XML export/import

    Posted 02-01-2018 08:33 AM
    Cheers Harald,

    This is the current scenario we have ..

    There will be 2 sets of objects:

    Generic 'executable' objects - This set of objects (currently about 300) will be stored in some sort of version control software. The objects I guess will be stored as individual xml objects in the version control.  Upon creating a new automic client these objects will be 'fetched' from version control and imported into the new automic client.  Most objects are just simple unix jobs with very limited logic in the process tabs.

    Generic 'environment' objects - This set of objects will again be stored in version control and pre import will be passed through a powershell script (already written) to parse in some environment specific variables.  These objects (about 7) will then be imported into the new automic client.

    In terms of the above, any idea which would be the best solution to go with?

    Cheers,

    Dan


  • 21.  Transport case unload/load vs. XML export/import

    Posted 02-16-2018 04:32 AM
    My WorkflowCommander Powershell module would allow you to do that in plain Powershell without breaking / distributing the logic. It would use the XML way to do the import. You might import a complete folder containing single-XML files or also import one XML file containing several objects. Been using this method to import hundreds of objects at the same time. It even allows you to batch-mutate existing objects without XML-exporting.

    You can find code-examples here: http://workflowcommander.ch/produkt_wfc-core_en.html

    Get in touch with me if you'd like to have more information on that.