Automic Workload Automation

 View Only
Expand all | Collapse all

Transport case unload/load vs. XML export/import

  • 1.  Transport case unload/load vs. XML export/import

    Posted Jun 21, 2016 10:30 AM
    Edited by Michael A. Lowry Sep 26, 2019 11:13 AM

    Two approaches to moving objects between AE systems

    Transport case

    1. Select objects, e.g. via the Transport Case in the User Interface or the TransportObject API class.
    2. Unload objects using AE DB Unload program, ucybdbun.
    3. Change objects using AE DB Change program, ucybchng.
    4. Load objects using AE DB Load program, ucybdbld.

    XML

    1. Select objects, e.g., in the User Interface or using the FolderList API class.
    2. Export objects to XML file, e.g., via the Export function in the User Interface or the ExportObject API class.
    3. Change objects in XML file, e.g., using XSLT.
    4. Import objects from XML file, e.g., via the Import function in the User Interface or the ImportObject API class.

    Pros & Cons of each approach

     

    Advantages

    Disadvantages

     

     

    DB Unload/Load

    • Faster, because it goes straight to the database.
    • Does not put an additional load on the Automation Engine
    • Recommended by Automic for making mass changes.
    • No maximum file size.

     

     

    XML Export/Import

    • Simultaneous exports.
    • XML file format largely backward-compatible.
    • Easy to specify new target folder.
    • Fully documented, human-readable XML schema:
      • Change any object attribute.
      • Validate objects’ compliance with standards & conventions.
      • Present a side-by-side comparison of original & changed XML prior to deployment.
    • Easier to integrate with enterprise applications using Java APIs (e.g.,ExportObject & ImportObject).
    • Going through the AE might be safer.
    • Object searches not impacted.
    • No reliance on ucybchng.
    • Slower, because it goes through the AE server.
    • Puts a higher load on the AE (DWPs).
    • No vendor-supported way of changing exported XML files.
    • Automic no longer publishes documentation of the XML schema.
    • Maximum file size for imports: 30 MB.
    • Not recommended by Automic for making mass changes.

     

    Original post

    Our current system for promoting batches between staging environments is based on the transport case mechanism. For a while now, we have been looking into the possibility of moving away from this approach, and switching to an approach based on XML. Before we put too much effort into this transition, I thought it wise to ask the broader community for input.

    Questions

    1. Has anyone switched from using the transport case to XML for batch deployments? How did it go? What did you learn?
    2. Has anyone performed benchmarks to compare the relative performance of the two approaches, and to measure the load each places on the AE/DB?
    3. How might it be possible to mitigate the disadvantages of the XML approach?


  • 2.  Transport case unload/load vs. XML export/import

    Posted Jun 21, 2016 10:48 AM

    Hi Michael

    Actually there is a supported XML based transportation & change software (WF2 by System Partners). We're using it for quiet a while and have no big issues so far. The tool exports on a per-object-level, so maximum file sizes are no problem. I regularily export & import around 300 objects - I don't know if you consider this amount as "mass change".

    Mutation is either done using the WF2 tool-features itself or self-written sed / xslt scripts - like removing "modified"-dates for Subversion checkins.

    Regards
    Joel



  • 3.  Transport case unload/load vs. XML export/import

    Posted Jun 21, 2016 10:52 AM
    Joel Wiesmann wrote:
     

    Actually there is a supported XML based transportation & change software (WF2 by System Partners). We're using it for quiet a while and have no big issues so far. The tool exports on a per-object-level, so maximum file sizes are no problem. I regularily export & import around 300 objects - I don't know if you consider this amount as "mass change".

    Mutation is either done using the WF2 tool-features itself or self-written sed / xslt scripts - like removing "modified"-dates for Subversion checkins.

    Thanks for the reply, Joel. We have worked with System Partners before, for a migration project. We have been meaning to run a PoC based on the idea of using WF2 for this, but have not gotten around to it yet. Exporting one XML file per object is certainly a way around the file size limitation; but I wonder if it does not impose other costs.


  • 4.  Transport case unload/load vs. XML export/import

    Posted Jun 22, 2016 03:32 AM
    Hi,

    Most often we are exporting/importing objects:

    Select objects.
    1. Export objects to XML file(s).
    2. Change objects in XML.
    3. Import objects from XML.

    Less often we use this approach:
    1. Select objects.
    2. Unload objects using AE DB Unload program, ucybdbun.
    3. Change objects using an Editor like Notepad++
    4. Load objects using AE DB Load program, ucybdbld.



  • 5.  Transport case unload/load vs. XML export/import

    Posted Jun 22, 2016 09:03 AM
    I can comment on using XML for this type of thing.  Performance WILL be an issue.  Additionally, don't forget that for every upgrade - you now have an added requirement of analyzing the XML files and updating your home grown solution.  This caused delays for us in upgrading.  Upgrade isn't hard, but now you've got this added dependency.  Now our issue was resources - we had built a home grown solution, but then by the time the upgrade happened - we no longer had anyone to support it, so this may not be as big of a deal for you - just something to realize.  Automic can and will change the format of the XML files.  As new features are added, new tags are added, etc.  It's something you have to be aware of.

    I'm also curious as to your statement about ucybchg not being reliable.  What do you mean by this?


  • 6.  Transport case unload/load vs. XML export/import

    Posted Jun 22, 2016 09:08 AM
    I'm also curious as to your statement about ucybchg not being reliable.  What do you mean by this?
    See this post:
    Problems with AE DB Change program ucybchng in v11.2.1

    Since we started using the Automation Engine, ucybchng has been by far the single component that has caused us the most trouble. We have identified at least 15 bugs in the program since 2012. (The most recent bugs described in the aforementioned post appear to be new in v11. These bugs were not present in v9sp11.) Yes, the bugs are eventually fixed; but each one imposes a non-negligible cost on our operation.

    Moreover, despite the fact that Automic recommends using the transport case mechanism for mass exports & imports, the poor quality of the AE DB Change program in recent years suggests that the company is not seriously committed to this approach.


  • 7.  Transport case unload/load vs. XML export/import

    Posted Jun 22, 2016 09:25 AM
    Laura Albrecht wrote:

    I can comment on using XML for this type of thing.  Performance WILL be an issue. Additionally, don't forget that for every upgrade - you now have an added requirement of analyzing the XML files and updating your home grown solution.  This caused delays for us in upgrading.  Upgrade isn't hard, but now you've got this added dependency.
    I have already begun to prepare to run a set of benchmarks to compare the relative performance of (and AE/DB load imposed by) each approach. I want to have at least some idea of what the impact will be, before we commit to this transition.

    We are aware that the XML schema will change. We hope that Automic will continue to support importing XML files from previous versions (as it does with transport case files). Thanks for your reply.


  • 8.  Transport case unload/load vs. XML export/import

    Posted Dec 12, 2016 05:38 AM
    I updated the table of advantages and disadvantages with details about the backward-compatibility of the two file formats.


  • 9.  Transport case unload/load vs. XML export/import

    Posted Dec 12, 2016 05:42 AM
    Michael Lowry wrote:
    Laura Albrecht wrote:
    I can comment on using XML for this type of thing... don't forget that for every upgrade - you now have an added requirement of analyzing the XML files and updating your home grown solution.  This caused delays for us in upgrading.  Upgrade isn't hard, but now you've got this added dependency.
    ...
    We are aware that the XML schema will change. We hope that Automic will continue to support importing XML files from previous versions (as it does with transport case files). Thanks for your reply.
    LauraAlbrecht608310: I started a new discussion on XML schema changes between versions. Perhaps we can get an official list of changes from Automic. Failing that, we can compile our own list.


  • 10.  Transport case unload/load vs. XML export/import

    Posted Dec 12, 2016 07:34 AM

    Automation Engine v12 introduces the ability to use aByteBufferduring export/import of XML files.

    Has anyone used this? We would be interested to know whether this capability delivers significant performance gains. If so, it might mitigate the relative performance disadvantage of XML over the transport case, and make XML more suitable for mass exports/imports.


  • 11.  Transport case unload/load vs. XML export/import

    Posted Dec 12, 2016 08:12 AM

    Hi Michael

    Just curious - are you unhappy with the XML import performance or is it because of the Atomic recommendation of not using it for batch import / exports? I don't know how your system is setup, here we usually deploy during daytime and most objects are scheduled to start in the evening. So importing is not time critical and we also don't notice any extra load on the system.

    Regards
    Joel



  • 12.  Transport case unload/load vs. XML export/import

    Posted Dec 12, 2016 08:22 AM
    Joel Wiesmann wrote:
    Just curious - are you unhappy with the XML import performance or is it because of the Atomic recommendation of not using it for batch import / exports? I don't know how your system is setup, here we usually deploy during daytime and most objects are scheduled to start in the evening. So importing is not time critical and we also don't notice any extra load on the system.
    First, we notice the warnings in the documentation, including this one in the javadoc page for ImportObject:
    The importing and exporting functionality is not suitable for mass transports! Use the UC4 Transport Case instead for this purpose.
    Second, it seems clear that Automic would not have added the capability to use ByteBuffer if performance were not a concern. Perhaps this is the first step toward supporting mass exports/imports via XML.


  • 13.  Transport case unload/load vs. XML export/import

    Posted Dec 12, 2016 08:49 AM

    Hi Michael

    I heard that the import / export uses temporary tables in the database. This is possibly the reason why it's not suiteable for "mass" export/import. I could imagine that AWI is the reason for supporting bytebuffer. That way AWI does possibly not need to handle temporary files but could directly read / write to the web client.

    Anyway if the mass export/import is the reason it's even better.



  • 14.  Transport case unload/load vs. XML export/import

    Posted Dec 20, 2016 10:02 AM
    Hi Everyone,

    Michael_Lowry, I could be wrong, but I think the two methods added to ImportObject and ExportObject classes (addImportedBinary and getBinaryContent) are meant to handle the content of the newish "STORAGE" object during export and import (the binaries within those objects can be potentially numerous / large).

    I'm not personally super familiar with the transport case, but one aspect of the XML export / import we have seen a limitation on is the ability to retrieve dependent objects (even with the getObjectReferences method that was introduced in v11). 

    The Export with dependencies will work to an extent (it will export the JOBS within JOBP, the promptsets.. but it will not export objects referenced in the Pre/Post/Process tabs etc.)

    To alleviate that, we built a utility (see link below) that is able to resolve such "deep" dependencies (or simply the standard dependencies of the v11 export feature).

    it also allows for the export / import of the underlying folder structure of exported objects and to only export objects that different from a particular package of existing xml files.

    more info here:

    https://github.com/brendanSapience/UC4-Automic-AE-CLI-Binary-Repository/wiki/OBJECTS_Export_Import.jar

    i can provide source code and help if needed!

    bren
     


  • 15.  Transport case unload/load vs. XML export/import

    Posted Dec 20, 2016 12:29 PM
    Speaking of XML, we have just discovered a potential problem with ImportObject or with the way we have implemented it. We are seeing a big jump in DWP usage and database locks when we use import an XML file containing AE object definitions. It happens when we submit the import using the sendRequestAndWait() method of the Connection class. The XML files we are importing are tiny — just single DOCU objects. So I do not think these imports should impact the system so much. We are currently looking into it.

    brendan_sapience_automic: have you observed anything like this before? I would be glad to see the source code of your tool so that I can learn the proper way to use the import and export classes.


  • 16.  Transport case unload/load vs. XML export/import

    Posted Dec 21, 2016 12:56 PM
    Michael_Lowry: the source code & most of the logic for it is here:

    https://github.com/brendanSapience/UC4-Automic-AE-CLI-OBJECTS-Export-Import

    And it also uses components from the following 2 repos (components shared across all binaries of the CLI):

    https://github.com/brendanSapience/UC4-Automic-AE-CLI-Std-Common

    https://github.com/brendanSapience/UC4-Automic---Java-API-Framework-Simplified

    Feel free to PM me if it seems cryptic, i'd be happy to explain all of it after the holidays :).

    bren



  • 17.  Transport case unload/load vs. XML export/import

    Posted Jan 20, 2017 03:41 AM
    I updated the table of advantages and disadvantages to note that starting with v12, Automic no longer publishes documentation of the XML schema for AE export files.


  • 18.  Transport case unload/load vs. XML export/import

    Posted Feb 01, 2018 07:58 AM
    Hello,

    I have just come across this thread and wondered if Automic has a 'recommended' approach of object import automation?

    I have used the API to create a client, attach an agent to that client and create a user.

    In terms of getting the Automic objects into the empty client, any idea what the best method is? The API or the transport case?

    Any update/answers greatly appreciated...

    Cheers,

    Dan


  • 19.  Transport case unload/load vs. XML export/import

    Posted Feb 01, 2018 08:12 AM
    Hi Dan,

    I hope I got the question right.

    Basically both approaces (Transport Case and XML export) are recommended and supported by Automic. Michael_Lowry in the first beginning has listed all the pros and cons for each approach.

    Before you migrate data three questions come into my mind you should ask yourself before you perform the transport:

    1.) how many objects do I want to export?
    if there are only just a few objects (a specific amount we really can not give, since the objects very strong vary in their size e.g. a windows job containing a simple "dir" command and nothing else is "smaller" than a calendar containing 100s of entries and dependencies to other calendars which you might also have to transport then) you can use the .xml export since it is easy to handle, and with the Java API (if i understood right) with some java programming you can run this also.
    if there is a large amount to objects, better go with the transport case, it will be quicker, and can also be kind of automated using OS jobs on the machine your utilities are installed on.

    2.) do i want to manipulate the objects before I import them to the new client?
    in that case definetly using the transport case since using the changeutility (ucybchng) is the only supported way to perform changes on objects if they are exported. Changing the .xml file is very dangerous and should not be done.

    3.) do I want to export all the objects from a client to import to another?
    in that case, you can use a third possibility, the clientcopy tool. ucybdbcc, this is used for copy the whole content of a client to another empty client without the need of the transport case or the .xml export function. You can also perform a client copy over two separated systems (e.g. from DEV to PROD system), prerequisition for this is, that both systems are on the same major and minor version.

    I hope this helps you with your decisions.


  • 20.  Transport case unload/load vs. XML export/import

    Posted Feb 01, 2018 08:33 AM
    Cheers Harald,

    This is the current scenario we have ..

    There will be 2 sets of objects:

    Generic 'executable' objects - This set of objects (currently about 300) will be stored in some sort of version control software. The objects I guess will be stored as individual xml objects in the version control.  Upon creating a new automic client these objects will be 'fetched' from version control and imported into the new automic client.  Most objects are just simple unix jobs with very limited logic in the process tabs.

    Generic 'environment' objects - This set of objects will again be stored in version control and pre import will be passed through a powershell script (already written) to parse in some environment specific variables.  These objects (about 7) will then be imported into the new automic client.

    In terms of the above, any idea which would be the best solution to go with?

    Cheers,

    Dan


  • 21.  Transport case unload/load vs. XML export/import

    Posted Feb 16, 2018 04:32 AM
    My WorkflowCommander Powershell module would allow you to do that in plain Powershell without breaking / distributing the logic. It would use the XML way to do the import. You might import a complete folder containing single-XML files or also import one XML file containing several objects. Been using this method to import hundreds of objects at the same time. It even allows you to batch-mutate existing objects without XML-exporting.

    You can find code-examples here: http://workflowcommander.ch/produkt_wfc-core_en.html

    Get in touch with me if you'd like to have more information on that.


  • 22.  RE: Transport case unload/load vs. XML export/import
    Best Answer

    Posted Sep 06, 2019 06:20 PM
    Good info.
    The docs for v12 still say that its not recommended to use import for 'mass' objects.

    But it doesn't define 'mass' - 
    Is it 20 objects or 2000?
    What if i run 3(or 10) DWPs on all 3 of my engine hosts to support this? Perhaps the DB is the bottleneck.
    Whats the impact on the users?

    Has anyone see a performance difference importing 2000 individual XML files, vs 1 large XML file containing all 2000 objects?

    I happen to want to import multiple different sets of 2000 objects concurrently.
    Perhaps updating 20,000 unique objects within a 4 hour period, on a weekly basis.
    Sounds like trouble right?!

    From the docs - 
    "Importing/exporting might cause a heavy workload for the Dialog Work Processes (DWP). as the Transport Case loads directly to the Database,     this is the better choice if you want to transfer large amounts of data."
    Do not use IMPORT for mass transport. Use the Transport Case when you need to move large amounts of content.
    https://docs.automic.com/documentation/webhelp/english/ALL/components/DOCU/12.2/AWA%20Guides/help.htm#Script/Reference/IMPORT.htm?Highlight=import


  • 23.  RE: Transport case unload/load vs. XML export/import

    Posted Sep 13, 2019 03:25 PM
    Another Q - 
    In the AWI if you go Admin tab > clients > monitoring > report, youll see a client specific report that contains import information as below.
    If you click directory in youll see a file name like 1234567.CLNT.txt and can download it.

    Does anyone know what table contains this data?
    These Ucodes U04005759  etc are not in the usual Automation engine logs, or any log AFAIK.
    I want to pull stats on the number of imports per day programatically.

    2019-09-13 10:52:14 - U04005755 Starting import of object 'QA.SCRIPT.X.'
    2019-09-13 10:52:14 - U04005756 Object 'QA.SCRIPT.X.' already exists and will be replaced.
    2019-09-13 10:52:14 - U04005759 Object 'QA.SCRIPT.X.' has been imported.
    2019-09-13 10:52:15 - U04002574 Import setting 'OVERWRITE_OBJECT': 1

    Thanks!


  • 24.  RE: Transport case unload/load vs. XML export/import

    Posted Sep 13, 2019 06:40 PM
    Edited by Pete Wirfs Sep 13, 2019 06:41 PM
    I think I found them in RT;

    select *
    from RT, AH
    where RT_Type = 'CLNT'
    and RT_MsgNr in (4005759)
    and ah_idnr = rt_ah_idnr

    ------------------------------
    Pete
    ------------------------------



  • 25.  RE: Transport case unload/load vs. XML export/import

    Posted Sep 18, 2019 12:30 PM
    Nice, thanks Pete!


  • 26.  RE: Transport case unload/load vs. XML export/import

    Posted Sep 18, 2019 12:42 PM
    Nice query to have for audit purposes.  However AH_USR_Idnr seems to always be set to "0"?   Please let me know if you ever figure out how to find "who" did the manual import.

    ------------------------------
    Pete
    ------------------------------



  • 27.  RE: Transport case unload/load vs. XML export/import

    Posted Oct 08, 2019 11:41 AM
    Edited by Michael A. Lowry Oct 08, 2019 11:41 AM
    I have a quick update on this topic. We have almost completed the migration from our old 'UC4 Toolkit' solution based on the transport case file format to a new app that used the XML file format. The two solutions have been running side-by-side for several months, and we have been migrating users from the old tool to the new one. Next week we plan to shut down UC4 Toolkit, leaving the new XML-based app as the only way for users to promote AE batches between environments.

    It has been a long journey, and we have run into several obstacles along the way. However, I am confident we made the right decision.

    The transport case format has never been officially documented. And sadly, despite several promises from Automic/CA/Broadcom, no documentation was ever provided. I did manage to reverse engineer the format, but it was never easy to work with. The XML file format, although no longer documented, is at least transparent enough to make sense of. It's also much easier to work with. There are myriad off-the-shelf tools for working with XML, and this has made development much more straightforward.

    The new solution is written entirely in Java, and has no dependencies on command-line 'utilitiy' tools from Broadcom. In particular, we will be glad to say goodbye to the AE DB Change program, ucybchng. It has been a thorn in our side for years. It was always at least a version behind in terms of capabilities, and each bug fix risked breaking things that worked or changing behaviors in unexpected ways. We are now in control of the transformation engine - a tool we wrote in-house. The trickiest part was reproducing the behavior of the OBJECT_USE rule - a change rule type that updates object references. There are many, many places in the AE XML schema that can contain references to AE objects, and covering most of them took quite a bit of trial and error.

    Now we look forward to leaving the old tool behind and committing 100% to the new approach.

    How about the rest of you? Do you still use the transport case, or have you also migrated to an XML-based solution?


  • 28.  RE: Transport case unload/load vs. XML export/import

    Posted Oct 09, 2019 04:40 AM
    ​Hi Michael.

    We use transport case for backups, for bulk propagation of objects between environments and whenever we need to make bulk changes to properties that TC is able to change (e.g. when retiring an agent used in multiple jobs via ucybchng).

    A predecessor of mine had automated the TC jobs to run every night, so users could propagate objects between environments. Since that once led to a wiped environment due to a (presumed bug), I disabled that automation in my time. Using TC to propagate objects between test and prod environments now requires users to explicitly request this, I then run the transfer on demand. Since that day, I also monitor the transport case's tables in the database to ensure they don't contain nonsensical amounts of entries.

    We allow all users to use XML import/export for all purposes using the AWI context menu. We also use XML export in place of TC when we have different versions in prod and test, and TC sometimes (but not always) refuses to work then. XML also sometimes refuses to work in that case, but we've always been able to make it work by manually editing the XML header's version flag with no ill effects.

    I myself never had a great impression of TC, mostly due to these design decisions and quirks:

    1. I never logically understood why a "transport" mechanism should be invoked for backups and archiving

    2. Initially and/or with new colleagues in the mix, the terminology of "unload" and "load" is often times confusing, doubly so when discussed across language barriers, since you can mentally apply these terms to the TC or to the database, which totally changes their respective meaning.

    n.b. I suspect that whole TC concept probably dates back to the Windows 95 transport case used for file synchronisation. Much like Clippy, Microsoft quickly abandoned that ill-fated representation.

    3. ucybchng is limited in what fields it can change. With an open and at the same time sensible format, this would not neccessarily have been the (no pun intended) case.

    4. TC always wipes the version history of any objects. So whether you "bulk transport" or "bulk transform", you always lose the old version info. This is a major drawback.

    Regardless, we'll probably stick with TC for the time being, using XML as a fallback. We didn't yet have any major destructive events with ucybchng, the limited things it does, it so far did reliably.

    Best,
    Carsten


  • 29.  RE: Transport case unload/load vs. XML export/import

    Posted Oct 09, 2019 08:11 AM
    I now work somewhere else and so here, we opted to go with the Transport Management System (TMS).  This can use either mechanism (export/import or transport case), but we use the transport case.  It facilitates being able to create multiple transport cases or "packages" and then deploy them to the other environments.  We abstract everything here, so there's nothing to change between the environments.  So far no issues.

    ------------------------------
    Enterprise Scheduling Lead
    Takeda
    ------------------------------