Automic Workload Automation

Expand all | Collapse all

Transport case parsing scripts

  • 1.  Transport case parsing scripts

    Posted 05-17-2017 08:05 AM

    Notice: I have moved this information to a separate document. All future updates will be made there:

    Transport case parsing scripts 

     

    Since deciphering and documenting the transport case file format, I have written a few scripts to parse such files, and extract various pieces of information.

     

    ScriptDescription & usageVersion / Last updated
    18bad3g2pytw.pnghttps://us.v-cdn.net/5019921/uploads/editor/gd/18bad3g2pytw.png" width="64">
    parse_ae_db_file.sh
    (Previously known as parse_transport_case.sh)

    General script for parsing Automation Engine database files. The following file types are supported:

    Usage:

    parse_ae_db_file.sh
    -o output_mode [-t y|n] [-v verbosity] input_file

    Arguments:

    -o output_mode:

    • a: all object details including scripting tab content & list of workflow tasks
    • c: only objects with matching custom attributes
    • i: Basic object information: OH_IDnr, OH_Name, OH_OType, etc.
    • p: scripting tab content (pre-process, process, and post-process)
    • s: standard object details
    • w: workflow tasks
    • z: z/OS job details

    -t table_descriptions:

    • n: do not read table descriptions (default)
    • y: read table descriptions (much slower)

    -v verbosity:

    • 0: no messages, only output (default)
    • 1 :slightly verbose
    • 2: more verbose
    • 3: extremely verbose

    The script can also be further customized to extract other information.

    v2.4
    2019.04.11
    18bad3g2pytw.pngparse_JBA_Rest.sh
    Parses and prints details from the JBA_Rest field of z/OS jobs.
    parse_JBA_Rest.sh JBA_Rest_input_file
    The input file should contain one line per job, with tab-delimited values:
    JOBS_ZOS OH_Name OH_OType OH_HostAttrTypeDst JBA_Rest
    (These lines are included in the output of the parse_transport_case.sh script when it is run with output mode a or z).
    v0.2
    2015.01.27
    18bad3g2pytw.pngvalidate_obj_script.sh
    Parse & validate scripting tab content of relevant AE objects.
    validate_obj_script.sh -p prefix [-v verbosity] ae_obj_script_file
    AE_OBJ_SCRIPT OH_Name OH_OType OT_Type OT_Lnr OT_Content
    (This corresponds to the AE_OBJ_SCRIPT lines output by the parse_transport_case.sh script when it is run with output mode a.)
    v0.4
    2017.06.28
    18bad3g2pytw.pngparse_zOS_job_script.sh
    Parse and validate scripting tab content of z/OS jobs of type Automation Engine.
    parse_zOS_job_script.sh zos_job_script_file
    The input file should contain one line per job script line, with tab-delimited values:
    JOBS_ZOS_SCRIPT zOS_Job_Type OH_Name OT_Type OT_Lnr OT_Content
    (This corresponds to the JOBS_ZOS_SCRIPT lines output by the parse_transport_case.sh script when it is run with output mode a or z.)
    v0.1
    2017.06.19
    18bad3g2pytw.pnglogging_functions.sh
    Logging functions used in other scripts. Not intended to be executed directly.NA


  • 2.  Transport case parsing scripts

    Posted 06-19-2017 08:41 AM

    I posted a new script, parse_zOS_job_script.sh. This script parses the output of parse_zOS_jobs_in_transport_case.sh, and allows validation of scripting tab content for z/OS jobs. We have adopted this approach to block transport/deployment of jobs containing specific AE scripting commands, such as :INCLUDE or :PUT_ATT.

    The first field of the input file is just a label identifying lines that contain scripting tab content.

    The second field of the input file is the z/OS job type:

    z/OS job typeDescription
    1Automation Engine (AE)
    2z/OS JCL
    3JCL incl. z/OS job card
    4AE JCL (undocumented)


    The third field contains the name of the job.

    The fourth field is the OT_Type:

    OT_TypeDescription
    0Process
    1Pre-process
    2Post-process
    3Child post-process (Not applicable to z/OS jobs)


    The fifth field of the input file, OT_Lnr, is just the line number within a particular scripting tab.

    The sixth field of the input file contains the content of the scripting line, OT_Content.

    The type of line is identified based on the first character of the scripting tab line:

    First characterLine type
    !AE comment
    :AE scripting
    Anything elsez/OS JCL


    It would be relatively straightforward to adapt this script to handle job types other than z/OS, or to do more detailed examination of scripting lines containing other specific content.



  • 3.  Transport case parsing scripts

    Posted 06-26-2017 08:33 AM
    Soon after I posted the script for parsing the script content of z/OS jobs, a new related requirement became apparent:
    • Validate that allACTIVATE_UC_OBJECTscripting statements point only to objects the user is authorized to execute.
    This necessitated rewriting the parse_transport_case.sh script so that it is capable of printing all scripting lines of objects. The updated parse_transport_case.sh (version 0.7) is now attached to the original post. Here is a list of changes:
    • Combined with parse_zOS_jobs_in_transport_case.sh script because the scripts share a lot of code. (Because its function is now included in parse_transport_case.sh, the parse_zOS_jobs_in_transport_case.sh script has now been deprecated.)
    • Factored out individual portions of readOtherTableRecords & printOutput to other more table-specific functions.
    • Added capability to print scripting tab lines of relevant object types (executable objects + JOBI).
    • Added support for O record followed by S record.
    • Factored out repeated statements.
    The new script also adds one mandatory and one optional command line flag. The updated syntax description follows:
    parse_transport_case.sh -o output_mode [-v verbosity] transport_case_file
    Where output_mode is one of:
    • a: all object details including scripting tab content
    • c: only objects with matching custom attributes
    • s: standard object details
    • z: z/OS job details
    and verbosity is one of:
    • 0: no messages, only output
    • 1: slightly verbose
    • 2: more verbose
    • 3: extremely verbose
    I am working on a new script, validate_obj_script.sh, that can be used to validate the content of object scripting tabs. I will post it when it is ready.


  • 4.  Transport case parsing scripts

    Posted 06-28-2017 11:22 AM
    I have added the new validate_obj_script.sh script I wrote to parse & validate the content of AE object scripting tabs. I developed this for our specific environment. We name all objects based on a three-character prefix. That’s the prefix the script takes as an argument.
    validate_obj_script.sh-p prefix [-v verbosity] ae_obj_script_file
    The script is meant to read a file containing only the AE_OBJ_SCRIPT lines output by the parse_transport_case.sh script.

    Here is an example of the output.
    ./validate_obj_script.sh -p ABC -v 0 Ae_obj_script_test2.txt
    INFO : Checking objets for disallowed script lines.
    INFO : SCRI object UC0.UAT.TEST#1_1.SCRI has 29 script lines with warnings or errors.
    INFO : Problem Description    Line#     __/ Process tab \______________________________
    ERROR: Wrong prefix  (XYZ)    00004 -> :SET &RUN_ID# = ACTIVATE_UC_OBJECT(XYZ.WF1.JOBP,,,,,PASS_VALUES,"ABC",)
    ERROR: Wrong prefix  (XYZ)    00005 -> :SET &RUN_ID# = ACTIVATE_UC_OBJECT('XYZ.WF1.JOBP',,,,,PASS_VALUES,"ABC",)
    ERROR: Wrong prefix  (GHI)    00006 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT(GHI.WF1.JOBP",,,,,PASS_VALUES,"ABC",)
    ERROR: Variable in obj. name  00007 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT(&VARIABLE#,,,,,PASS_VALUES,"ABC",)
    ERROR: Variable in obj. name  00008 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT('&VARIABLE#',,,,,PASS_VALUES,"ABC",)
    ERROR: Variable in obj. name  00009 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT("&VARIABLE#,,,,,PASS_VALUES,ABC,)
    ERROR: Prefix missing (ABC)   00010 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT(LONGNAME,,,,,PASS_VALUES,ABC,)
    ERROR: Prefix missing (ABC)   00011 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT('LONGNAME',,,,,PASS_VALUES,ABC,)
    ERROR: Prefix missing (ABC)   00012 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT("LONGNAME",,,,,PASS_VALUES,ABC,)
    WARN : Use only w/ own tasks  00013 -> :SET &RC# = MODIFY_TASK(123456789,STOP_MODIFY)
    WARN : Use only w/ own tasks  00014 -> :SET &RC# = MODIFY_TASK(&RUNID#,STOP_MODIFY)
    WARN : Use only w/ own tasks  00015 -> :SET &RC# = MODIFY_TASK('&RUNID#',STOP_MODIFY)
    WARN : Use only w/ own tasks  00016 -> :SET &RC# = MODIFY_TASK("&RUNID#",STOP_MODIFY)
    WARN : Use only w/ own tasks  00017 -> :SET &STATUS# = CANCEL_UC_OBJECT(&JOBNR#)
    ERROR: Prohibited element     00018 -> :SET &MODOBJ# = MODIFY_UC_OBJECT(&RUNNR#, EARLIEST_STARTTIME, "MM.CLOSING","150000")
    ERROR: Prefix missing (ABC)   00019 -> :   REPLACE_STRUCTURE OBJECT="MM.DAY.MOD.051231"
    ERROR: Prohibited element     00020 -> :SET &RB# = RERUN_UC_OBJECT(&PARENT_RUNID#)
    ERROR: Prefix missing (ABC)   00021 -> :SET &RET# = RESTART_UC_OBJECT ("JOBS.SYSTEM.CHECK",LAST,,"GEN_JCL,VAR_MOD,MAN_RELEASE")
    ERROR: Prohibited element     00022 -> : SET &RB# = ROLLBACK_UC_OBJECT(&ACT#)
    ERROR: Prohibited element     00023 -> :SET &RET# = TOGGLE_OBJECT_STATUS(&RUNNR#, "STOP")
    ERROR: Prefix missing (ABC)   00024 -> :SET &RET# = SET_SYNC("DB.STATUS","EXCLUSIVE")
    ERROR: Prohibited element     00025 -> :SET &RET# = CREATE_OBJECT("CALE","FIRM.CALENDAR2003",,"Firm calendar for 2003")
    ERROR: Prohibited element     00026 -> :SET &RET# = MODIFY_OBJECT("FIRM.CALENDAR",,"WORKDAYS","DD.MM.YYYY:01.05.2011","DD.MM.YYYY:01.05.2012")
    ERROR: Prohibited element     00027 -> :SET &RET# = MOVE_OBJECT ("VIENNA","TIME_ZONES/OBJECTS")
    ERROR: Prohibited element     00028 -> :SET &RET# = REMOVE_OBJECT("FIRM.CALENDAR2003")
    ERROR: Prohibited element     00030 -> :TERMINATE SERVER,"AE#WP001"
    ERROR: Prohibited element     00031 -> :   SHUTDOWN AE1
    ERROR: Prohibited element     00032 -> :SET_UC_SETTING WORKLOAD_MAX, WIN01, 1000
    ERROR: Prohibited element     00033 -> :SET &RET# = MODIFY_SYSTEM("MODE", "QUEUE.JOBS", "GO")
    INFO :
    INFO : ================================================================================
    INFO : AE objects processed           : 1
    INFO :
    INFO : Total script lines processed   : 33
    INFO :   AE script lines              : 33
    INFO :   Native OS script lines       : 0
    INFO :
    INFO : Controlled function/statement  : ACTIVATE_UC_OBJECT
    INFO : Total number of lines          : 12
    INFO :   OK statements                : 3
    INFO :   NOT OK statements            : 9
    INFO :
    INFO : Controlled function/statement  : REPLACE_STRUCTURE
    INFO : Total number of lines          : 1
    INFO :   OK statements                : 0
    INFO :   NOT OK statements            : 1
    INFO :
    INFO : Controlled function/statement  : RESTART_UC_OBJECT
    INFO : Total number of lines          : 1
    INFO :   OK statements                : 0
    INFO :   NOT OK statements            : 1
    INFO :
    INFO : Controlled function/statement  : SET_SYNC
    INFO : Total number of lines          : 1
    INFO :   OK statements                : 0
    INFO :   NOT OK statements            : 1
    INFO :
    INFO : Prohibited function/statement  : MODIFY_UC_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : RERUN_UC_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : ROLLBACK_UC_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : TOGGLE_OBJECT_STATUS
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : CREATE_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : MODIFY_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : MOVE_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : REMOVE_OBJECT
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : TERMINATE
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : SHUTDOWN
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : SET_UC_SETTING
    INFO : Total number of lines          : 1
    INFO :
    INFO : Prohibited function/statement  : MODIFY_SYSTEM
    INFO : Total number of lines          : 1
    INFO :
    INFO : Summary of errors & warnings:
    INFO :
    INFO :   Objs w/ prohibited elements  : 1
    INFO :   Lines w/ prohibited elements : 12
    INFO :
    INFO :   Objs w/ disallowed lines     : 1
    INFO :   Disallowed script lines      : 12
    INFO :
    INFO :   Objs w/ warnings             : 1
    INFO :   Warning script lines         : 5
    INFO :
    INFO : ********************************************************************************
    INFO : PROHIBITED SCRIPT ELEMENTS
    INFO :
    INFO : 1 object contains a prohibited script element.
    INFO :
    INFO : The following script elements are NOT allowed:
    INFO : MODIFY_UC_OBJECT, RERUN_UC_OBJECT, ROLLBACK_UC_OBJECT,
    INFO : CREATE_OBJECT, MODIFY_OBJECT, MOVE_OBJECT, REMOVE_OBJECT,
    INFO : MODIFY_SYSTEM, :SET_UC_SETTING, :SHUTDOWN, :TERMINATE, TOGGLE_SYSTEM_STATUS
    INFO : ********************************************************************************
    INFO :
    INFO : ********************************************************************************
    INFO : SCRIPT ELEMENTS WITH DISALLOWED CONTENT
    INFO :
    INFO : 1 object contains 12 script lines with disallowed content.
    INFO :
    INFO : The script elements ACTIVATE_UC_OBJECT, RESTART_UC_OBJECT, SET_SYNC, and
    INFO : :REPLACE_STRUCTURE must refer only to objects belonging to the same prefix.
    INFO :
    INFO : The following are NOT allowed:
    INFO :     *  ACTIVATE_UC_OBJECT or RESTART_UC_OBJECT statements that execute objects
    INFO :        belonging to a different prefix.
    INFO :     *  SET_SYNC statements that refer to SYNC objects belonging to a different
    INFO :        prefix.
    INFO :     *  :REPLACE_STRUCTURE statements that load workflows belonging to a
    INFO :        different prefix.
    INFO :
    INFO : All such statements must refer to objects belonging to the same prefix.
    INFO : E.g., if your prefix is ABC and you want to execute the workflow ABC.WF1.JOBP,
    INFO : the statement should look something like one of these examples:
    INFO : :SET &RUN_ID# = ACTIVATE_UC_OBJECT("ABC.WF1.JOBP",,,,,PASS_VALUES,"ABC",)
    INFO : :SET &RUN_ID# = RESTART_UC_OBJECT("ABC.WF1.JOBP",,)
    INFO : :SET &RUN_ID# = SET_SYNC("ABC.DB2.SYNC","LOCK",)
    INFO :
    INFO : If you use a variable to set the name of the referenced object, your prefix
    INFO : must be hard-coded and must appear at the beginning of the first argument.
    INFO : Here are a few examples:
    INFO : :SET &RUN_ID# = ACTIVATE_UC_OBJECT("ABC.&WF_NAME#",,,,,PASS_VALUES,"ABC",)
    INFO : :SET &RUN_ID# = RESTART_UC_OBJECT("ABC.&WF_NAME#",,)
    INFO : :SET &RUN_ID# = SET_SYNC("ABC.&SYNC_NAME#","LOCK",)
    INFO : ********************************************************************************
    INFO :
    INFO : ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    INFO : AUDITED SCRIPT ELEMENTS
    INFO :
    INFO : 1 object contains an audited script element.
    INFO :
    INFO : The following script elements are allowed, but their use is carefully monitored:
    INFO : MODIFY_TASK, MODIFY_UC_OBJECT
    INFO :
    INFO : Please ensure that you use these functions only with tasks belonging to the same
    INFO : prefix.
    INFO : ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    Obviously, this script would have to be adapted to fit your particular needs. But I think a lot of the basic structure and logic could be reused. The regular expressions for matching AE scripting functions and statements took a while to get just right. I hope this work benefits others.

    Oh, one more thing... I have also included a small script called logging_functions.sh that is sourced by the new script.


  • 5.  Transport case parsing scripts

    Posted 08-28-2017 12:12 PM

    Version 0.8 of parse_transport_case.sh adds the ability to handle a couple of additional rather unusual record sequences:

    • TOH,F...,R,O,S— A blank object (in a folder) at the end of the transport case file.
    • TOH,F...,R,S— A blank object in <No_Folder> at the end of the transport case file.

    Changing states (switching from reading one record type to another) should now also be more reliable.



  • 6.  Re: Transport case parsing scripts

    Posted 03-07-2019 12:57 PM

    I have updated the transport case parsing script again. Version 1.2 handles unusual sequences of records a bit better and now prints a list of workflow tasks to the output.



  • 7.  Re: Transport case parsing scripts

    Posted 03-12-2019 02:49 PM

    After encountering yet another heretofore unseen sequence of records in a transport case file, I decided to sit down and do the major rewrite of the parsing script that I had been putting off. The new script parse_ae_db_file.sh, is much less complex than the parse_transport_case.sh script it replaces. It adds a couple of new output modes to extract a specific subset of object information.

     

    It also newly adds the ability to extract some basic information (a list of objects) from AE initial data files. I do not anticipate adding much more related to initial data files, but the foundation is now there in case someone else wants to build on it.

     

    As a part of this effort, I made a document describing the common high-level structure of both file formats, as well as one documenting the initial data file format specifically.



  • 8.  Re: Transport case parsing scripts

    Posted 03-15-2019 12:27 PM

    I updated the parse_ae_db_file.sh script with a fix for printing of workflow task aliases.



  • 9.  Re: Transport case parsing scripts

    Posted 03-19-2019 09:44 AM

    I’ve posted yet another small update to parse_ae_db_file.sh. Version 2.3 adds these features:

    • You can specify whether to read table descriptions using the -t parameter. The default is not to read table descriptions.
    • When reading a file of type TRANSPORT with -t y (reading of table descriptions enabled), the script now stores column names so that they can be printed in the output (verbosity 2 or 3). This makes it easier to read detailed output.

     

    Here’s an example showing the beginning of table data for a transport case file (verbosity 2).

    ...
    ================================================================================
    END table descriptions. BEGIN table data.
    ================================================================================

    ================================================================================
    BEGIN object #1
    BEGIN data for table OH
    ----------------------------------------
    BEGIN row 1 for table OH
    Field name    : OH_Client
    Field number  : 001
    Field type    : +
    Field content : 00100
    --
    Field name    : OH_OType
    Field number  : 002
    Field type    : C
    Field content : VARA
    --
    Field name    : OH_Name
    Field number  : 003
    Field type    : C
    Field content : UC0.DEV.TEST-1_1.VARA_FILELIST
    --
    Field name    : OH_Idnr
    Field number  : 004
    Field type    : +
    Field content : 0004281059
    --
    Field name    : OH_CrUserIdnr
    Field number  : 006
    Field type    : +
    Field content : 0001107131
    --
    Field name    : OH_CrDate
    Field number  : 007
    Field type    : 2
    Field content : 0170917101630
    --
    Field name    : OH_ModUserIdnr
    Field number  : 008
    Field type    : +
    Field content : 0001107131
    --
    Field name    : OH_ModDate
    Field number  : 009
    Field type    : 2
    Field content : 0170917101630
    --
    Field name    : OH_ModCnt
    Field number  : 010
    Field type    : +
    Field content : 0000000001
    ...


  • 10.  Re: Transport case parsing scripts

    Posted 04-11-2019 10:45 AM

    I posted parse_ae_db_file.sh v2.4. This version fixes a bug with the way the script handled OT rows with no OT_Content field. This could cause the array used to store OT_Content to to get out of sync with the arrays used to store the other OT fields. From now on, all default values for fields will be set using the setFieldDefaultValues() function called when an R record is encountered.