Transport case parsing scripts

[Michael A. Lowry]

Notice: I have moved this information to a separate document. All future updates will be made there:

Transport case parsing scripts 

 

Since deciphering and documenting the transport case file format, I have written a few scripts to parse such files, and extract various pieces of information.

 

Script	Description & usage	Version / Last updated
https://us.v-cdn.net/5019921/uploads/editor/gd/18bad3g2pytw.png" width="64"> parse_ae_db_file.sh (Previously known as parse_transport_case.sh)	General script for parsing Automation Engine database files. The following file types are supported: transport case unload files (type TRANSPORT) initial data files (type INITIAL) Usage: parse_ae_db_file.sh -o output_mode [-t y|n] [-v verbosity] input_file Arguments: -o output_mode: a: all object details including scripting tab content & list of workflow tasks c: only objects with matching custom attributes i: Basic object information: OH_IDnr, OH_Name, OH_OType, etc. p: scripting tab content (pre-process, process, and post-process) s: standard object details w: workflow tasks z: z/OS job details -t table_descriptions: n: do not read table descriptions (default) y: read table descriptions (much slower) -v verbosity: 0: no messages, only output (default) 1 :slightly verbose 2: more verbose 3: extremely verbose The script can also be further customized to extract other information.	v2.4 2019.04.11
parse_JBA_Rest.sh	Parses and prints details from the JBA_Rest field of z/OS jobs. parse_JBA_Rest.sh JBA_Rest_input_file The input file should contain one line per job, with tab-delimited values: JOBS_ZOS OH_Name OH_OType OH_HostAttrTypeDst JBA_Rest (These lines are included in the output of the parse_transport_case.sh script when it is run with output mode a or z).	v0.2 2015.01.27
validate_obj_script.sh	Parse & validate scripting tab content of relevant AE objects. validate_obj_script.sh -p prefix [-v verbosity] ae_obj_script_file AE_OBJ_SCRIPT OH_Name OH_OType OT_Type OT_Lnr OT_Content (This corresponds to the AE_OBJ_SCRIPT lines output by the parse_transport_case.sh script when it is run with output mode a.)	v0.4 2017.06.28
parse_zOS_job_script.sh	Parse and validate scripting tab content of z/OS jobs of type Automation Engine. parse_zOS_job_script.sh zos_job_script_file The input file should contain one line per job script line, with tab-delimited values: JOBS_ZOS_SCRIPT zOS_Job_Type OH_Name OT_Type OT_Lnr OT_Content (This corresponds to the JOBS_ZOS_SCRIPT lines output by the parse_transport_case.sh script when it is run with output mode a or z.)	v0.1 2017.06.19
logging_functions.sh	Logging functions used in other scripts. Not intended to be executed directly.	NA

[Michael A. Lowry]

I posted a new script, parse_zOS_job_script.sh. This script parses the output of parse_zOS_jobs_in_transport_case.sh, and allows validation of scripting tab content for z/OS jobs. We have adopted this approach to block transport/deployment of jobs containing specific AE scripting commands, such as :INCLUDE or :PUT_ATT.

The first field of the input file is just a label identifying lines that contain scripting tab content.

The second field of the input file is the z/OS job type:

z/OS job type	Description
1	Automation Engine (AE)
2	z/OS JCL
3	JCL incl. z/OS job card
4	AE JCL (undocumented)

The third field contains the name of the job.

The fourth field is the OT_Type:

OT_Type	Description
0	Process
1	Pre-process
2	Post-process
3	Child post-process (Not applicable to z/OS jobs)

The fifth field of the input file, OT_Lnr, is just the line number within a particular scripting tab.

The sixth field of the input file contains the content of the scripting line, OT_Content.

The type of line is identified based on the first character of the scripting tab line:

First character	Line type
!	AE comment
:	AE scripting
Anything else	z/OS JCL

It would be relatively straightforward to adapt this script to handle job types other than z/OS, or to do more detailed examination of scripting lines containing other specific content.

[Michael A. Lowry]

Soon after I posted the script for parsing the script content of z/OS jobs, a new related requirement became apparent:

• Validate that allACTIVATE_UC_OBJECTscripting statements point only to objects the user is authorized to execute.

This necessitated rewriting the parse_transport_case.sh script so that it is capable of printing all scripting lines of objects. The updated parse_transport_case.sh (version 0.7) is now attached to the original post. Here is a list of changes:

• Combined with parse_zOS_jobs_in_transport_case.sh script because the scripts share a lot of code. (Because its function is now included in parse_transport_case.sh, the parse_zOS_jobs_in_transport_case.sh script has now been deprecated.)
• Factored out individual portions of readOtherTableRecords & printOutput to other more table-specific functions.
• Added capability to print scripting tab lines of relevant object types (executable objects + JOBI).
• Added support for O record followed by S record.
• Factored out repeated statements.
  

The new script also adds one mandatory and one optional command line flag. The updated syntax description follows:

parse_transport_case.sh -o output_mode [-v verbosity] transport_case_file

Where output_mode is one of:

• a: all object details including scripting tab content
• c: only objects with matching custom attributes
• s: standard object details
• z: z/OS job details

and verbosity is one of:

• 0: no messages, only output
• 1: slightly verbose
• 2: more verbose
• 3: extremely verbose

I am working on a new script, validate_obj_script.sh, that can be used to validate the content of object scripting tabs. I will post it when it is ready.

[Michael A. Lowry]

I have added the new validate_obj_script.sh script I wrote to parse & validate the content of AE object scripting tabs. I developed this for our specific environment. We name all objects based on a three-character prefix. That’s the prefix the script takes as an argument.

validate_obj_script.sh-p prefix [-v verbosity] ae_obj_script_file

The script is meant to read a file containing only the AE_OBJ_SCRIPT lines output by the parse_transport_case.sh script.

Here is an example of the output.

./validate_obj_script.sh -p ABC -v 0 Ae_obj_script_test2.txt
INFO : Checking objets for disallowed script lines.
INFO : SCRI object UC0.UAT.TEST#1_1.SCRI has 29 script lines with warnings or errors.
INFO : Problem Description    Line#     __/ Process tab \______________________________
ERROR: Wrong prefix  (XYZ)    00004 -> :SET &RUN_ID# = ACTIVATE_UC_OBJECT(XYZ.WF1.JOBP,,,,,PASS_VALUES,"ABC",)
ERROR: Wrong prefix  (XYZ)    00005 -> :SET &RUN_ID# = ACTIVATE_UC_OBJECT('XYZ.WF1.JOBP',,,,,PASS_VALUES,"ABC",)
ERROR: Wrong prefix  (GHI)    00006 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT(GHI.WF1.JOBP",,,,,PASS_VALUES,"ABC",)
ERROR: Variable in obj. name  00007 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT(&VARIABLE#,,,,,PASS_VALUES,"ABC",)
ERROR: Variable in obj. name  00008 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT('&VARIABLE#',,,,,PASS_VALUES,"ABC",)
ERROR: Variable in obj. name  00009 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT("&VARIABLE#,,,,,PASS_VALUES,ABC,)
ERROR: Prefix missing (ABC)   00010 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT(LONGNAME,,,,,PASS_VALUES,ABC,)
ERROR: Prefix missing (ABC)   00011 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT('LONGNAME',,,,,PASS_VALUES,ABC,)
ERROR: Prefix missing (ABC)   00012 -> : SET &RUN_ID# = ACTIVATE_UC_OBJECT("LONGNAME",,,,,PASS_VALUES,ABC,)
WARN : Use only w/ own tasks  00013 -> :SET &RC# = MODIFY_TASK(123456789,STOP_MODIFY)
WARN : Use only w/ own tasks  00014 -> :SET &RC# = MODIFY_TASK(&RUNID#,STOP_MODIFY)
WARN : Use only w/ own tasks  00015 -> :SET &RC# = MODIFY_TASK('&RUNID#',STOP_MODIFY)
WARN : Use only w/ own tasks  00016 -> :SET &RC# = MODIFY_TASK("&RUNID#",STOP_MODIFY)
WARN : Use only w/ own tasks  00017 -> :SET &STATUS# = CANCEL_UC_OBJECT(&JOBNR#)
ERROR: Prohibited element     00018 -> :SET &MODOBJ# = MODIFY_UC_OBJECT(&RUNNR#, EARLIEST_STARTTIME, "MM.CLOSING","150000")
ERROR: Prefix missing (ABC)   00019 -> :   REPLACE_STRUCTURE OBJECT="MM.DAY.MOD.051231"
ERROR: Prohibited element     00020 -> :SET &RB# = RERUN_UC_OBJECT(&PARENT_RUNID#)
ERROR: Prefix missing (ABC)   00021 -> :SET &RET# = RESTART_UC_OBJECT ("JOBS.SYSTEM.CHECK",LAST,,"GEN_JCL,VAR_MOD,MAN_RELEASE")
ERROR: Prohibited element     00022 -> : SET &RB# = ROLLBACK_UC_OBJECT(&ACT#)
ERROR: Prohibited element     00023 -> :SET &RET# = TOGGLE_OBJECT_STATUS(&RUNNR#, "STOP")
ERROR: Prefix missing (ABC)   00024 -> :SET &RET# = SET_SYNC("DB.STATUS","EXCLUSIVE")
ERROR: Prohibited element     00025 -> :SET &RET# = CREATE_OBJECT("CALE","FIRM.CALENDAR2003",,"Firm calendar for 2003")
ERROR: Prohibited element     00026 -> :SET &RET# = MODIFY_OBJECT("FIRM.CALENDAR",,"WORKDAYS","DD.MM.YYYY:01.05.2011","DD.MM.YYYY:01.05.2012")
ERROR: Prohibited element     00027 -> :SET &RET# = MOVE_OBJECT ("VIENNA","TIME_ZONES/OBJECTS")
ERROR: Prohibited element     00028 -> :SET &RET# = REMOVE_OBJECT("FIRM.CALENDAR2003")
ERROR: Prohibited element     00030 -> :TERMINATE SERVER,"AE#WP001"
ERROR: Prohibited element     00031 -> :   SHUTDOWN AE1
ERROR: Prohibited element     00032 -> :SET_UC_SETTING WORKLOAD_MAX, WIN01, 1000
ERROR: Prohibited element     00033 -> :SET &RET# = MODIFY_SYSTEM("MODE", "QUEUE.JOBS", "GO")
INFO :
INFO : ================================================================================
INFO : AE objects processed           : 1
INFO :
INFO : Total script lines processed   : 33
INFO :   AE script lines              : 33
INFO :   Native OS script lines       : 0
INFO :
INFO : Controlled function/statement  : ACTIVATE_UC_OBJECT
INFO : Total number of lines          : 12
INFO :   OK statements                : 3
INFO :   NOT OK statements            : 9
INFO :
INFO : Controlled function/statement  : REPLACE_STRUCTURE
INFO : Total number of lines          : 1
INFO :   OK statements                : 0
INFO :   NOT OK statements            : 1
INFO :
INFO : Controlled function/statement  : RESTART_UC_OBJECT
INFO : Total number of lines          : 1
INFO :   OK statements                : 0
INFO :   NOT OK statements            : 1
INFO :
INFO : Controlled function/statement  : SET_SYNC
INFO : Total number of lines          : 1
INFO :   OK statements                : 0
INFO :   NOT OK statements            : 1
INFO :
INFO : Prohibited function/statement  : MODIFY_UC_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : RERUN_UC_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : ROLLBACK_UC_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : TOGGLE_OBJECT_STATUS
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : CREATE_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : MODIFY_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : MOVE_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : REMOVE_OBJECT
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : TERMINATE
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : SHUTDOWN
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : SET_UC_SETTING
INFO : Total number of lines          : 1
INFO :
INFO : Prohibited function/statement  : MODIFY_SYSTEM
INFO : Total number of lines          : 1
INFO :
INFO : Summary of errors & warnings:
INFO :
INFO :   Objs w/ prohibited elements  : 1
INFO :   Lines w/ prohibited elements : 12
INFO :
INFO :   Objs w/ disallowed lines     : 1
INFO :   Disallowed script lines      : 12
INFO :
INFO :   Objs w/ warnings             : 1
INFO :   Warning script lines         : 5
INFO :
INFO : ********************************************************************************
INFO : PROHIBITED SCRIPT ELEMENTS
INFO :
INFO : 1 object contains a prohibited script element.
INFO :
INFO : The following script elements are NOT allowed:
INFO : MODIFY_UC_OBJECT, RERUN_UC_OBJECT, ROLLBACK_UC_OBJECT,
INFO : CREATE_OBJECT, MODIFY_OBJECT, MOVE_OBJECT, REMOVE_OBJECT,
INFO : MODIFY_SYSTEM, :SET_UC_SETTING, :SHUTDOWN, :TERMINATE, TOGGLE_SYSTEM_STATUS
INFO : ********************************************************************************
INFO :
INFO : ********************************************************************************
INFO : SCRIPT ELEMENTS WITH DISALLOWED CONTENT
INFO :
INFO : 1 object contains 12 script lines with disallowed content.
INFO :
INFO : The script elements ACTIVATE_UC_OBJECT, RESTART_UC_OBJECT, SET_SYNC, and
INFO : :REPLACE_STRUCTURE must refer only to objects belonging to the same prefix.
INFO :
INFO : The following are NOT allowed:
INFO :     *  ACTIVATE_UC_OBJECT or RESTART_UC_OBJECT statements that execute objects
INFO :        belonging to a different prefix.
INFO :     *  SET_SYNC statements that refer to SYNC objects belonging to a different
INFO :        prefix.
INFO :     *  :REPLACE_STRUCTURE statements that load workflows belonging to a
INFO :        different prefix.
INFO :
INFO : All such statements must refer to objects belonging to the same prefix.
INFO : E.g., if your prefix is ABC and you want to execute the workflow ABC.WF1.JOBP,
INFO : the statement should look something like one of these examples:
INFO : :SET &RUN_ID# = ACTIVATE_UC_OBJECT("ABC.WF1.JOBP",,,,,PASS_VALUES,"ABC",)
INFO : :SET &RUN_ID# = RESTART_UC_OBJECT("ABC.WF1.JOBP",,)
INFO : :SET &RUN_ID# = SET_SYNC("ABC.DB2.SYNC","LOCK",)
INFO :
INFO : If you use a variable to set the name of the referenced object, your prefix
INFO : must be hard-coded and must appear at the beginning of the first argument.
INFO : Here are a few examples:
INFO : :SET &RUN_ID# = ACTIVATE_UC_OBJECT("ABC.&WF_NAME#",,,,,PASS_VALUES,"ABC",)
INFO : :SET &RUN_ID# = RESTART_UC_OBJECT("ABC.&WF_NAME#",,)
INFO : :SET &RUN_ID# = SET_SYNC("ABC.&SYNC_NAME#","LOCK",)
INFO : ********************************************************************************
INFO :
INFO : ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO : AUDITED SCRIPT ELEMENTS
INFO :
INFO : 1 object contains an audited script element.
INFO :
INFO : The following script elements are allowed, but their use is carefully monitored:
INFO : MODIFY_TASK, MODIFY_UC_OBJECT
INFO :
INFO : Please ensure that you use these functions only with tasks belonging to the same
INFO : prefix.
INFO : ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Obviously, this script would have to be adapted to fit your particular needs. But I think a lot of the basic structure and logic could be reused. The regular expressions for matching AE scripting functions and statements took a while to get just right. I hope this work benefits others.

Oh, one more thing... I have also included a small script called logging_functions.sh that is sourced by the new script.

[Michael A. Lowry]

Version 0.8 of parse_transport_case.sh adds the ability to handle a couple of additional rather unusual record sequences:

• TOH,F...,R,O,S— A blank object (in a folder) at the end of the transport case file.
• TOH,F...,R,S— A blank object in <No_Folder> at the end of the transport case file.

Changing states (switching from reading one record type to another) should now also be more reliable.

[Michael A. Lowry]

I have updated the transport case parsing script again. Version 1.2 handles unusual sequences of records a bit better and now prints a list of workflow tasks to the output.

[Michael A. Lowry]

After encountering yet another heretofore unseen sequence of records in a transport case file, I decided to sit down and do the major rewrite of the parsing script that I had been putting off. The new script parse_ae_db_file.sh, is much less complex than the parse_transport_case.sh script it replaces. It adds a couple of new output modes to extract a specific subset of object information.

 

It also newly adds the ability to extract some basic information (a list of objects) from AE initial data files. I do not anticipate adding much more related to initial data files, but the foundation is now there in case someone else wants to build on it.

 

As a part of this effort, I made a document describing the common high-level structure of both file formats, as well as one documenting the initial data file format specifically.

[Michael A. Lowry]

I updated the parse_ae_db_file.sh script with a fix for printing of workflow task aliases.

[Michael A. Lowry]

I’ve posted yet another small update to parse_ae_db_file.sh. Version 2.3 adds these features:

• You can specify whether to read table descriptions using the -t parameter. The default is not to read table descriptions.
• When reading a file of type TRANSPORT with -t y (reading of table descriptions enabled), the script now stores column names so that they can be printed in the output (verbosity 2 or 3). This makes it easier to read detailed output.

 

Here’s an example showing the beginning of table data for a transport case file (verbosity 2).

...
================================================================================
END table descriptions. BEGIN table data.
================================================================================

================================================================================
BEGIN object #1
BEGIN data for table OH
----------------------------------------
BEGIN row 1 for table OH
Field name    : OH_Client
Field number  : 001
Field type    : +
Field content : 00100
--
Field name    : OH_OType
Field number  : 002
Field type    : C
Field content : VARA
--
Field name    : OH_Name
Field number  : 003
Field type    : C
Field content : UC0.DEV.TEST-1_1.VARA_FILELIST
--
Field name    : OH_Idnr
Field number  : 004
Field type    : +
Field content : 0004281059
--
Field name    : OH_CrUserIdnr
Field number  : 006
Field type    : +
Field content : 0001107131
--
Field name    : OH_CrDate
Field number  : 007
Field type    : 2
Field content : 0170917101630
--
Field name    : OH_ModUserIdnr
Field number  : 008
Field type    : +
Field content : 0001107131
--
Field name    : OH_ModDate
Field number  : 009
Field type    : 2
Field content : 0170917101630
--
Field name    : OH_ModCnt
Field number  : 010
Field type    : +
Field content : 0000000001
...

[Michael A. Lowry]

I posted parse_ae_db_file.sh v2.4. This version fixes a bug with the way the script handled OT rows with no OT_Content field. This could cause the array used to store OT_Content to to get out of sync with the arrays used to store the other OT fields. From now on, all default values for fields will be set using the setFieldDefaultValues() function called when an R record is encountered.