Automic Workload Automation

The traditional way of passing information from OS jobs back to the Automation Engine has been to make the OS job print values to the standard output, and then parse the job log using PREP_PROCESS_REPORT, either in the job post-process, or in the subsequent job.

Automation Engine version 11 introduced a useful new scripting statement, :REGISTER_VARIABLE, that provides an alternative to this approach. With :REGISTER_VARIABLE, it is no longer necessary to have the AE store and parse the whole job log, just to look for one piece of information. Instead, a :REGISTER_VARIABLE statement in the job’s Process tab takes a specific piece of information from the job (usually from script variable), and passes it back to the Automation Engine, where it can be for example published to an object variable on the parent workflow, to be used by subsequent jobs in the workflow.

It works like this: During generation of the job JCL, each :REGISTER_VARIABLE statement in the job’s Process tab is replaced by a line that runs the Agent’s Job Messenger Daemon with a specific set of arguments. The JMD contacts the Automation Engine directly, and sets the variable.

So for example, this line in a UNIX (Bash) job’s Process tab:

:REGISTER_VARIABLE MY_VAR1#,$my_var1

becomes this in the script that is sent to the agent:

/opt/uc4/agent/bin/ucxjlx6m JNR=0055002430 MNR=0100 PNR=22160 IPA=192.168.1.56 TYP=V VTYPE=T NAME=MY_VAR1# VALUE="$my_var1"

The Automic Documentation states that the :REGISTER_VARIABLE scripting statement can be used only inside a :BEGIN_EXT_INTERPRETER … :END_EXT_INTERPRETER block. Philipp Elmer discovered that this is not actually a hard and fast requirement, but in practice it is still a good idea. The main advantage of using :REGISTER_VARIABLE (and :REGISTER_OUTPUTFILE) only inside an external interpreter block is that because the script interpreter is explicitly specified, this information can be used to programmatically choose the correct commands to run to register a variable (or output file). This is accomplished by customizing REGISTER_VARIABLE.UNIX and REGISTER_VARIABLE.WINDOWS to run the job messenger in the right way for each interpreter, based on the value of the &$EXT_INTERPRETER#. The examples below work with an assortment of external interpreters (Perl, Python, Ruby, Bash, Powershell, & Node.js).

Setting up external interpreters

External interpreters are external programs used by an agent to interpret (and execute) scripts stored in a JOBS object. You can define an arbitrary number of external interpreters in the UC_EXT_INTERPRETERS_WINDOWS and UC_EXT_INTERPRETERS_UNIX VARA objects in client 0. Below are the ones I defined.

UC_EXT_INTERPRETERS_UNIX

UC_EXT_INTERPRETERS_WINDOWS

I then copied the REGISTER_VARIABLE.UNIX and REGISTER_VARIABLE.WINDOWS JOBI objects from client 0 to my main non-0 client, and customized them as follows:

REGISTER_VARIABLE.UNIX

:PRINT " -- BEGIN REGISTER_VARIABLE.UNIX -- "
:SET &UC_JOB_MD_TRC = &UC_JOB_MD_TRC
:IF &UC_JOB_MD_TRC = '' or '&&UC_JOB_MD_TRC'
: SET &UC_JOB_MD_TRC = '0'
:ENDIF
:set &uc_register_varname = sys_last_registered_varname()
:set &uc_register_valuename = sys_last_registered_valuename()
:PRINT "&&uc_register_varname : &uc_register_varname"
:PRINT "&&uc_register_valuename : &uc_register_valuename"
:set &uc_register_otype# = sys_act_me_type()
:if &uc_register_otype# = 'JOBS'
: SWITCH &$EXT_INTERPRETER#
: CASE 'PERL'
: INC REGISTER_VARIABLE.PERL
: CASE 'PYTHON'
: INC REGISTER_VARIABLE.PYTHON
: CASE 'RUBY'
: INC REGISTER_VARIABLE.RUBY
: CASE 'NODEJS'
: INC REGISTER_VARIABLE.NODEJS
: CASE 'BASH'
: INC REGISTER_VARIABLE.BASH
: OTHER
&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename" TRC=&UC_JOB_MD_TRC
: ENDSWITCH
:endif
:PRINT " -- END REGISTER_VARIABLE.UNIX -- "

REGISTER_VARIABLE.WINDOWS

:PRINT " -- BEGIN REGISTER_VARIABLE.WINDOWS -- "
:SET &UC_JOB_MD_TRC = &UC_JOB_MD_TRC
:IF &UC_JOB_MD_TRC = '' or '&&UC_JOB_MD_TRC'
: SET &UC_JOB_MD_TRC = '0'
:ENDIF
:set &uc_register_varname = sys_last_registered_varname()
:set &uc_register_valuename = sys_last_registered_valuename()
:PRINT "&&uc_register_varname : &uc_register_varname"
:PRINT "&&uc_register_valuename : &uc_register_valuename"
:set &uc_register_otype# = sys_act_me_type()
:if &uc_register_otype# = 'JOBS'
: set &uc_windows_typ = get_att(win_typ)
: if &UC_WINDOWS_TYP = "BAT"
&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename" TRC=&UC_JOB_MD_TRC
: ELSE
! Set variables that may not already have been set.
: SET &UC_JOBMD = GET_VAR(UC_EX_JOB_MD)
: SET &UC_IP_PORT = GET_VAR(UC_EX_IP_PORT)
: SET &UC_IP_ADR = GET_VAR(UC_EX_IP_ADDR)
! Set both versions of this variable so that we can use the same JOBIs for WINDOWS & UNIX
: SET &UC_IP_ADDR = GET_VAR(UC_EX_IP_ADDR)
: SWITCH &$EXT_INTERPRETER#
: CASE 'POWERSHELL'
: INC REGISTER_VARIABLE.POWERSHELL
: CASE 'PERL'
: INC REGISTER_VARIABLE.PERL
: CASE 'PYTHON'
: INC REGISTER_VARIABLE.PYTHON
: CASE 'RUBY'
: INC REGISTER_VARIABLE.RUBY
: CASE 'NODEJS'
! Escape backslashes in job messenger daemon path.
: SET &UC_JOBMD = STR_SUB(&UC_JOBMD,"\","\\")
: INC REGISTER_VARIABLE.NODEJS
: CASE 'BASH'
: INC REGISTER_VARIABLE.BASH
: OTHER
&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename" TRC=&UC_JOB_MD_TRC
: ENDSWITCH
: endif
:endif
:PRINT " -- END REGISTER_VARIABLE.WINDOWS -- "

Note that these two JOBIs in turn include child JOBIs — one for each interpreter. These are discussed next.

Interpreter-specific JOBIs

Remember that each :REGISTER_VARIABLE command is converted into an interpreter-specific command at script generation time. The statements entered here must therefore be able to be run inside the particular interpreter. The command that needs to be run is the Job Messenger Daemon or JMD. Depending on the interpreter type, the JMD command is run in slightly different ways: via Perl’s system function, via Python’s subprocess module, via Ruby’s system method, via Node.js’s exec method, and via PowerShell’s & operator. These JOBIs are interpreter-specific, but platform-independent. E.g., REGISTER_VARIABLE_NODEJS.JOBI is used to register a variable in Node.JS on both UNIX and Windows platforms. Below are a few examples.

REGISTER_VARIABLE_PERL.JOBI

:PRINT " -- BEGIN REGISTER_VARIABLE_PERL.JOBI -- "
# -- BEGIN REGISTER_VARIABLE_PERL.JOBI --
printf("Registering variable %s with value \"%s\".\n","&uc_register_varname", "&uc_register_valuename");
system("&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE=\"&uc_register_valuename\" TRC=&UC_JOB_MD_TRC");
# -- END REGISTER_VARIABLE_PERL.JOBI --
:PRINT " -- END REGISTER_VARIABLE_PERL.JOBI -- "

REGISTER_VARIABLE_NODEJS.JOBI

:PRINT " -- BEGIN REGISTER_VARIABLE_NODEJS.JOBI -- "
// -- BEGIN REGISTER_VARIABLE_NODEJS.JOBI --
console.log("Registering variable '&uc_register_varname' with value \"" + &uc_register_valuename + "\".");
var cmd = "&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE=\"" + &uc_register_valuename + "\" TRC=&UC_JOB_MD_TRC";
console.log("Running command: " + cmd );
const rvexec = require('child_process').exec
rvexec(cmd)
// -- END REGISTER_VARIABLE_NODEJS.JOBI --
:PRINT " -- END REGISTER_VARIABLE_NODEJS.JOBI -- "

REGISTER_VARIABLE_POWERSHELL.JOB

:PRINT " -- BEGIN REGISTER_VARIABLE_POWERSHELL.JOBI -- "
# -- BEGIN REGISTER_VARIABLE_POWERSHELL.JOBI --
Write-Host "Registering variable `"&uc_register_varname`" with value `"&uc_register_valuename`"."
& &UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename" TRC=&UC_JOB_MD_TRC
# -- END REGISTER_VARIABLE_POWERSHELL.JOBI --
:PRINT " -- END REGISTER_VARIABLE_POWERSHELL.JOBI -- "

Sample jobs

Next, it’s time to create some jobs that actually use the external interpreters. Below are a few examples.

UC0.MAL.REGISTER_VARIABLE.PERL.JOBS_UNIX

:BEGIN_EXT_INT PERL
:INCLUDE UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
my $username = $ENV{LOGNAME} || $ENV{USER} || getpwuid($<);
printf("User name : %s\n",$username);
:REGISTER_VARIABLE UNIX_PERL_USERNAME#,$username
:END_EXT_INT PERL

UC0.MAL.REGISTER_VARIABLE.PYTHON.JOBS_UNIX

:BEGIN_EXT_INT PYTHON
:INCLUDE UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
var1="Hello, world."
:REGISTER_VARIABLE VARIABLE_1#,var1
import getpass
username=getpass.getuser()
print "User name :",username
:REGISTER_VARIABLE UNIX_PYTHON_USERNAME#,username
:END_EXT_INT PERL

UC0.MAL.REGISTER_VARIABLE.RUBY.JOBS_UNIX

:BEGIN_EXT_INT RUBY
:INCLUDE UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
var1='Hello, world.'
:REGISTER_VARIABLE VARIABLE_1#,var1
require 'etc'
username=Etc.getlogin
printf("User name : %s\n",username)
:REGISTER_VARIABLE UNIX_RUBY_USERNAME#,username
:END_EXT_INT RUBY

UC0.MAL.REGISTER_VARIABLE.NODEJS.JOBS_UNIX

:BEGIN_EXT_INT NODEJS
:INCLUDE UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
var var1 = "Hello, world.";
console.log(var1);
:REGISTER_VARIABLE VARIABLE_1#,var1
var username = require("os").userInfo().username;
console.log(username);
:REGISTER_VARIABLE UNIX_NODEJS_USERNAME#,username
:END_EXT_INT NODEJS

UC0.MAL.REGISTER_VARIABLE.NODEJS.JOBS_WIN

:BEGIN_EXT_INT POWERSHELL
:INCLUDE UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
$var1="Hello, world."
Write-Host "Variable 1 : $var1"
:REGISTER_VARIABLE VARIABLE_1#,$var1
$username=[Environment]::UserName
Write-Host "User name : $username"
:REGISTER_VARIABLE WIN_POWERSHELL_USERNAME#,$username
:END_EXT_INT POWERSHELL

Note the :INCLUDE statement in these jobs. The object UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI contains interpreter-specific steps to optionally save a copy of the temporary interpreter script. This can be useful for troubleshooting. This JOBI also optionally registers the copy of the external interpreter script, making it easy to view in the AWI or JUI. This JOBI is discussed in detail below.

In the job’s post-process tab, I added a single :PUBLISH statement. This statement takes the value set on the job with :REGISTER_VARIABLE, and publishes it to the parent workflow.

:PUBLISH &UNIX_PERL_USERNAME#,,WORKFLOW

This variable is also defined (with no value) in the workflow’s Variables & Prompts tab.

With one job for each interpreter (and in a couple of instances, for the same interpreter on both platforms), we collect all of the jobs in a workflow, along with a script to collect and print the values of all of the registered variables.

Each of the interpreter-specific jobs runs a command (in this case, a command to determine the name of the executing user), and assigns the result to a variable. It then registers the value of this variable in an AE variable, using the :REGISTER_VARIABLE statement. After all jobs have completed, a successor AE script runs and prints the values of the five variables. I have attached an XML export of the workflow and its constituent jobs so you can experiment with them.

Advantages of using :REGISTER_VARIABLE instead of PREP_PROCESS_REPORT

• Once you have set it up, you can pass a variable from the OS job to the parent AE workflow with the addition of just a couple of statements:
  • a :REGISTER_VARIABLE statement in the process tab, and
  • a :PUBLISH statement in the post-process.
• It is not necessary to store the job log in the AE DB.
• It’s probably somewhat faster, because the AE does not need to parse the whole job log.
• Setting the variable in the AE is more reliable (e.g., if the log is too large to be stored in the AE DB)

Saving a copy of the temporary interpreter script

When an external interpreter block is used in a job, the AE generates a temporary script file from the commands inside the block, and sends this file to the agent. The agents writes the file to its resources directory, runs the script, and then removes the file. If there was any problem running the job, the fact that the file is no longer present can make troubleshooting difficult. To work around this obstacle, I created UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI. This include object contains interpreter-specific steps to save a copy of the temporary interpreter script to a temporary directory, whence it can be retrieved to aid troubleshooting.

To use this JOBI, simply include it in an external interpreter block, and set the variable &SAVE_INTERPRETER_SCRIPT# to YES.

To register the copied interpreter script with the AE server, set &REGISTER_INTERPRETER_SCRIPT# to YES.

UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI

:PRINT " -- BEGIN UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI -- "
:SET &SAVE_INTERPRETER_SCRIPT# = &SAVE_INTERPRETER_SCRIPT#
:SET &REGISTER_INTERPRETER_SCRIPT# = &REGISTER_INTERPRETER_SCRIPT#
! Set location for writing copy of script based on platform.
:PRINT "Agent : &$Agent#"
:PRINT "Platform : &$Platform#"
:IF &$Platform# = "WINDOWS"
: SET &Win_Type# = get_att(WIN_TYP)
:PRINT "Windows job type : &Win_Type#"
: IF &Win_Type# = "EXTCOMPROC"
: PRINT "Interpreter : &$EXT_INTERPRETER#"
: ENDIF
:ENDIF
:SWITCH &$PLATFORM#
:CASE "UNIX"
: SET &TEMP_DIR# = "/tmp"
:CASE "WINDOWS"
: SET &TEMP_DIR# = 'C:\temp'
:ENDSWITCH
:PRINT "Temporary directory : &TEMP_DIR#"
:IF &SAVE_INTERPRETER_SCRIPT# = "YES"
: PRINT "Saving a copy of the temporary external interpreter job script."
: SWITCH &$EXT_INTERPRETER#
: CASE 'BASH'
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_BASH.JOBI
: CASE 'PERL'
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_PERL.JOBI
: CASE 'PYTHON'
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_PYTHON.JOBI
: CASE 'RUBY'
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_RUBY.JOBI
: CASE 'POWERSHELL'
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_POWERSHELL.JOBI
: CASE 'NODEJS'
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_NODEJS.JOBI
: CASE ''
: PRINT "ERROR: &&$EXT_INTERPRETER# variable is null"
: PRINT "Cannot copy temporary intepreter script."
: OTHER
: INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_OTHER.JOBI
: ENDSWITCH
: IF &REGISTER_INTERPRETER_SCRIPT# = "YES"
: INC UC0.MAL.REGISTER_INTERPRETER_SCRIPT.JOBI
: ENDIF
:ENDIF
:PRINT " -- END UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI -- "

Registering output files

As noted above, in addition to saving a copy of the temporary interpreter script, the UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI JOBI object can also register the saved script. It does this if &REGISTER_INTERPRETER_SCRIPT# is set to YES.

You can also include your own :REGISTER_OUTPUTFILE statements in your interpreter scripts, to register arbitrary output files with the AE server. 

Behind the scenes, this is accomplished using a similar approach to the one employed for registering variables: each :REGISTER_OUTPUTFILE statement is replaced at generation time by a set of interpreter-specific commands that run the Job Messenger Daemon in an interpreter-specific way. See the interpreter-specific REGISTER_OUTPUTFILE VARAs to see how this works.

Attachments

All objects described above are attached as XML.

• REGISTER_VARIABLE & REGISTER_OUTPUTFILE VARAs uc4_export.xml
  My customized versions of four system VARAs:
  • REGISTER_VARIABLE.UNIX
  • REGISTER_VARIABLE.WINDOWS
  • REGISTER_OUTPUTFILE.UNIX
  • REGISTER_OUTPUTFILE.WINDOWS
  • Plus, the interpreter-specific child JOBIs required by these four JOBIs
• REGISTER_VARIABLE examples uc4_export.xml
  A workflow and eight example jobs, plus a bunch of JOBIs useful for troubleshooting:
  • UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
  • UC0.MAL.REGISTER_INTERPRETER_SCRIPT.JOBI
  • Plus, the interpreter-specific child JOBIs required by these two JOBIs

This document was generated from the following discussion: :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell