Automic Workload Automation

 View Only
  • 1.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Jun 04, 2016 10:30 AM
      |   view attached

    Notice: An updated version of this document is available here:
    :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, PowerShell, & Node.js 

     

    The traditional way of passing information from OS jobs back to the Automation Engine has been to have the OS job print the values to the standard output, and then parse the job log using PREP_PROCESS_REPORT, either in the job post-process, or in the subsequent job.

     

    Automation Engine version 11 introduces a useful new scripting statement, :REGISTER_VARIABLE, that provides an alternative to this approach. With :REGISTER_VARIABLE, it is no longer necessary to have the AE store and parse the whole job log, just to look for one piece of information. Instead, a :REGISTER_VARIABLE statement in the job’s Process tab takes a specific piece of information from the job (usually from script variable), and passes it back to the Automation Engine, where it can be for example published to an object variable on the parent workflow, to be used by subsequent jobs in the workflow.

     

    It works like this: During generation of the job JCL, each :REGISTER_VARIABLE statement in the job’s Process tab is replaced by a line that runs the Agent’s Job Messenger Daemon with a specific set of arguments. The JMD contacts the Automation Engine directly, and sets the variable.

     

    So for example, this line in a UNIX (Bash) job’s Process tab:

    :REGISTER_VARIABLE MY_VAR1#,$my_var1

    becomes this in the script that is sent to the agent:

    /opt/uc4/agent/bin/ucxjlx6m JNR=0055002430 MNR=0100 PNR=22160 IPA=192.168.1.56 TYP=V VTYPE=T NAME=MY_VAR1# VALUE="$my_var1"

    The Automic Documentation states that the :REGISTER_VARIABLE scripting statement can be used only inside a :BEGIN_EXT_INTERPRETER:END_EXT_INTERPRETER block¹, but Philipp Elmer discovered that this is not actually a hard and fast requirement. If you use :REGISTER_VARIABLE inside an ordinary UNIX shell script or Windows BAT script, it will probably work fine even if it is not inside an external interpreter block.

     

    What is clear is that if you wish to use :REGISTER_VARIABLE inside an external interpreter block, then you may have to customize the REGISTER_VARIABLE JOBIs so that they run the job messenger in the right way for each interpreter. In my examples, I set up an assortment of external interpreters (Perl, Python, Ruby, Bash, and Powershell), just to figure out how to run the job messenger correctly in each one.

     

    Setting up external interpreters

    External interpreters are external programs used by an agent to interpret (and execute) scripts stored in a JOBS object. You can define an arbitrary number of external interpreters in the UC_EXT_INTERPRETERS_WINDOWS and UC_EXT_INTERPRETERS_UNIX VARA objects in client 0. Below are the ones I defined.

    UC_EXT_INTERPRETERS_UNIX

    KeyValue 1Value 2
    PERL.pl/usr/bin/perl <FILE>
    PYTHON.py/usr/bin/python <FILE>
    RUBY.rb/usr/bin/ruby <FILE>
    BASH.sh/bin/bash <FILE>

    UC_EXT_INTERPRETERS_WINDOWS

    KeyValue 1Value 2
    POWERSHELL.ps1PowerShell -File <FILE>


    I then copied the REGISTER_VARIABLE.UNIX and REGISTER_VARIABLE.WINDOWS JOBI objects from client 0 to my main non-0 client, and customized them as follows:

    REGISTER_VARIABLE.UNIX

    :set &uc_register_varname = sys_last_registered_varname()
    :set &uc_register_valuename = sys_last_registered_valuename()
    :set &uc_register_otype# = sys_act_me_type()
    :if &uc_register_otype# = 'JOBS'
    : SWITCH &$EXT_INTERPRETER#
    : CASE 'PERL'
    system("&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE=\"&uc_register_valuename\"");
    : CASE 'PYTHON'
    from subprocess import call
    call_args=['&UC_JOBMD', 'JNR=&UC_REALNR', 'MNR=&UC_MANDANT', 'PNR=&UC_IP_PORT', 'IPA=&UC_IP_ADDR', 'TYP=V', 'VTYPE=T', 'NAME=&uc_register_varname']
    value_arg="VALUE="+&uc_register_valuename
    call_args.append(value_arg)
    call(call_args)
    : CASE 'RUBY'
    system_args="&UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="+&uc_register_valuename
    system(system_args)
    ! No separate case for BASH, because the default approach below works fine.
    : OTHER
    &UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename"
    : ENDSWITCH
    :endif

    Note that depending on the interpreter type, the JMD command is run in slightly different ways: via Perl’s system function, via Python’s subprocess module, and via Ruby’s system method.

     

    REGISTER_VARIABLE.WINDOWS

    :set &uc_register_varname = sys_last_registered_varname()
    :set &uc_register_valuename = sys_last_registered_valuename()
    :set &uc_register_otype# = sys_act_me_type()
    :if &uc_register_otype# = 'JOBS'
    : set &uc_windows_typ = get_att(win_typ)
    : if &UC_WINDOWS_TYP = "BAT"
    &UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename"
    : endif
    :endif

    (The same JMD command appears to work fine in both Windows BAT and PowerShell scripts.)

     

    Next, I created a workflow with five child jobs — one each for Bash, Perl, Python, Ruby, and PowerCenter. Here is the Perl job:

    UC0.MAL.REGISTER_VARIABLE.PERL.JOBS

    :BEGIN_EXT_INT PERL
    :INCLUDE UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI
    my $username = $ENV{LOGNAME} || $ENV{USER} || getpwuid($<);
    printf("User name : %s\n",$username);
    :REGISTER_VARIABLE UNIX_PERL_USERNAME#,$username
    :END_EXT_INT PERL

    Note the :INCLUDE statement. The object UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI contains interpreter-specific steps to optionally save a copy of the temporary interpreter script. (See the comment below more details about this JOBI.)

     

    In the job’s post-process tab, I added a single :PUBLISH statement. This statement takes the value set on the job with :REGISTER_VARIABLE, and publishes it to the parent workflow.

    :PUBLISH &UNIX_PERL_USERNAME#,,WORKFLOW

    And naturally, the variable is also defined (with no value) in the workflow’s Variables & Prompts tab.

    Each of the five jobs runs a command (in this case, a command to determine the name of the executing user), and assigns the result to a variable. It then registers the value of this variable in an AE variable, using the :REGISTER_VARIABLE statement. After all five jobs (Bash, Perl, Python, Ruby, and PowerShell) have completed, a successor AE script runs and prints the values of the five variables. I have attached an XML export of the workflow and its constituent jobs so you can experiment with them.

    Using :REGISTER_VARIABLE has a few advantages over using PREP_PROCESS_REPORT:

    • Once you have set it up, you can pass a variable from the OS job to the parent AE workflow with the addition of just a couple of statements:
      • a :REGISTER_VARIABLE statement in the process tab, and
      • a :PUBLISH statement in the post-process.
    • It is not necessary to store the job log in the AE DB.
    • It’s probably somewhat faster, because the AE does not need to parse the whole job log.
    • Setting the variable in the AE is more reliable (e.g., if the log is too large to be stored in the AE DB).

    Notes:
    1. I do not know why :REGISTER_VARIABLE can be used only inside an external interpreter script block. Perhaps someone can answer this question. (Update 2016.11.22: Philipp Elmer discovered that this is not actually a requirement.)

     

    Updates:
    2016.06.06 13:35 CEST:
    Added example for Python scripts.
    2016.06.07 8:00 CEST: Added example for Ruby scripts.
    2016.06.07 11:05 CEST: Factored-out file copy step to separate JOBI.
    2016.11.22 10:30 CET: Added note based on finding by Philipp Elmer.

    Attachment(s)



  • 2.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Jun 07, 2016 05:30 AM

    When an external interpreter block is used in a job, the AE generates a temporary script file from the commands inside the block, and sends this file to the agent. The agents writes the file to its resources directory, runs the script, and then removes the file. If there was any problem running the job, the fact that the file is no longer present can make troubleshooting difficult. To work around this obstacle, I created UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI. This include object contains interpreter-specific steps to save a copy of the temporary interpreter script to a temporary directory, whence it can be retrieved to aid troubleshooting.

     

    To use this JOBI, simply include it in an external interpreter block, and set the variable &SAVE_INTERPRETER_SCRIPT# to YES.

    UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI

    :SET &SAVE_INTERPRETER_SCRIPT# = &SAVE_INTERPRETER_SCRIPT#
    :IF &SAVE_INTERPRETER_SCRIPT# = "YES"
    : PRINT "Saving a copy of the temporary external interpreter job script."
    : SWITCH &$EXT_INTERPRETER#
    : CASE 'BASH'
    AE_EXT_INTERP_scriptname=$0
    AE_EXT_INTERP_tempdir="/tmp"
    echo "Saving a copy of ${AE_EXT_INTERP_scriptname} in ${AE_EXT_INTERP_tempdir}."
    cp $AE_EXT_INTERP_scriptname $AE_EXT_INTERP_tempdir
    : CASE 'PERL'
    my $AE_EXT_INTERP_scriptname = $0;
    my $AE_EXT_INTERP_tempdir = "/tmp";
    use File::Copy qw(copy);
    printf("Saving a copy of %s in %s.\n",$AE_EXT_INTERP_scriptname,$AE_EXT_INTERP_tempdir);
    copy $AE_EXT_INTERP_scriptname,$AE_EXT_INTERP_tempdir;
    : CASE 'PYTHON'
    from shutil import copy2
    AE_EXT_INTERP_scriptname=__file__
    AE_EXT_INTERP_tempdir="/tmp"
    print "Saving a copy of ",AE_EXT_INTERP_scriptname," in ",AE_EXT_INTERP_tempdir,"."
    copy2(AE_EXT_INTERP_scriptname, AE_EXT_INTERP_tempdir)
    : CASE 'RUBY'
    require 'fileutils'
    AE_EXT_INTERP_scriptname=__FILE__
    AE_EXT_INTERP_tempdir="/tmp"
    printf("Saving a copy of %s in %s.\n",AE_EXT_INTERP_scriptname,AE_EXT_INTERP_tempdir)
    FileUtils.cp(AE_EXT_INTERP_scriptname, AE_EXT_INTERP_tempdir)
    : CASE 'POWERSHELL'
    $AE_EXT_INTERP_scriptname = $MyInvocation.MyCommand.Definition
    $AE_EXT_INTERP_tempdir = $env:Temp
    Write-Host "Saving a copy of $AE_EXT_INTERP_scriptname in $AE_EXT_INTERP_tempdir."
    Copy-Item -Path $AE_EXT_INTERP_scriptname -Destination $AE_EXT_INTERP_tempdir
    : CASE ''
    : PRINT "Cannot copy temporary intepreter script."
    : PRINT "&&$EXT_INTERPRETER# variable is null"
    : PRINT "Was this JOBI included inside an external interpreter block?"
    : OTHER
    : PRINT "Cannot copy temporary intepreter script."
    : PRINT "Unknown interpreter type: &$EXT_INTERPRETER#"
    : ENDSWITCH
    :ENDIF

    Obviously, this will only work if the external interpreters have been configured.

     

    And for those of you who are curious, the first line sets the value of the &SAVE_INTERPRETER_SCRIPT# variable to itself to avoid a runtime error if the variable is not already set; this makes setting the variable optional.



  • 3.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Nov 22, 2016 04:30 AM
    According to Philipp Elmer, :REGISTER_VARIABLE works just fine even when it is not inside a
    :BEGIN_EXT_INTERPRETER … END_EXT_INTERPRETER block. I have updated the original post with a note to this effect.


  • 4.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Apr 04, 2017 06:46 AM

    Hi

    Michael A. Lowry

    ,

    Do you know if there is a way to make the :REGISTER_VARIABLE work using the Interpreter type instead of BAT from the Windows Tab?



  • 5.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Apr 04, 2017 07:21 AM
    Lucas_Amorim_9853: You can probably do this by modifying REGISTER_VARIABLE.WINDOWS in client 0. (It may also work to copy this object to the client where you are working and to modify that copy; I have not tested this.)


  • 6.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Apr 04, 2017 12:15 PM

    Michael A. Lowry

    , good one. I will make some tests.


    I've added below to theREGISTER_VARIABLE.WINDOWS.


    ! IF INTERPRETER
    : IF &UC_WINDOWS_TYP = "EXTCOMPROC"
           &UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADR TYP=V VTYPE=T NAME=&uc_register_varname VALUE=&uc_register_valuename
    : ENDIF

    Edit: And added below to WINDOWS.HEADER:

    :IF &UC_WIN_TYP = "EXTCOMPROC"
    :set &UC_JOBMD   = get_var(UC_EX_JOB_MD)
    :set &UC_IP_PORT = get_var(UC_EX_IP_PORT)
    :set &UC_IP_ADR  = get_var(UC_EX_IP_ADDR)
    :ENDIF


  • 7.  :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Jun 13, 2017 09:38 AM

    I updated the JOBI for saving a copy of the interpreter script. Here's a list of changeS:

    1. The platform-specific bits have been factored out to other JOBIs
    2. There are platform-specific default interpreters if the interpreter type is unknown.
    3. There's now an option to register the temporary interpreter script as an output file on the job.

    UC0.MAL.SAVE_INTERPRETER_SCRIPT.JOBI

    :SET &SAVE_INTERPRETER_SCRIPT# = &SAVE_INTERPRETER_SCRIPT#
    :SET &REGISTER_INTERPRETER_SCRIPT# = &REGISTER_INTERPRETER_SCRIPT#
    ! Set location for writing copy of script based on platform.
    : SWITCH &$PLATFORM#
    : CASE "UNIX"
    : SET &TEMP_DIR# = "/tmp"
    : CASE "Windows"
    : SET &TEMP_DIR# = "C:\temp"
    : ENDSWITCH
    :IF &SAVE_INTERPRETER_SCRIPT# = "YES"
    : PRINT "Saving a copy of the temporary external interpreter job script."
    : SWITCH &$EXT_INTERPRETER#
    : CASE 'BASH'
    : PRINT "Bash interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_BASH.JOBI
    : CASE 'PERL'
    : PRINT "Perl interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_PERL.JOBI
    : CASE 'PYTHON'
    : PRINT "Python interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_PYTHON.JOBI
    : CASE 'RUBY'
    : PRINT "Ruby interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_RUBY.JOBI
    : CASE 'POWERSHELL'
    : PRINT "PowerShell interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_POWERSHELL.JOBI
    : CASE ''
    : PRINT "ERROR: &&$EXT_INTERPRETER# variable is null"
    : PRINT "Cannot copy temporary intepreter script."
    : OTHER
    : PRINT "WARNING: Unhandled interpreter type: &$EXT_INTERPRETER#"
    : PRINT "Copying temporary intepreter script might not work."
    ! If the external interpreter is not known, try Bash on UNIX and PowerShell on Windows.
    : SWITCH &$PLATFORM#
    : CASE "UNIX"
    : PRINT "UNIX platform. Defaulting to Bash interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_BASH.JOBI
    : CASE "Windows"
    : PRINT "Windows platform. Defaulting to PowerShell interpreter."
    : INC UC0.MAL.SAVE_INTERPRETER_SCRIPT_POWERSHELL.JOBI
    : ENDSWITCH
    : ENDSWITCH
    :ENDIF

    Here is the Bash-specific JOBI:

    AE_EXT_INTERP_scriptname=$0
    AE_EXT_INTERP_tempdir="&TEMP_DIR#"
    echo "Saving a copy of ${AE_EXT_INTERP_scriptname} in ${AE_EXT_INTERP_tempdir}."
    cp $AE_EXT_INTERP_scriptname $AE_EXT_INTERP_tempdir
    :IF &REGISTER_INTERPRETER_SCRIPT# = "YES"
    AE_EXT_INTERP_regout="${AE_EXT_INTERP_tempdir}/$(basename ${AE_EXT_INTERP_scriptname})"
    chmod a+r $AE_EXT_INTERP_regout
    echo "Registering $AE_EXT_INTERP_regout with Automation Engine."
    : REGISTER_OUTPUTFILE $AE_EXT_INTERP_regout,"Y"
    :ENDIF

     

    I will update this post with the other platform-specific JOBIs when they’re ready.



  • 8.  RE: :REGISTER_VARIABLE — practical examples with Bash, Perl, Python, Ruby, and PowerShell

    Posted Jul 03, 2019 08:05 PM

    Hi, 

    this didnt work properly for UNIX/python EXT interpreter. We use different setup of REGISTER_VARIABLE.UNIX /part CASE PYTHON/

    : CASE 'PYTHON'
    :set &uc_register_varname = sys_last_registered_varname()
    :set &uc_register_valuename = sys_last_registered_valuename()
    :set &uc_register_otype# = sys_act_me_type()
    :if &uc_register_otype# = 'JOBS'
    : IF &$EXT_INTERPRETER# = 'PYTHON'
    : SET &UC_PATH_JOBMD# = STR_SUB("&UC_JOBMD", "\", "\\")
    import subprocess
    command = "&UC_PATH_JOBMD# JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADDR TYP=V VTYPE=T NAME=&uc_register_varname VALUE=\"%s\" " % (&uc_register_valuename, )
    retcode = subprocess.call(command, shell=True)
    print retcode
    print command
    if retcode < 0:
    print("Child was terminated by signal", -retcode)
    else:
    print("Child returned", retcode)
    :else
    &UC_JOBMD JNR=&UC_REALNR MNR=&UC_MANDANT PNR=&UC_IP_PORT IPA=&UC_IP_ADR TYP=V VTYPE=T NAME=&uc_register_varname VALUE="&uc_register_valuename"

    :endif
    :endif

    So if someone of you have also problem with this - try this one
    Michal



    ------------------------------
    Ing. Michal Navratil
    ------------------------------