ESP dSeries Workload Automation

 View Only
  • 1.  Exploring DE with Powershell Part 1

    Posted Jun 27, 2020 03:14 AM
    Edited by Andy Reimer Jun 27, 2020 03:29 PM
    First, I'm no expert with Powershell; I know enough to get by.
    Second, I worked through a bunch of this in order to solve some of my own problems. I don't know if this will solve any of yours, but hopefully it is helpful.
    Third, I use Powershell but I don't think there is anything here that couln't be accomplished in PERL or several other scripting languages.

    When I started working with DE, we were already in the practice of doing a daily export of all artifacts (Appl, Events, Javascript, etc). We have a scheduled job that calls imexutil.bat and exports everything available to us. The original intended purpose was to be a backup in case something was deleted and needed to be imported back in. I would also use this export to do things like search for an email address. I could find the jobs that send emails to specific people, but it involved a few steps. Because the exported files have no file type, I would have to tell Notepad++ that the language was XML. Then I'd have to use the XMLTools plugin to lay out the XML in a more human readable format (pretty print), then collapse all the nodes, do my search and only then could I easily see the job name.  Useful, but a bit cumbersome

    I figured there had to be an better way, and since I knew Powershell can read, parse, and modify xml, I leaned into that direction. Enough background, here is the nuts and bolts.

    You need to export your artifacts. That is covered in the documentation here.
    You probably want a good text editor with some XML specific features. I use Notepad++ with the XMLTools plugin installed.  This will allow you to familiarize yourself with the layout of the XML.

    If you open some of your exported applications in your text editor you may see a few differences. At some point (I think v12) DE changed the format of the XML, specifically around xml namespaces. XML for older applications will look like this:
    <appl name="OLDER_APP" xmlns="http://dto.wa.ca.com/application">
    <appl_runtime_name>OLDER_APP</appl_runtime_name>
    <version>12.0</version>

    XML for applications that were created 'net new' in v12 will look like this:
    <?xml version="1.0" encoding="UTF-8"?>
    <app:appl xmlns:app="http://dto.cybermation.com/application" name="NEWER_APP">
    <app:appl_runtime_name>NEWER_APP</app:appl_runtime_name>
    <app:version>12.0</app:version>

    This distinction becomes important later if you want to update the XML or use XPath to find nodes.

    Onto the PowerShell.  First your going to want to get the content of a single xml file and assign it to a variable with xml variable type.  While doing some testing for another project, I created a DE application called BORN_TO_FAIL.  It was full of jobs that failed (intentionally).  I'll use that as an example
    [xml]$xmlData = Get-Content -Path 'D:\xml\BORN_TO_FAIL'

    If you start exploring the $xmlData variable in Powershell you will see some familiar nodes like 'name', 'appl_runtime_name', 'comments' etc.


    You will also see the option to view all nodes called 'nt_job' and if you type in $xmlData.appl.nt_job you get the following array
    name                             : BORN_TO_FAIL
    qualifier                        : 1
    dependencies                     : dependencies
    notifynodefaults                 : false
    alert_notifynodefaults           : false
    snmp_notifynodefaults            : false
    hold                             : false
    request                          : false
    conditional                      : false
    criticaljob                      : false
    tag                              : Andy
    job_ancestor_wait_default_ignore : true
    retry                            : retry
    subappl_wait                     : false
    exitcodelist                     : exitcodelist
    agent                            : FAKE_AGENT01
    userid                           : FAKE_USER01
    cmdname                          : D:\Cyb_exec\andy_test\born_to_fail.bat
    is_interactive_job               : false
    
    name                             : BORN_TO_FAIL
    qualifier                        : 2
    dependencies                     : dependencies
    notifynodefaults                 : false
    alert_notifynodefaults           : false
    snmp_notifynodefaults            : false
    hold                             : false
    request                          : false
    conditional                      : false
    criticaljob                      : false
    tag                              : Andy
    job_ancestor_wait_default_ignore : true
    retry                            : retry
    subappl_wait                     : false
    exitcodelist                     : exitcodelist
    agent                            : FAKE_AGENT01
    userid                           : FAKE_USER01
    cmdname                          : D:\Cyb_exec\andy_test\DOA.bat
    is_interactive_job               : false
    
    name                             : BORN_TO_FAIL
    qualifier                        : 3
    dependencies                     : dependencies
    notifynodefaults                 : false
    alert_notifynodefaults           : false
    snmp_notifynodefaults            : false
    hold                             : false
    request                          : false
    conditional                      : false
    criticaljob                      : false
    tag                              : Andy
    job_ancestor_wait_default_ignore : true
    retry                            : retry
    subappl_wait                     : false
    exitcodelist                     : exitcodelist
    agent                            : FAKE_AGENT01
    userid                           : FAKE_USER01
    cmdname                          : D:\Cyb_exec\andy_test\NO_CHANCE.bat
    is_interactive_job               : false
    Here are my three NT jobs. Because $xmldata.appl.nt_job is just an array, if you want to look at a specific job you need to provide the array index (note that Powershell arrays start with an index number of 0). In this way $xmldata.appl.nt_job[1].cmdname will give you
    D:\Cyb_exec\andy_test\DOA.bat​

    Since $xmldata.appl.nt_job.count will tell you how many NT jobs are in the application, you can easily set up a loop to go through each nt_job and pull out the information you want.  Cool, but it gets better with XPath.  I'll talk about that in Part 2, because I really only came down to my computer to check an email, but then started writing this.  Hopefully I can find some time on the weekend to expand further.

    ------------------------------
    Andy Reimer
    ------------------------------


  • 2.  RE: Exploring DE with Powershell Part 1

    Posted Jul 08, 2020 09:20 PM
    First off thanks so much Andy for taking the time to write this up!! This has been great to read. I've wrote some basic code so this is not entirely new to me but where I struggle with is iterating through each job to extract information (such as command to run) for all jobs. For some reason my output was getting messed up but I will take another stab at it now.

    One thing I came across in the past is that I found certain fields were not displayed if there is no value for that field. Example, referring to your example where you have the appl_runtime_name node. In an app I was looking at I did not have that node because there was no value in that field. Due to my limitation of PS this was throwing me off because I was trying to write my code to say "if node xyz is not null then write abc" else write def. Because there was simply no node for some my results were not entirely accurate. I was then trying to explore how to determine if certain nodes exist. I think this was what xpath can do??

    I'm probably over complicating things here, I am definitely not someone who does this enough where I can sit down and begin writing code, I always have to refer to something I have already created through a lot of trial and error.

    Thank you so much for your time! Look forward to part 2. This has sparked a flame again for me to being working more at this. We do the same thing where we export the artifacts daily so that we can restore if needed.


  • 3.  RE: Exploring DE with Powershell Part 1

    Broadcom Employee
    Posted Jul 09, 2020 02:46 AM
    Thank you very much Andy for the detailed procedure. I am sure many users will be benefited by this.

    - Ravi Kiran