Automate Your Own CA DE Application Documentation.

[David Lock]

Hi Guys,

RE: Automate Your Own CA DE Application Documentation.

Firstly, I hope you guys are safe and keeping healthy during this pandemic.   I wish you and your families well at this time.

This is a very low level question.

I want to automate Documentation for my CA DE Applications.

I'd like to pull some data from the DB ESP_APPLICATION and a job definition table like ESP_GENERIC_JOB but that doesn't appear to have a record until the job has run, so not sure where the job definitions are stored on creation of APPL.

Perhaps it can be done through the Reporting Capabilities of CA Workload Automation.

e.g.

1. Application Name
2. Description as provided
3. Each Job Name (ordered by job sequence ideally)
4. Description as provided
5. Script Names 
6. Java script

Ideally the output could be in HTML or soemthing that could be linked to?

Etc.

[TRAVIS ANDERSON]

This is something that I have always wanted but from my knowledge and experience, there is no way to easily get the information. By all means someone please correct me if I am wrong here but the data is stored in table ESP_AMS_DF (select * from esp_ams_df where artifacttype = 'APPLICATION') and is stored in one column. Within that one column is all of the XML associated with the application. I've been working on trying to extract the XML with a SQL statement but have not yet figured it out.

For now what I do is export the XML definition then run a PowerShell script to extract values such as the app name and job names within the app. I also have a PowerShell script to extract schedule statements for events.

My opinion is this is one of the biggest things the product is lacking. There is no built-in easy way for all customers using the product to be able to use to get data such as what you are asking. We have had more than numerous requests to have this sort of data in any way such as a CSV format. Right now it requires an admin at our shop to provide the data but then requires someone proficient with PowerShell (which I am not there yet) and on top of that, you have to get a current backup of the XML to run your PS against. If you could get it from the database then that would be the best route.

[Don Powell]

Hi, 
This is only partially relevant. 

To get the XML for an application. 
1) Open De Desktop Client and go to define tabl. Download the application. 
2) Right click and do "Save As"  
NOTE 1: If it is opened in notepad it displays better than if it is opened in Notepad++
NOTE 2: Parsing this file can be tricky. I have had problems with carriage returns.... 

2¢
Don

[Andy Reimer]

Like Travis, we are using PowerShell to extract useful information from the XML exports.  We use imexutil.bat to export all our applications daily then run a series of subsequent PowerShell jobs to create reports.  Note that the files created by imexutil.bat are not named with a .xml suffix but they are just normal XML files.  One example report lists all the external jobs in every application.  This is useful when you are about to complete a job or application and you want to make sure this won't release some other series of jobs elsewhere.   The output is a simple text file that I can search, an example of which is below.  Here I can see that there are 4 external jobs in my application called ACT010

ACT010 name     qualifier applid scheduled
------ ----     --------- ------ ---------
       DSNPBUDE                  TODAY
       ESS9809D RUN2
       DSNPBUWA
       ACR010EN           ACR010 TODAY

------------------------------
Andy Reimer
------------------------------

[David Lock]

Andy, Travis,
Yes some great thoughts.
I could probably use the good old XSLT to transform XML to HTML and then use as a web page reference.
By doing this you can instruct XSLT to ignore certain elements etc.
https://stackoverflow.com/questions/12796433/convert-xml-to-html-using-xslt

I am on Linux.

I will need to investigate how to use the imexutil and how much space it consumes generating the report, as I will need to manage the space and delete the report after it is consumed.   I will use a mount point as a way to preserve space on the actual CA DE Server.

Great discussion guys.

I think CA could have come up with a XSLT conversion for reporting as that would have been a useful utility.


Cheers guys.

[Ravi Kiran Kunduri]

Hi,

From R12 SP1 release onwards , there is a CLI command - JOBMAP cli . Here it lists all applications and job definitions. You can filter what kind of data you want based on the parameters. The output is going to be in simple text file. Here the input will be event name .So you can provide wild card characters and get the required information.

Please look at the cli command and its sample output here - https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/intelligent-automation/ca-workload-automation-de-web-client/12-2/utilities-and-soap-web-services-functions/using-the-cli/applications-and-jobs-cli-commands/jobmap-command-create-a-job-activity-report.html

Hope it helps!
Ravi Kiran

[TRAVIS ANDERSON]

Hi Kiran, not sure how I overlooked this CLI command but it does seem to help out in our situation. A few things from my initial look at it:

1. The command shows a "Last Run:Thu Feb 27 15:21:35 EST 2020" for jobs within an app that has never ran. In my example I created an application that has never been created or never ran before then ran that CLI command against it and I got a result saying the job in that app last ran at the date and time I specified above

2. The information around Last Run does not show for some jobs in apps that I know have ran and run daily. Command I am using is jobmap events("TEST.IOPS_HOUSEKEEPING_CAWATEST") DisplayFields("SUCC,PRED"). This shows last run for some jobs in the app but then does not show last run for others in the app

Same thing goes for your Job Command field. Some jobs have the command that is being run while others do not

3. Is there a way to be able to limit who can run what types of CLI commands? I believe that was not possible at least with previous releases. So basically we are wanting others app teams (not admins or schedulers of CAWA DE) to be able to run this one CLI command to self serve themselves

4. Any concern with running this to view maybe hundreds of apps at one time? My only concern is if it could cause issues with the product while running to pull a lot of data. I was wanting to wildcard the events which would result in maybe pulling data for a hundred plus applications

5. I can't get the command to work for an application on our prod env for an app that runs every day but the command does work for another app on prod. Any suggestions around what the issue may be?

[Ravi Kiran Kunduri]

Hi,

Thank you Travis Anderson for your comments. Good to know it helped!
My comments for the queries below -

1.  If the cli command is returning last run , it could be for any previous last run. But as you mention that it never ran , then it must be a problem. I will suggest you raise a ticket for this.
2. If for few jobs it is not showing , then the last run of the job is not yet complete. May be they are in waiting, predwait , etc state.
3.  No we dont have permissions for cli command specific.
4.  You can run such command when the workload processing is low . The command is going to get a lot of data , so recommended when the processing is low on the server side.
5.  Does it throw any message for that app which it is not able to generate the report.

[TRAVIS ANDERSON]

Hi Kiran thanks for the input. My responses are below.

1. Thanks, have opened case 31835830 for this
2. These are jobs that I know have ran and run every day to completion
3. Any idea if this is on the roadmap? Would be nice for there be more flexibility around this so that access to CLI commands does not have to be an admin task
4. Appreciate that, will keep that in mind
5. The command seems to have executed but just does not display any information. I tried finding more info in some sort of log such as the CLI log but could not find anything. I have also put this into the case mentioned above

[David Lock]

Travis,

Ha!  Well I have been battling away with the XML unbeknownst to the 12.1 option.

###########################################################################
# Build the HTML for the file
#########################################################################
def createHTML():
    global applicationCount, today, htmlFile
    htmlStr = None
    list = os.listdir(source_path)
    list.sort()

    for file in list:
        htmlStr = None
        htmlStr = '' current = os.path.join(source_path, file)
        if os.path.isfile(current):
            applicationCount = applicationCount + 1
            print str(applicationCount) + ": Processing: " + file data = open(current, "rb")
            file_name = source_path + file
            full_file = os.path.abspath(os.path.join(file_path, file_name))
            tree = ET.parse(full_file)
            root = tree.getroot()
            htmlStr = htmlStr + "<TABLE align=\"center\" border=\"5\">\n"
            htmlStr = htmlStr + "<thead><tr><th colspan=\"20\">" + str(applicationCount) + ": Application:" + file + "</th></tr></thead>"
            htmlStr = htmlStr + "<TD><strong>FIELD</strong></TD><TD>VALUE</TD><TD><strong>ATTRIBUTE NAME</strong></TD><TD>ATTRIBUTE VALUE</TD></TH>"

            for elt in root.getiterator():
                elt.tag = elt.tag.rsplit('}', 1)[-1]
                htmlStr = htmlStr + "<TR>" htmlStr = htmlStr + "<TD><strong>" + str(elt.tag) + "</strong></TD>"
                #htmlStr = htmlStr + "<TD>" + str(elt.text) + "</TD>"
                if "args" in str(elt.tag) or "script_definition" in str(elt.tag):
                    htmlStr = htmlStr + "<TD>" + "SCRUBBED FOR SECURITY" + "</TD>"
                else: htmlStr = htmlStr + "<TD>" + str(elt.text) + "</TD>"
                     # Get any Attributes
                      myAttribKeys = elt.attrib
                      if myAttribKeys:
                          htmlStr = htmlStr + "<TD></TD><TD></TD>"
                          for key in myAttribKeys:
                              htmlStr = htmlStr + "<TR><TD></TD><TD></TD><TD><strong>" + key.upper() + "</strong></TD><TD>" + elt.get(key) + "</TD><TR>" else: 
                              htmlStr = htmlStr + "<TD><strong>" + "" + "</strong></TD>"
                              htmlStr = htmlStr + "<TD>" + "" + "</TD>"
                              htmlStr = htmlStr + "</TR>" + "\n"
                              htmlStr = htmlStr + "</TABLE>\n"
                              htmlStr = htmlStr + "</BR>"


                    htmlFile.write(htmlStr)
                    #print("Closing file")
                    data.close()
    print("We processed: " + str(applicationCount) + " Applications")

This seemed to work for me.

But the 12.1 CLI command looks promising.

Cheers