Automation Engine tracing

[Michael A. Lowry]

Frustrated by the inadequate tools available for this, I created a little workflow (along with related objects) that facilitates collection of rolling traces in the Automation Engine.


The workflow has two main modes of operation:

1. Disable tracing: Run the workflow without changing any of the prompt set values (just leave them set to their defaults), and it will disable tracing completely.
2. Enable tracing: Select something other than 0=Trace disabled from the AE Processes menu, and select something other than 0=no output from one or more of the Trace options menus, and the workflow will activate tracing according to the options you selected.

The script uses :SET_UC_SETTING to enable tracing initially, but uses CHANGE_LOGGING to begin writing new files. You can specify an interval between writing subsequent files, a maximum duration, and a maximum number of times to begin writing new files. The workflow disables tracing when the maximum duration has been reached, or the maximum number of files has been written, whichever comes first.

The workflow also includes a couple of extra safety mechanisms:

• It checks whether the target file system has enough free space; if the amount of free space goes below this limit (default: 15%), tracing is disabled.
• It turns off tracing if the trace options have been changed, either manually or by another task. So if you run another instance of the workflow to disable tracing (or simply disable tracing manually), any currently running instances of the workflow will detect the change and quit (upon completion of their currently-running :WAIT interval — see below).

These checks are performed every two minutes.

To use the workflow in your environment, you will need to adjust a few things:
1. Set the queue of the workflow and script to a queue that exists in your system.
2. Adjust these variables in the process tab of the workflow:

:PSET &Trace_Filesystem# = "/var"
:PSET &AE_Agent_or_AgentGroup# = "AE_SERVERS"
:PSET &AE_Agent_Login# = "UC4.LOGIN"
:PSET &Queue# = "UC4"
:PSET &MinFilesystemFreePercent# = "15"

Update 2018.01.31 16:45 CET: I updated the workflow so that it performs the periodic checks at two minute intervals, regardless how often new log/trace files are written. To change the check interval, change this line in the script:

:SET &CheckInterval# = 120

I recommend keeping this interval between 30 seconds and 5 minutes.

This workflow builds on some things I have shared here before:

• Script to toggle AE server tracing
• JOBI to resolve an agent group
• General-purpose EXEC VARA for performing back-end tasks, used here to check the free space in the target file system.
• JOBI that runs different commands depending on context, in pre-process/process vs. post-process

Enjoy!

Attachment(s)

AE_TRACE uc4_export.xml   115 KB 1 version

[Michael A. Lowry]

Here are some of my ideas for improvement:

• Split loop into outer loop for changing logging and inner loop for other checks. This would make it possible to check more frequently whether the file system is getting full.
• Save last-used settings and load them as prompt set defaults. (This is a bit involved, but I came up with anapproachthat would probably work.)
• Create a separate, independent, workflow to periodically search archived logs and/or traces for a specified search string, and remove batches of logs & traces that do not contain the string. This would make it easier to leave tracing running for longer periods, while still preserving the specific logs & traces needed to debug a particular problem.
  

Any other suggestions?

[Anon Anon]

My suggestions:

• after trace switch a parser that seeks for some defined keywords or Run IDs
  
• after trace switch a (optional) file transfer to an UNC path (e.g. if keywords above were found)
  
• optional a Gzip or 7zip routine of the trace files
  

cheers, Wolfagng

[Michael A. Lowry]

Thanks for the ideas, FrankMuffke.

I already have a separate workflow for collection of AE logs & traces. This includes copying the file to a network share and compressing it. I doubt it makes sense to combine the workflow for enabling tracing with the one for collecting the files. (I will post the much-improved new version of this workflow soon.)

What I’d like to develop next is a tool that can correctly distinguish the files that should be preserved from the ones that should be discarded. This is not always a straightforward task, because sometimes the generation numbers of the logs and traces do not match up. As I see it, it’s going to be a multi-step process:

1. Identify files that contain a relevant search string (e.g., an error message or run ID.)
2. Identify all of the other files corresponding to the same time period(s).
3. Save these files and discard the rest.
   

[Michael A. Lowry]

I have checked the first item off of my list of improvements:

I updated the workflow so that it performs the periodic checks at two minute intervals, regardless how often new log/trace files are written.

This should reduce the likelihood that a long trace interval could result in a full file system. It also means that if you cancel the workflow/script, it will end within two minutes instead ofhanging in Generating statusuntil the end of the trace interval.

The updated workflow is attached to the original post.

[Michael A. Lowry]

I began thinking about how best to implement the second item on my to-do list: adding the option to save previously-used trace settings. It occurred to me that the last-used settings might not be the ones that one is most interested in. Rather, it would make sense to be able to save several commonly-used presets. I have an idea of how to implement this:

• A newPreset menu lists saved presets. When the user selects a preset from the list (e.g., “TCP/IP 2, DB 3, 15m, 2h, 10 files”), it changes the values of the other settings in the prompt set.
• How the presets would be stored is a good question, but I thought an ordinary static VARA would do the trick.

Dynamic elements in prompt sets can impact the data sources of other prompt set elements. They can also impact initial default values. Unfortunately though, they cannot cause the default value of other elements to be updated after the prompt set has been loaded. I have an enhancement request open about this.

I cannot think of a good way around this, so I’ll probably just give up the idea for now.

[Michael A. Lowry]

I developed an updated version of the AE tracing workflow that saves each user’s last used settings. After a user has run the workflow once, the prompt set will automatically load his or her last-used settings upon subsequent runs. I employed the approach of setting PRPT defaults using {VARA,KEY,COLUMN}-style references to EXEC VARA objects.

 

I’m providing the updated workflow as a demonstration of this approach..

Be aware that simply generating the prompt set now causes 21 tasks to be started in the background — one for each prompt set default value loaded.

 

The new version of the workflow is called UC4.AE_TRACE_2.JOBP. The old one is also included. If you want to play around with the prompt set without actually changing AE trace settings in the system, set the UC4.AE_TRACE.SCRI task in the workflow to inactive before running the workflow.

 

There are two hard-coded folder paths. You’ll probably need to create these folders or change the values to point to folders that exist in your system.

• /APPS/UC4/AE_TRACE/TEMP — in UC4.AE_TRACE.LOAD_USER_SETTINGS.SCRI
• /APPS/UC4/AE_TRACE/SAVED_SETTINGS — in UC4.AE_TRACE.SAVE_USER_SETTINGS.SCRI

 

Enjoy, and feel free to give me your feedback.

Attachment(s)

AE_TRACE 2 uc4_export.xml.zip   20 KB 1 version