Script to enable/disable AE tracing for quick collection of traces

[Michael A. Lowry]

I developed a script to  quickly enable/disable tracing for the whole AE. Details follow.

 

Create an SEC_SQLI VARA object called UC4.AE_PROCCESSES_SHORT.VARA_SEC_SQLI, with the following query:

select substr(OH_Name,instr(OH_Name,'#') + 1) as Process_Name
from OH
where OH_OType = 'SERV';

Next, create a SCRI object called UC4.TRACE.SYSTEM_WIDE.SCRI with the following process content:

:SET &MyWPTraceFlags# = "0000000000000000"
:SET &MyCPTraceFlags# = "0000000000000000"
! :SET &MyWPTraceFlags# = "2400000000000000"
! :SET &MyCPTraceFlags# = "9000000000000000"
:SET &ChangeLogging# = "YES"
:PRINT "System name: &$SYSTEM#"
:P
! For work processes, trace options are set for the whole system (all WPs).
:PRINT "Setting trace options of all WPs."
:SET &MyTraceFlags# = &MyWPTraceFlags#
:SET_UC_SETTING SET_TRACE, &$SYSTEM#, &MyTraceFlags#
:P
:SET &MyVara# = "UC4.AE_PROCCESSES_SHORT.VARA_SEC_SQLI"
:SET &HND# = PREP_PROCESS_VAR(&MyVara#)
:PROCESS &HND#
: SET &ProcessName# = GET_PROCESS_LINE(&HND#,1)
: PRINT "Process name: &ProcessName#"
: SET &FullProcessName# = &$SYSTEM##&ProcessName#
: SET &ProccessStatus# = SYS_SERVER_ALIVE("&FullProcessName#")
: IF &ProccessStatus# = "Y"
: PRINT "Process &FullProcessName# is active."
! For communication processes, trace options must be set on a per-process basis
: IF STR_CUT(&ProcessName#,1,1) = "C"
: SET &MyTraceFlags# = &MyCPTraceFlags#
: PRINT "Setting trace options of &ProcessName# to &MyTraceFlags#"
: SET_UC_SETTING SET_TRACE, &FullProcessName#, &MyTraceFlags#
: ENDIF
: IF &ChangeLogging# = "YES"
! For all processes (WPs & CPs), initiate a logging change.
: PRINT "Starting new log for &ProcessName#."
: SET &RET# = CHANGE_LOGGING ("&FullProcessName#")
: IF &RET# <> 0
: PRINT "ERROR: Problem changing logging. Return code: &RET#"
: ENDIF
: ENDIF
: ELSE
: PRINT "Process &FullProcessName# is NOT active!"
: ENDIF
: P
:ENDPROCESS
:CLOSE_PROCESS &HND#

The script sets the trace flags according to the values of &MyWPTraceFlags# and &MyCPTraceFlags#, and then (optionally) initiates a logging change for each process. This script can be used when troubleshooting AE problems and when collecting logs & traces for Automic Support staff. The script is meant to be used in the following way:

1. Set the trace flags variables at the top of the script to the desired trace levels and run the script.
2. Reproduce the problem you are troubleshooting.
3. Set the trace flag variables back to 0 and run the script again.

If all goes well, the logs and traces with _01.txt at the end of their file names should now contain only the messages pertaining to the time window between subsequent runs of the script.

 

Enjoy!

[Pete Wirfs]

Very useful!

I'm testing this in my non-production environments today and it is looking good.  I added a promptset for the input variables.  If I want to have different settings "in the can", all I have to do is build workflows around it with promptset overrides.

We run our UC4 on SQLServer instead of Oracle.  So I had to change the query to the following;

select substring(OH_Name,charindex('#',OH_Name) + 1,10) as Process_Name

from OH

where OH_OType = 'SERV';

[Mitch_Denis_38]

Hi Michael, this looks good.  


This looks like it would work very well in combination with Automic Support's Trace Scheduler tool.  It allows for setting a rolling trace to start and end at a certain time.  It looks like the part it does not include that you have above is the rollover of the WP log files, which is a really cool feature.
Take a look at the embedded video on this page.  The part on the Trace Schedule starts around the minute and a half mark.  
The actual .xml files and for this can be found here along with a tutorial on how to use the Trace Scheduler.


Another really helpful tool is the Logfile Collector - an article on this is here.  The logfile collector will help to grab all necessary log files related to a specific run of an object that behaved differently than it should have.  It collects the agent log files, WP and CP log files, and job reports from the time the job ran as well as xml exports of the object that had issues and the parent object in certain situations.  This helps support tremendously in being sure that the correct files are collected for troubleshooting an issue.

[Michael A. Lowry]

As a companion to the AE trace script, I made a workflow that collects the logs and traces and puts them all in one place. The workflow performs the following steps:

1. Present the user with a prompt set, asking for details about which logs/traces to collect:
• Enter an incident number or other label for the new folder/zip file in which the files will be stored.
• Choose whether to collect only logs, or logs and traces.
  
• Select whether to collect the latest files, the second-most-recent, or those from a specific date.
2. Collect the requested logs/traces from all AE nodes.
3. Transfer files to a shared Windows DFS folder in a new sub-folder whose name is the user-supplied label.
4. Compress (zip) the folder.

This workflow makes it super easy to collect logs for Automic Support. Instead of taking 10 or 15 minutes, it now takes less than one minute. Let me know if anyone is interested, and I’ll share it here. (Note: we run our AEs on Linux servers, so the jobs that run on the AE servers to collect the pertinent files are written for UNIX servers, so they would have to be rewritten to run on Windows AE servers. And obviously, the agent names and folder locations would need to be changed before this would work in another environment.)

[laura_albrecht_automic]

I was unable to use the links that Mitch_Denis_38 provided, but I did a search manually and found the tools.  However, the one for the Logfile Collector didn't have an XML file attached anywhere to the Article 8411 anywhere that I could see.  That solution also only appeared to be provided for Windows AE's, which is fine - I'm sure I could adjust it. 

I'd be interested in seeing your workflow Michael_Lowry even though I'm on Solaris, not Linux. 

[Michael A. Lowry]

I am attaching an XML export of the folder containing the workflow and dependent objects. I call it GET_AE_LOGS_AND_TRACES, or GAELAT for short.
https://us.v-cdn.net/5019921/uploads/editor/4k/sclxnaiw7hsg.png" width="745">

Here are some basic instructions, which can also be found in the Documentation tab of the workflow:

The purpose of this workflow is to collect Automation Engine logs and traces from two servers, transfer these files to a DFS file system mounted on a Windows server, and zip them up into a single file.

This worfklow was designed to run in a company that has completely separate experimental, development, testing, and production environments -- each with its own Automation Engine system, and where each AE system runs on two UNIX (Linux) nodes. The logs/traces are collected, transferred to a Windows shared folder, and zipped.

Some things will likely need to be changed to make this work in your environment:

1. In the workflow process tab:
• &System_Name_X#,&Env_Descr_X#: the name and description of each AE system
• &Server_A_X#,&Server_B_X#: the names of the UNIX agents running on the AE server nodes
• &Login_Obj_X#: the name of the login object to use in each environment
• &Log_Filename_Pattern#,&Trace_Filename_Pattern#: the filename patterns for logs/traces
  
2. In the workflow variables & prompts tab:
   
• &COMPANY_NAME#: The name of your company. This will form part of the name of the ZIP file containing all the logs/traces.
• &TARGET_DIR#: The name of the target directory into which the logs/traces should be
    written. This could be a local directory or a shared (e.g., DFS) directory
• &ZIP_COMMAND#: The path to the WinZIP executable file, or other ZIP program.
• &ZIP_OPTIONS#: The command line options to use when creating the ZIP file
3. In the file transfer jobUC4.TRACE.GAELAT.TRANSFER_FILES.JOBF, and the Windows job
   UC4.TRACE.GAELAT.ZIP_FILES.JOBS, change the agentWINFT to the name of an agent group or agent that has access to the target directory to which you wish to write the ZIP file.

Laura Albrecht said:

I'd be interested in seeing your workflow Michael_Lowry even though I'm on Solaris, not Linux. 

You might also have to update the UNIX bits — e.g., the find command — to adapt to Solaris’s quirks.

Attachment(s)

GET_AE_LOGS_AND_TRACES uc4_export.xml   65 KB 1 version

[Michael A. Lowry]

By the way, I wanted to make it easier to specify the name of the target agent, e.g., with a workflow variable called &TARGET_AGENT#, but :PUT_ATT HOST doesn’t work with agent groups, and working around this limitation is more trouble than I’m willing to go to at the moment. :smile: 

[Eric Felker]

Michael Lowry said:

By the way, I wanted to make it easier to specify the name of the target agent, e.g., with a workflow variable called &TARGET_AGENT#, but :PUT_ATT HOST doesn’t work with agent groups, and working around this limitation is more trouble than I’m willing to go to at the moment. :smile: 

One way around this limitation is to wrap your entire workflow inside a parent workflow. The parent/top level workflow can hold your variable, and should ensure this value is set before the child workflow is activated and needs that variable to be defined. joel_wiesmann_automic helped me a lot here: https://community.automic.com/discussion/5515/dynamically-assign-a-hostg-to-a-jobs.

It is a little ugly but works!

[Michael A. Lowry]

Eric Felker said:

One way around this limitation is to wrap your entire workflow inside a parent workflow. The parent/top level workflow can hold your variable, and should ensure this value is set before the child workflow is activated and needs that variable to be defined. joel_wiesmann_automic helped me a lot here: https://community.automic.com/discussion/5515/dynamically-assign-a-hostg-to-a-jobs.

It is a little ugly but works!

Thanks. Before I saw your reply, I came up with my own solution. based on PREP_PROCESS_AGENTGROUP. It’s a bit different, and it might be better in some situations. 

[Michael A. Lowry]

I have completely rewritten the workflow for collecting logs & traces.https://us.v-cdn.net/5019921/uploads/editor/4y/lekwtichzby8.png" width="727">
The new workflow now works like this:

1. Run a test to make sure it’s possible to SSH unprompted to the remote AE server hosts.
2. On both of the remote AE server hosts:
1. List all of the files to be collected and add up their sizes.
2. Make sure there’s enough free space on the staging server.
3. Copy the files from the AE server host to the staging server.
   
3. Transfer the files to the Windows file share.
4. ZIP all the files into a single archive.

The improved workflow completely eliminates jobs on the remote AE server hosts. Everything on those hosts is now done via SSH/SCP. It also factors out previously duplicated activities to JOBIs and combines several previously separate jobs.

I have to do a few more tests and then make the workflow generic, removing any information specific to our environment. I hope to post it next week.