QA test automation can be tedious and frustrating to troubleshoot, but utilizing debugging tools can expedite the process and provide you with a meaningful starting point to solve issues with your scripts.

There are pros and cons to various methods of debugging related to QA test automation. In this scenario, I was using PTG2 (PragueTestingGears2) — a framework written in python that simplifies the process of automating mainframe testing — to write a large set of test case scripts designed to validate signon/signoff processes performed by the ACF2™ CICS interface transaction: ACFM. These cases are all online, panel processes to ensure that ACF2™ CICS handles sign-on and access attempts as designed under various combinations of ACFM option specifications. Using a blend of screen logging and a live z/OS 3270 terminal emulator, I was able to follow the status of my scripts with ease and save a lot of time searching through recorded result data. 

Screen Logging 

    A good place to start after running  a test script is with the screen logging data. Whether or not the script was successful, it is still beneficial to browse the script steps execution results to ensure that the script's actions were performed as expected; even with a passed assertion step, there is still a chance that other steps did not go as planned. 

Snippet of Logged Test Suite Results

The issue with solely using screen logs, is the sheer amount of data that has to be parsed to gain any real information. Especially, when dealing with suites containing many scripts made up of many steps, it can be very time consuming to search through the logs and pinpoint exactly where, if at all, the script experienced a block.

Snippet of Logged ACF2™ CICS ACFM Screen

For example, here we have a logged screen where we can tell that an error occurred with a simple text search. However, due to the nature of screen logging, we lose valuable information to help us troubleshoot the issue. From knowledge of the product, we know that a problem arose when the script attempted to change a field value. Unfortunately, the logged screen has taken away our capability to view the highlighted field, thus preventing us from identifying which field caused the error. With simple scripts that only edit one field, it is possible to discern where to find the issue, but with longer scripts that touch various options on various screens, tracing backwards through the processes becomes very complicated and time consuming.

Live z/OS Terminal Emulator

Here is where enabling a live z/OS 3270 terminal emulator becomes greatly valuable. With PTG2, if you set your environment variable PTG_3270_WATCH equal to 1 in your IDE, you gain the ability to watch a live terminal emulation of your script’s panel operations hosted by your internal web server. The URL to watch the live emulator is found in the logs and looks like this:

INFO:ptg2.zos.emul:Emulator connected to example.system.net. You can watch it at: http://localhost:15000/emul/0

Snippet of IDE Environment Variables

 With this capability, the moment a script starts running, you can click the link and follow your script in real-time while its steps and actions are being performed. In debug mode, you can even pause it at a specific screen in order to observe or capture error messages: thus, allowing you to instantly identify where the issue is located and make quick and efficient adjustments. 

Snippet of Locally Hosted Emulator

One downside to this setting is that, especially with longer and more complicated cases, by the time you find the URL link, click it, and get the emulator up and running, your script may have already progressed moderately far into the series of steps. By adding one more environment variable (PTG_3270_VISIBLE = 1), the live emulator pops-up as its own window the moment the script run commences, without requiring any further user action. 

Snippet of IDE Environment Variables

Here is the same example as before, but this time with the live and visible terminal emulator, instead of the logged screen: 

Snippet of Live z/OS Terminal Emulator of ACF2™ CICS ACFM Screen

Now, we can immediately, and visually, discern that the problem arose from the “Option” field, and make the necessary changes to solve the issue. When we re-run the script, we already know what to look out for and at which point in time. Just this basic color change can save time in troubleshooting a simple issue. 

Additionally, it is extremely gratifying to be able to watch your script “in action” and follow along live. While watching, it is clearly visible if your script gets terminated early, gets hung up somewhere, or runs through smoothly as planned. 

Summary

It is important to note that although this example favors the live emulator versus the logged screens, there are other cases where searching text in logs can return more relevant information. It is on a case by case basis that you need to assess which method (or a combination) would be the most beneficial to solve the issues you are experiencing. Overall, it is not just simply using the most amount of tools available to you, but instead it is about critically analyzing when to use which specific tools and leverage them to your advantage in order to minimize the time spent tackling each issue.