Michael Schafer: During my summer with Broadcom, I joined the NetMaster team. Like the name suggests, this team develops NetMaster, a powerful network monitoring tool for mainframe software. This tool allows mainframe users to monitor mainframe network connections and quickly address any networking issues, thereby minimizing the downtime of vital services.
As I joined, the team was in the midst of developing the new NetMaster 13 release. To ensure this release is error free, timely, and aligned with customer expectations, the team is performing manual regression testing on the mainframe. This manual testing, however, is tedious, time consuming, and rife with opportunities for human error. These downsides of manual testing prompt an important question: is there a more efficient way to test this critical application? Therefore, to guarantee a high quality application and timely future releases, I created an Automated Regression Test Suite for NetMaster.
My Journey
After settling in for the summer and completing training modules on mainframe software, I began working on automating various tests for NetMaster. These tests would ensure NetMaster functions as expected in its new releases. To create these tests, I deployed a Broadcom in-house Python module to directly interact with information on the greenscreen using high level code. This high level code would manipulate the greenscreen without having to manually traverse through NetMaster. This process is helpful in streamlining mainframe usage and testing.
As the amount of test cases increased, I needed to make my project both scalable and accessible to other developers for collaboration.
A scalable project and codebase is integral for seamlessly developing new tests and functionality for the application. I made my code more scalable by:
-
creating generalized functions so that future tests with a similar structure can re-use my code
-
organizing my functions into several files within a ‘utils’ folder, categorized according to the function’s test type and purpose
-
leveraging the “invoke” module to create custom commands in a tasks.py file that run the tests
-
selecting the “ruff” linter and “mypy” type checker to enforce a standardized code format across my project and catch issues before runtime
Just as scalability is important for software development, so too is collaborating with peers on this development. Collaboration allows for developers of differing experience levels and backgrounds to come together and create a quality product. An attentiveness to this value is critical for success in an organization and is especially necessary for my project, as I will be providing it to my team at the end of my experience for its future development. Collaboration is also a core principle of Agile/Scrum, the software development methodology used to develop NetMaster. It is Agile/Scrum’s emphasis on “individuals and interactions” that contributes to current successes on the NetMaster team and the methodology’s widespread adoption in software development more generally.
Because of these many benefits of collaboration, I made my code more amenable to future development by:
-
transfering all of my tests and functions to a GitHub repository in compliance with NetMaster’s standard software development practices
-
utilizing the “pipenv” virtual environment management tool to ensure developers have easy access to the required project dependencies
-
reorganizing the directory structure so that the tests are categorized by which NetMaster feature they are testing
-
creating a README file to explain how to use and further develop the project
-
documenting all of my tests and functions to make them easily accessible to developers who have limited experience with Python and the greenscreen manipulation module
While aiming towards the values I described above, I stumbled upon one glaring performance issue. Testing all the files at once was becoming very time consuming since each test required a new emulator to start up and shut down. The impact of these two processes became even more apparent as the number of total tests increased. To solve this problem, I used Python's 'exec()’ function to execute all test files within the tasks.py file, thereby giving these files access to the same system shell and the single emulator launched from it. This solution dramatically reduced the amount of time for each test to complete and minimized the effects of adding new tests on the suite’s performance.
After addressing this issue, I integrated pytest into my testing logic, allowing me to further streamline the testing process. For this integration, I first needed to replace the exec() logic with pytest.main(). Fortunately, this replacement still allowed me to run the tests in the same system shell and squashed any potential security risks that can arise from using the “exec()” function. Then, I needed to change all the file names so they are recognizable by pytest and move all the tests inside functions. Finally, I employed pytest’s setup and teardown procedures to validate that each test starts on the NetMaster Primary Menu and to automatically return to the primary menu after a test. These procedures minimize any issues transitioning between tests if more than one runs.
To better view the results of the tests, I generated an xml report of the test results with pytest. The formatting of these results, though, was very messy. Therefore, I converted the xml file into json and neatly rendered the json in html for easy viewing in a browser. This file presents each test’s outcome in a compact and readable manner, where tests can be expanded with further information. The report is hosted locally on a simple python web server.
After creating this simple report ****, I set up a Jenkins pipeline to run my tests. This pipeline is configured to run daily to ensure changes to NetMaster and its desired functionality persist during development. It accepts parameters to specify which region and version of NetMaster it accesses as well as the maximum time a test can run. It also provides easy access to the report as an artifact of the pipeline.
To conclude my experience, I incorporated the secure dependency management product “Blackduck” and the static analysis tool ”Coverity” into a jenkins pipeline. The Coverity pipeline ensures my code is free from bugs and security vulnerabilities, while the Blackduck pipeline checks for known vulnerabilities in my project’s dependencies.
Why Is This Project Useful?
This automated regression testing is valuable because it allows for faster, reoccurring, and more reliable testing of the NetMaster application. In turn, this improved testing will contribute to a faster and more reliable delivery window for updates and a better product overall. Now, instead of spending valuable development cycles on manual quality assurance, developers can focus on implementing new features and refining the product.
Regression testing ensures that new development to NetMaster does not inadvertently affect other areas of the product. The jenkins pipeline guarantees this feature by running all tests in the suite at daily intervals, allowing developers on the team to observe any failing tests and act accordingly.
Moving Forward
For my project, I created a scalable, user friendly, and accessible Automated Regression Testing Suite. This suite resides in a GitHub repository for future development, with a tailored report **** for debugging and a jenkins pipeline to automate tests at certain events. This project will better guarantee the quality of NetMaster while also providing opportunities for faster and more frequent releases.
While completing this project, I learned much about the capabilities of mainframe systems, different programming languages, testing frameworks, and DevOps.
Mainframes were a new concept to me as I joined Broadcom, but they continue to draw my attention as I learn about their integral part in the current functioning of society. Because of the vital services that use mainframes, such as banks and the airline industry, mainframes can have essentially zero down time. This standard is daunting to uphold, but necessary for our current society to keep moving forward. One increasingly prevalent concern in the mainframe community, though, is that these systems are built on older code and skills that are less prevalent in the 21st century. Therefore, Broadcom’s mainframe modernization efforts are important for attracting new coders to maintain these systems and operators to manage and understand them.
DevOps was also foreign to me at the beginning of my project, but the benefits of and opportunities for DevOps, especially a streamlined development, are now very apparent to me. Streamlining software development with tools such as jenkins ensures a consistent quality and timely release of a product.
Outside of my understanding of mainframes, I gained new coding skills and improved existing skills. Coding in python allowed me to explore different modules that I never used previously, including invoke, pipenv, and pytest. The invoke module was particularly useful to me, as creating custom commands with python can make passing values into code very straightforward. Learning pytest, though, offered me my first experience with testing frameworks. I found its functionality to be very helpful in streamlining my testing.
Overall, I enjoyed my experience at Broadcom, not only because of how much I learned, but also because of how deeply it forced me to think about certain issues. It forced me to look at problems from radically different angles to guarantee I arrived at an “airtight,” 100% verifiable solution for testing. Specifically, it made me think of how to completely eliminate opportunities for false positive and false negative test results. Integrating this “all-angles” mentality will help me hold myself to high standards both in and outside of software development.
Finally, I greatly appreciate everyone who helped me during my project, especially my manager Michael Bauer, my project mentor Phillip Schoen, and everyone on the NetMaster team. Wherever I end up, I look forward to using the knowledge you all have given me as I go out into the world. I wish you all well, and thank you for a great summer!