My Summer Experience with Broadcom Mainframe

This summer, I had the incredible opportunity to work with Broadcom’s Mainframe Division, specifically with the Precision SDK teams, including Dashboard and Core SDK. As it was my first experience with a summer-long position, I wasn’t sure what to expect—but any uncertainty quickly disappeared once I dove into my project.

The Problem: Visual Defects Hurt User Trust

Visual bugs are more than cosmetic—they erode customer trust. In fact, 88% of users say they’re unlikely to return after a poor experience. At present, Precision’s visual checks are done manually. This process is time-consuming, costly, error-prone, and often lacks sufficient test coverage.

The earlier these bugs are found, the cheaper they are to fix. On average, fixing a bug during the design phase is 6x cheaper than during implementation and 15x cheaper than during testing. This made improving our visual testing process a clear priority.

What Is Visual Regression Testing?

Visual regression testing catches unintended changes in the UI caused by code updates—such as layout shifts, color mismatches, or font issues. These tests compare screenshots pixel-by-pixel against a known baseline to ensure UI consistency across browsers.

My goal was to integrate visual regression testing into our CI pipelines, so we could automatically catch issues early, save time, and ship with greater confidence. Due to initial budget constraints, I focused on open-source tools to minimize costs.

Example of a Visual Regression

Researching The Right Tools

After setting up local development environments for both Core and Dashboard SDK, I began exploring visual regression tools. Though our focus was open-source, I evaluated both free and commercial options.

The top priority: compatibility with Storybook, which both teams use for isolated UI development and testing. After a detailed analysis, I identified three strong open-source candidates:

• Lost Pixel: Well-documented and widely adopted, but captures full Storybook pages—leading to flaky results. It lacks a free UI for reviewing failures and suffers from slow performance.

• Loki: Easy to integrate and produces accurate component-level screenshots, but was unreliable due to frequent breakages.

• Creevey: Offers precise component-level screenshots, a useful UI for reviewing diffs, and better performance—but has a complex setup and sparse documentation.

Despite its setup complexity, Creevey stood out due to its built in UI and robust approval flow for updating baselines. I chose to move forward with it and began integrating it into the Core SDK pipeline.

Creevey detecting a visual regression

Integrating Creevey into the Jenkins Pipeline

Implementing Creevey in our Jenkins CI pipeline came with a steep learning curve and required lots of debugging. One major issue I faced was that visual tests kept failing even though the baseline images looked correct. I initially speculated that “pixeldiff,” the image comparison engine, was at fault.

Eventually, I discovered the real cause: subtle rendering differences between my local Mac environment and the Jenkins Linux environment. Once I updated the baseline images to match Jenkins’ output, the tests passed reliably.

After resolving these issues, I successfully configured the pipeline to:

• Automatically fail builds on visual test failures

• Clearly highlight which components had issues

Integration of Visual Testing with the Jenkins pipeline

Evaluating Paid Solutions: Is It Worth It?

After integrating and testing Creevey, I met with the team to evaluate its long-term viability. While Creevey worked, it had some limitations:

• Manual and clunky baseline updates

• No collaboration features

• High maintenance overhead

• No enterprise-grade features

• UI configuration added complexity and slowed PR workflows

Ultimately, despite being free, Creevey's total cost of ownership was higher than expected. That’s when we considered Chromatic, a commercial tool purpose-built for this use case.

Why Chromatic?

Chromatic, developed by the creators of Storybook, stood out for several reasons:

• Seamless Storybook integration, with automatic test generation per component

• TurboSnap, for faster test runs by only testing what changed

• Cloud-based snapshot storage, reducing the burden on GitHub

• Parallelized test execution

• Advanced diffing tools, including spotlight mode and ignored regions

• Low maintenance, freeing up developer time

Given its robust features and reduced maintenance, Chromatic delivered strong ROI and was clearly the best long-term choice for Core and Dashboard SDK visual testing.

Expanding Visual Testing to Mainframe Products

Next, I turned my focus to visual testing for products beyond SDKs. This time the goal was full page visual testing over individual components. I followed a similar strategy: evaluate open-source tools first, then compare to commercial alternatives.

I chose Playwright for its reliability, built-in test viewer, and existing use in our end-to-end (E2E) testing. I created a set of sample visual tests for Vantage, automating login and integrating the tests into our CI pipeline and I looked into the viability of containers for visual testing to ensure consistent execution.

To compare, I also evaluated Chromatic for product testing. While it’s built for Storybook, Chromatic can still be used by integrating with visual tests written for Playwright, allowing for easier full page testing while providing the same benefits mentioned above.

I’m currently working on documentation to help other teams replicate this visual testing setup for their own products.

Wrapping Up: Lessons from an Amazing Summer

This summer spent with Broadcom has been a fantastic experience. I gained hands-on exposure to real-world technologies and learned how collaborative, cross-functional teams operate in industry. Furthermore, working with developers from different teams gave me new perspectives and taught me how to create solutions that work across multiple domains.

Key areas of technical growth include:

• DevOps: Working with Jenkins for CI/CD and understanding pipeline integration

• Containers: Understanding Docker and Podman and their utility for environment consistency

• Testing: Learning the impact and value of automated visual testing in modern development

I'm incredibly grateful to everyone who supported me—especially my manager Brian Steele, my assigned buddy Chaaru Desikachari, and the entire Dashboard SDK team. Thank you all for making this such a memorable and rewarding experience.