I started working in earnest with Automation Point in 2019 I had experience with Event Monitoring software and how it benefits operations. Staring at consoles is ineffective and as we continued to grow so would the human errors. When I started this project, we had 80 LPARs in scope today we are in the range of 250-450 LPARs plus IBMi, distributed scheduling, and zVSE. We utilize console scraping, SNMP and API's.
The spark of this project was the fact we had 2 monitoring solutions. Unicenter via SNMP direct to ticketing which created tickets but required eyes on consoles, the other IO Concepts which was dated and no active development. Automation Point was being retooled and fit our needs. We needed a way to keep operations accountable and reduce human errors. We achieved this early on and our scope expanded beyond mainframes.
Today we have 95% consoles off, 99% of ticketing is automatic to the correct assignment groups with the other 1% manual tickets during "**** periods" such as IPLs or upgrades where an issue needs a ticket. We have servers across multiple networks and utilize an AP HUB and connect all the others using Monitoring of Multiple servers. This is significant due to acquisitions and the need to collapse networks this provides for a more consistent experience for operations.
We are developing more with APIs with AP and Alert Central as well as HMC monitoring. We have several deduplication techniques deployed and are improving our REST APIs between AP and our ticketing to reduce touches i.e. close a ticket assigned to Operations it closes the alert as well.
We really do not overuse rules and rely on data files in PPQs. This allows us to not have to write rules for all MSGIDs and is similar to an SQL table. We utilize a Local and global data file that reduces our need to repeat rules.
Standards are indeed key, and we continue to improve them every few months.