Hello Team,
We have been facing this issue for some time now. We have setup our ticketing system with ServiceNow using sdgtw probe.
We have set delays in our alerts before assigning and creating a ticket. We see this issue mostly with CA Spectrum related tickets as Network deivces are random and might come up and down sometimes.
Issue we face (will explain with an example):
Lets consider an example where a network device in Spectrum goes down. So from Spectrum we have connected UIM using logmon probe. Logmon checks the logs in the spectrum server and grabs this device down alert. This would happen in max 3 mins from when the device went down.
Now we have a device down alert in Spectrum and the same alert in UIM as well. (UIM is the tool which is connected with ServiceNow for us. This is why we have set it up in such a way. Plus our teams could see all alerts in a single place.)
Inside the nas probe, we have setup an Auto-Operator profile to wait for 6 mins before assigning a ticket to the alert. (This will help the Network team by only notifying them of the devices which are actually down for more than a few mins. So that other devices which comes up in a bit or alerts caused by a glitch as avoided)
This is where we start facing our issue for some alerts. Lets say this alert got assigned to sdgtw and sdgtw is working with ServiceNow to get a ticket created and get the info back to UIM alert. This process usually takes about 1 - 2 mins. During this time, some devices comes up. If this device which we have taken into account comes up. The alert will get cleared from Spectrum, then logmon will clear the alert in UIM as well.
At this time, we have not received data of the ServiceNow incident back to UIM. When the alert is cleared, sdgtw cannot find an alert to assign this new ticket to.
Note: Our system has the auto-close feature setup. UIIM will close off tickets in Serivce now if the alert is cleared.
In this case, unfortunantely, as the ticket details came back to UIM only after the alert got cleared off, it wont get auto-closed and the teams would need to manually close these tickets off. This has started to be time consuing for the team and we have been trying to figure out a way to fix this.
I wanted to ask if anyone has seen such an issue. Please try to test it out and see if you are able to see such a scinario. If so, I wanted to request Broadcom to try and create a fix for this. Another question would be, what is another way to speed up ticket creation proccess. What other methods would be available.
NOTE: I beleive this same issue is not only for Spectrum alerts. It is the only one which I have noticed this happening a lot.
Thanks,
Abhimanue