Every now and then we see an alarm get stuck in the SDP. For example we may have a CPU alarm for a server that is active for a few minutes then clears. From history we can see the alarm cleared and it is no longer in the alarm list in the manager. However from the SDP and the Flash based alarm console, we still see it active. If we delete the alarm from the Flash based alarm console it will clear, for a few seconds then show up again.
Has anyone else seen something like this? Also, does the SDP use a different source for it's alarms? For example does the SDP local database where alarms are stored and maybe sync'ed up with what the NAS is seeing?
I have seen this on a regular basis and the way to clean it up is a restart of Dashboard engine which may not seem to great but it only happens every once in a while and I really don't know why but I know this clears it.
That's what we are doing now, restarting the dashboard services. We also do the DAP and WASP at the same time, just for good measure. What I'd like to figure out is if there is a way to weed out that stuck alarm without having to drop the service('s). I'll keep digging as it seems like those alarms should be stored somewhere, as they are certainly not 100% in sync with what the manager sees.