That’s partially a function of the OS doing its job prioritizing CPU cycles for the high priority application needs. If your CPU load is excessive, then external communication protocols such as SNMP, SMTP, etc. tend to be suspended until the load subsides. Since monitoring programs are typically deemed less important than whatever critical applications the server is running, this actually helps your system try to survive during these periods of high CPU loading.
Most servers will still respond to ping in these instances, but other communications with them will be sporadic at best. The management agent lost alerts are good for identifying these circumstances as they occur.
If, however, you find that you are getting the slow responses during times when the CPU loading is normal, then that’s a where my previous suggestions may be helpful.