Joe Ackley with CA had posted a while back on how spectrum actually does polling/fault isolation:
Besides the "normal" polling that is done during the polling interval (5 minutes by default for most model), Spectrum is polling the models for other reasons as well. For example:
1. Thresholds. If you have enabled the polling for thresholds at the device level for cpu and memory, Spectrum is polling the device for that information. In addition, if you have enabled the thresholding of the associated interface models, Spectrum is polling for that information as well.
2. SpectroWatches. For out fo the box or custom SpectroWatches, Spectrum will poll for that information both at the device level and interface level.
3. On demand polls. As users are clicking on the different tabs and folders within the tabs, these are on demand polls where the data is polled at the time the tab and folder is selected.
4. Fault isolation. When Spectrum loses contact with a model, it does not automatically assert a lost contact condition. Fault isolation checks the models neighbor models to determine if it can communicate with those devices. A neighbor model is a model connected to another model. If it can't communicate with those neighbers, it checks the neighbors neighbors. If it can't communicate with the neighbors neighbors, etc, etc, etc.
Basically, how fault isolation works is as follows:
You have device A connected to device B connected to device C connected to device D modeled in the database:
A <--->B<--->C<--->D
For network traffic purposes, when Spectrum communicates with device D, the network path is A to B to C to D.
For this scenario, lets assume that device B goes down which will prevent Spectrum from communicating with device C and D.
Spectrum tries to communicate with device D. This could be for any reason as noted above. The communication is going to fail because device B is down. Spectrum does not know that yet. It only knows that device D did not respond.
Spectrum checks the neighbors for device D. According to the modeling in the database, the only neighbor for device D is device C.
Spectrum tries to communicate with device C. The communication is going to fail because device B is down. Spectrum does not know that yet. It only knows that device C did not respond.
Since Spectrum cannot communicate with all of the neighbors for device D, logically, device D could be up and running so Spectrum turns device D grey lost contact unknown.
Spectrum will check the neighbors for device C. According to the modeling in the database, device B and device D are the neighbors. Communication to both of these neighbors is going to fail because device B is down. Spectrum does not know that yet. It only knows that device B and D did not respond.
Since Spectrum could not communicate with all of the neighbors for device C, logically, device C could be up and running so Spectrum turns device C grey lost contact unknown.
Spectrum will check the neighbors for device B. The communication with C will fail but the communication with A will be successful. In this case, Spectrum will logically conclude that since it can communicate with device A but cannot communicate with device C, according to the modeling in the database, device B is the root cause for the loss of communication and turn device B to red device lost contact.
It is important to note here that for fault isolation to determine the root cause for the loss of contact, there has to be at least one contactable neighbor.
All of the communication that FI does with the neighbors is in real time. It does not wait for poll intervals.
After FI determines where the fault lies and changes the conditions of the models involved, Spectrum is still attempting communication with these models for the various reasons noted above. One thing to remember about the "normal" polling interval ( 5 minutes by default for most models), is Spectrum does not poll every model at the same 5 minute interval. For example, in the four devices used in our example, the normal 5 minute poll is not done at the same time for all four devices. Spectrum spaces the normal polling.
Using our FI scenario above, after FI determines where the fault lies and changes the condition of the models involved, let's say that 30 seconds later, device B comes back online. Spectrum does not know that yet. Spectrum polls device D 10 seconds after that. Again, the poll could be for any reason noted above. This time, the poll to device D was successful because
device B is back up. Since the poll was successful this time, it turn device D green normal. Same for device C and B.
In the above scenario, although the "normal" poll interval was 5 minutes, Spectrum only showed these deivces as unavailable for about a minute.
Chapter 7 "Fault Management" of the Spectrum 9.2.2 "Modeling and Managing Your IT Infrastructure Administrator Guide" contains more details on how Spectrum Fault Management/Isolation works.
=========
The way I think of it is like this: the poll interval is the longest period of time that Spectrum will go without otherwise talking to a device for reasons outlined above. Of far more importance is the retry and timeout values.
--Richard