Performance Center - Reachability Unknown Status for device even though it is pingable from server

[Razien Suffla]

Hi,

I am having an issue where a couple of devices are discovered in the Performance Management, where they are contactable via ICMP and SNMP from Performance Center but when i navigate to the device to pull "reachability" statistics. No data is shown. The Device "reachability" status is showing "unknown status" when on device details page. 

The work around that i found was 'deleting' the device then discovered it again in the system. This i tested with a couple of devices, however i loose all previous statistics for that device. Could anyone advise another workaround to resolving this "unknown reachability status"? 

fyi, i tried rediscovering, but that didnt work. 

Regards,
Raz

[Bruno Sousa]

Hello Razien,

Have you tried using the "Quick Device Discovery"option? (In administration->Monitored Items management->Quick Device Discovery)

BR
Bruno Sousa

[Razien Suffla]

Hi Bruno,

Yes i have tried that. It still shows  "unknown" reachability status.



any other suggestions?

Regards,
Raz

[Bruno Sousa]

Raz,

Did you also tried to do a new "discovery profile" only with that device? Making sure the in the advanced options you have the "ICMP Discovery" marked as true?

BR
Bruno Sousa

[Razien Suffla]

yes i have tried that also..sigh..

[Razien Suffla]

 Hi Bruno,

I found a way to fix it, so if you find yourself in the same boat here is the steps i did to resolve. *actually simple* xD

Go to Administration/Monitored Item Management/Monitored Devices/
       - Search and select the device affected
       - Under details tab, click on "stop polling"
       - wait a poll cycle
       - then "start polling"
       - wait another poll cycle
       - confirm in inventory list if the unknown status has changed to reachable.

It will then start collecting the reachability statistics. 

[Bruno Sousa]

Thank you for sharing!

BR
Bruno Sousa

[Jeffrey Pinard]

If any metric family appears to not be polling, you can always try these steps.

http://DA:8581/dcdebug
Enter IP and choose IP domain for device.  Select Polling Config, then View data button.
NOTE: if the view data button is greyed out, re-select the checkbox or confirm the IP and IP Domain is correct.

This will show you the polling config for the device.  This is the list of metric families and items being polled in each. 
If you don't see the metric family shown, it means DA failed to create polling config successfully, or an update to the polling config failed that resulted in it being removed and not recreated.  It could mean the device is no longer associated to a monitoring profile with that metric family.
If the poll config exists for the MF, but the poll item is not listed, it could be there was a write issue when saving the new poll config items.

If the metric family is there, then it could be when we sent the poll config updates to DC, we failed to schedule the polling.  This could be either the message didn't reach the DC due to some AMQ issue, or java GC issue.  Or some issue scheduling the poll request.
Another reason could be the time shifted on the DC and the item is failing to be polled anymore because scheduler is not requesting that poll request.

Any of these possible issues have the workaround you found of doing stop/start polling on the item.  We will delete all poll configs and recreate them, send them to the DC and cause the scheduler to reschedule them.

[Nathan Dorman]

Thanks for this page and documentation Jeff.  

I'm having a problem in that a device does not show up in the monitored devices section (it does show in the groups page and I can see it getting added to various groups) making it impossible to "reboot" the polling.

I've tried changing the device life cycle with no luck.  (when I search for the ip in the dcdebug, the buttons remain greyed out.  The ip is correct though.)

Running the various discoveries just results in device already exists.

Any other ideas on how to get this device back to a monitored state?

Thanks,
Nate

[Jeffrey Pinard]

So the device can exist in PC UI because it's coming from maybe spectrum, NFA, or another data source, but not the DA.

If you have the PC itemid for the device (DeviceID= in url if you click on device to goto device context page), you can log into mysql and run:

select sourceid,localid,itemname,v6_ntoa(address) from dst_device where itemid = <PC itemid>;

It will show the data source devices consolidated to that PC itemid.  You can see which one comes from the DA, and what it's IP Address is.
Could be a different IP than what you are searching for in Monitored Devices.  Search for that IP.

Also, when you do discovery of the IP you searched for, and it says already existing, click the Unchanged checkbox and apply.  It should show the existing device that matches the IP you tried to discover.   The IP could be in the IP address list for the existing device, so we don't discover a new device.

[Nathan Dorman]

Thanks Jeff,  the device had a different ip address than I expected.  (I'm thinking the hardware was updated recently)  Updating the ip address on the second model resolved the issue for me.


Nate

[Robert Zilliox]

Is there a fix in for this?

[Jeffrey Pinard]

Sorry Bob, fix for what?

We don't update the current polling DA IP based on if Spectrum sends us a different IP.
Our consolidation code handles looking at primary IP between data sources, and if not the same, we try and use the interface addresses synced to consolidate devices with diff IPs.  We have put fixes in to try and handle multiple devices that have the same IP on one of their interfaces.

[Robert Zilliox]

This answered my question.