i have problems to add F5 Loadbalancer devices in Spectrum via snmpv3. It only happens with certain devices, not all.
After a small tcpdump session i discovered that spectrum doesn't submit necessary snmpv3 authentication Parameter in the get-request.
I also attached a picture with a screen cap of the tcpdump. Anyone has any idea?
- add new snmp Profile in spectrum
- add new Profile on the devices i try to discover
- changed DCM retry Count and timers
- snmpwalk with snmpv3 (works!)
- add device via snmpv2 (works!)
- add similar devices with same OS Version etc. (works!)
Please follow this KB document:
Devices configure SNMPv3 SHA/AES are not able to be discovered and modeled in Spectrum
If this document does not help, follow these steps.
If you have sensitive data, please open a case at CA Support.
Enable the SNMPv3 Profile debug by the following actions:1. Open a Windows Command line and start a "bash -login" shell2. cd $SPECROOT/vnmsh3. ./connect4. ./update action=0x10329 mh=<VNM Model Handle>
The debug output is written in the $SPECROOT/SS/VNM.OUT file.Send the action again to disable the debug.
Flush the entire SNMPv3 stack of the landscape: (clears the cache)1. Open a Windows Command line and start a "bash -login" shell2. cd $SPECROOT/vnmsh3. ./connect4. ./update action=0x10330 mh=<VNM Model Handle>
This will flush the entire SNMPv3 stack for the landscape, and will recreate it on the fly at the time the devices are polled.
The debug output is written in the $SPECROOT/SS/VNM.OUT file.
Dump the SNMPv3 Profile data by sending the following update action:1. Open a Windows Command line and start a "bash -login" shell2. cd $SPECROOT/vnmsh3. ./connect4. ./update action=0x10331 mh=<VNM Model Handle>
This starts out ok. The first two packets are normal snmp engine discovery. Spectrum sends the empty get request to the device. The device is responding with usmStatsUnknownEngineIDs counter, but if you look at the message data you'll also see that it's populated the context engine ID. Spectrum is supposed to take the engine id, boots, and time and send the first OID get using that. Instead, it appears that Spectrum is looping on the engine discovery process.
See my attached screenshots for how engine discovery SHOULD happen.
I've got the same issue going on right now and I've also got a ticket open (00897842). I've got a few Linux hosts running net-snmp that respond to snmp queries, but when I try to discover them in Spectrum they just get stuck on snmp engine discovery and the packet capture looks just like yours.
We're seeing the exact same problem where spectroserver loops on the discovery process and the snoop/wireshark ouput looks the same as posted by mwegner . This has happened about 2-3 times and on all these occasions we had to bounce spectroserver on our production system to temporarily fix the problem as this was affecting our monitoring.
This problem started happening after we installed Spectrum_10.02.00.PTF_10.2.050 recently on our Solaris 11 spectrum box :
10.2.0.0.244 06/28/2017 00:56Spectrum_10.02.00.PTF_10.2.050 installed on 01/30/2018 00:25:19.
As per the release notes , this patch has overhauled the snmp subsystem , but appears to have introduced this bug.
Is a fix available for this problem ?
thank you for the quick response.
I tried your tips but it still doesn't work.
In the past everything was working fine. I added dozens of devices sometime with all kind of snmp profiles and combinations of auth and privacy protocols.
And now i can set the snmp settings to whatever i want, spectrum sends out get-request with missing parameters.
The only things that changed in the last weeks are:
1. The Loadbalancer Software is on a newer version but that should not be the problem because the initial spectrum request already looks malformed
2. i updated spectrum 4 weeks ago from version 10.1.1 to version 10.2.2 but didn't noticed any snmp problems after that.
But i also found hundrets of the following entry in the /SS/VNM.OUT and i am not sure what that means:
Nov 06 13:27:40 WARNING at SnmpItcInterface.cc(156): SNMP failsafe - Retrying request( IP=x.x.x.x, community=blablabla)
So, i hope you have some other ideas! ;D
I know you shouldn't have to...but if you stop/restart the SS can you then model them? I ask b/c it sounds like the SS is hanging onto engineids and the action code is not clearing it properly. When stopping the SS it completely wipes the v3 cache, and when you restart it builds the v3 cache from scratch.
This thread is a few months old but we do have a fix for the SNMPv3 issues that were being seen. The fixes are all included in the bi monthly rollup patch for 10.2.3 (Spectrum_10.02.03.BMP_10.2.301). There are a couple of fixes in the bi monthly rollup patch for 10.2.2 (Spectrum_10.02.02.BMP_10.2.202) as well however I would recommend installing 10.2.3 and then the Spectrum_10.02.03.BMP_10.2.301 for all v3 fixes that we have.
You'll need to open a case to request the files as those patches are not posted publicly.
Also, please take note, we found one of the issues was in using the update action code 0x10330 to clear the cache. Come to find out the action code only cleared part of the cache so it has been deprecated. If you use it, and the discovery goes into a loop, the only way to fix it is to stop/restart the SS. With the above patches, the update action code to clear the cache is now 0x10333.
I would suggest you to open a ticket with CA Support
I'm not sure that that is what is going on. If that were the case, it wouldn't keep trying the engine discovery, would it? I would expect that if there was a conflicting engine ID, we'd see the initial query from Spectrum to include an incorrect engine ID. I think that this is a slightly different variation on the problems that we've seen earlier.
I'd prefer not to restart the server until we've tried everything that we can to get debug data out of the server. If we restart it and that fixes the problem, then we're no closer to finding the root cause and we have to wait for the problem to happen again to do more troubleshooting.
First i tried to delete the snmp cache via the db command stated above silvio.
That didn't worked out very well. Yesterday i completely restarted the SS and now everything is working again!
So the reboot works good .. i hope this will last for long. ;D
Glad to hear that helped.
Unfortunately it looks like the cache clearing is not working. What version of Spectrum are you running? I thought we had a patch for this, but I believe Mathias already has it. Maybe it needs to be fixed again?
@Mathias, are you seeing this problem in dev? If so, can you try stopping/restarting the SS and see if that fixes it?
Yeah, agreed...and my apologies, I missed the statement you had about the engine id discovery that is looping. That does sound a bit different and obviously there's something wrong there (and that's why you have the case opened ).
I'm running the latest - 10.2.2.0.71 for 4 weeks now (since 06.11) on a Solaris Env. and MAYBE we had the problems since then. But i am not realy sure in that point because we didn't test the discovery process right after the system update...
But it's still running fine till now.. ;D