SNMP problem with ESXi 5 Update 1

[MichealBlack]

We upgraded 6 of our HP BL460c blades to ESXi 5 Update 1 for Windows 8 and Server 2012 development. Everything is working OK except that I cannot SNMP monitoring to work on any of them. I have gone over the documentation and various online posts on the subject and all my settings look OK. Both port UDP161 and 162 are open in the firewalls, The /etc/snmp/xml file has the correct entries and is enabled. But I get nothing to my Cacti monitor. snmpwalk fails for all 6.However, I am seeing a few queries that seem to be returning some data.

I also deoloyed the vMA .ovf which will not start as there is no IP pool available. All the corp servers are fixed IPs and the other VMs are getting addresses from our DHCP servers. The vMA has a fixed IP in the server range. What do I need to do to get this thing working?

[Argyle]

We had some issues with SNMP and I got some tips in this thread http://communities.vmware.com/message/2103274 around snmp setting "--hwsrc sensors". We also had issues with dell and emulex vibs impacting the snmp service on some hosts.

Note that Update 1 might not be enough for full Win 8 / 2012 support. You need Patch 4 for that. We've had problems with vmotion of Win 8 / 2012 on U1.

http://kb.vmware.com/selfservice/microsites/microsite.do?cmd=displayKC&docType=kc&externalId=2006859

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014508

ESXi 5.0 U1	2012-03-15	623860
ESXi 5.0 Patch 4	2012-09-27	821926

For the vMA there is something about "vapp options" that you can disable, might be an advanced option. Then the VM will boot and you can set an IP.

[jrmunday]

Hi Argyle - in our environment the "--hwsrc sensors" was only required for the Dell hosts - HP hosts worked fine with the defaults. Will be interesting if Cacti also needs this changed?

Here is a thread for vMA IP Pool solution;

http://communities.vmware.com/message/1826215#1826215

[MichealBlack]

I resolved the IP POOL issue by disabeling the vApp option in Edit Settings and then starting the VM. After configuring the networking options at the startup of the VM I re-enabled the vApp option and it carries across reboots.

[MichealBlack]

Argyle,

We have been running the prerelease versions of Win8 and Server 2012 since July without issues, however I will be updating at the next chance I get - probably after Yule.

[Argyle]

Thats odd that its not enabled. If you can not enable it via CLI I have a vague memory that it should be possible to configure snmp via console as well but I can't find the link.

We run into Win 8 vmotion issues when started to upgrade to 5.1.

Win8 vmotion 5.0u1 -> 50u1 = ok

Win8 vmotion 5.1 -> 5.1 = ok

Win8 vmotion 5.1 -> 5.0u1 = ok

Win8 vmotion 5.0u1 -> 5.1 = Not ok. So Win 8 / Win 2012 VMs had to be turned of too be able to move them to 5.1 hosts

[MichealBlack]

Is there some secret handshake to get any output from the VMware vSphere CLI? I have the damned thing installed on my workstation, on the vSohere server and also have a vMA VM and when I try to run any command other than a "Get-VIServer" all I get is a >>. Running the command from the directory where vicfg-snmp.pl is located (C:\Program Files (x86)\VMware\VMware vSphere CLI\bin on my workstation has the same result. I have triple checked the settings in /etc/vmware/snmp.xml and SNMP is enabled and the settings are correct. when I run esxcli network ip connection list there is no output showing :161. When I run /sbin/services.sh restart there is no snmp listed and there is no snmp entry in /etc/init.d/. A service snmpd restart fails. It is like there is no snmp deamon or service installed.

What am I missing?

[Argyle]

How did you enable it in the first place if you can't get vSphere CLI to work?

According to link below:

http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.monitoring.doc_50%2FGUID-8EF36D7D-59B6-4C74-B1AA-4A9D18AB6250.html

"By default, the embedded SNMP agent is disabled. To enable it, you must configure it using the vSphere CLI command vicfg-snmp."

Get-VIServer sounds like a PowerCLI command and not a CLI command. Are you using the right CLI :smileygrin:?

[jrmunday]

If you are using PowerCLI the following scriot should configure your host;

## SNMP Configuration - using PowerCLI
## ==============================================

## ESXi host to be configured (NOT vCenter -- use a specific host!!)
$esxihost = 'myesxihost.mydomain'

## Conenct to ESXi Host (NOT vCenter -- use a specific host!!)
Connect-VIServer $esxihost -user root -password 'mypassword'

## Enable SNMP
Get-VMHostSnmp | Set-VMHostSnmp -Enabled:$true

## Add SNMP Communities
Get-VMHostSnmp | Set-VMHostSnmp -ReadOnlyCommunity public,community1,community2

## Add SNMP Targets
Get-VMHostSnmp | Set-VMHostSnmp -TargetCommunity "public" -TargetHost "mysnmptarget1.mydomain" -TargetPort 162 -AddTarget
Get-VMHostSnmp | Set-VMHostSnmp -TargetCommunity "community1" -TargetHost "mysnmptarget2.mydomain" -TargetPort 162 -AddTarget
Get-VMHostSnmp | Set-VMHostSnmp -TargetCommunity "community2" -TargetHost "mysnmptarget3.mydomain" -TargetPort 162 -AddTarget

## Send Test Traps
Test-VMHostSNMP -HostSNMP (Get-VMHostSNMP)

## Disconnect from ESXi Host
Disconnect-VIServer $esxihost -Confirm:$false

[MichealBlack]

When I run this script with the proper IPs and community strings I get the following:

>>## SNMP Configuration - using PowerCLI
>>## ==============================================

>>

>>## ESXi host to be configured (NOT vCenter -- use a specific host!!)
>>$esxihost = 'myesxihost.mydomain'

>>

>>## Conenct to ESXi Host (NOT vCenter -- use a specific host!!)
>>Connect-VIServer $esxihost -user root -password 'mypassword'

>>

>>## Enable SNMP
>>Get-VMHostSnmp | Set-VMHostSnmp -Enabled:$true

>>

>>## Add SNMP Communities
>>Get-VMHostSnmp | Set-VMHostSnmp -ReadOnlyCommunity public,community1,community2

>>

>>## Add SNMP Targets
>>Get-VMHostSnmp | Set-VMHostSnmp -TargetCommunity "public" -TargetHost "mysnmptarget1.mydomain" -TargetPort 162 -AddTarget
>>Get-VMHostSnmp | Set-VMHostSnmp -TargetCommunity "community1" -TargetHost "mysnmptarget2.mydomain" -TargetPort 162 -AddTarget
>>Get-VMHostSnmp | Set-VMHostSnmp -TargetCommunity "community2" -TargetHost "mysnmptarget3.mydomain" -TargetPort 162 -AddTarget

>>

>>## Send Test Traps
>>Test-VMHostSNMP -HostSNMP (Get-VMHostSNMP)

>>

>>## Disconnect from ESXi Host
>>Disconnect-VIServer $esxihost -Confirm:$false

>>

I do not see any evidence of a snmp service or deamon installed on any of my ESXi 5 blades. The ESX 4.1 systems are working fine.

[jrmunday]

Try close your CLI and open it again and then send a screenshot so that we ca see what it looks like. The >> chevrons look suspiciously like the last command hasn't finished properly and is waiting for something?

If this doesnt work, download and re-install CLI / Power-CLI - use the current version. Try the configuration scripts again with the fresh client install.

[MichealBlack]

I upgraded the CLI to the current version. Here ia a screen shot of how far I get before it goes south. I renamed the snmp.xml file to snmp.xml.old and created an empty new one  with 777 rights as it looked like it might be a write permissions issue. I am having trouble figuring out what it wants now.

[jrmunday]

You seem to have split this into two separate commands, rather than one singe command;
Get-VMHostSnmp | Set-VMHostSnmp -Enabled:$true

Try again copying and pasting this line only into your PowerCLI command line.

[MichealBlack]

As a whole line it fails.

I split it to test each part.

[jrmunday]

... This should enable SNMP - I can see I is currently disabled.

And then you can go thought the rest of the script one line at a time to check for any errors.

[MichealBlack]

I have also tried manually editing /etc/vmware/snmp.xml but that makes no difference.

[jrmunday]

Can you run the original test I posed using the vSphere CLI (not PowerCLI) --- what does this return? Just wondering if the PERL script will help?

[MichealBlack]

When I run: vicfg-snmp.pl --server myesxihost --username root --password mypassword --show I get:

Current SNMP agent settings:
Enabled  : 0
UDP port : 161

Communities :

Notification targets :

When I run vicfg-snmp.pl --server myesxihost --username root --password mypassword -E I get:

Enabling agent...
Failed : A general system error occurred: Load persistant store failed

I am, of course, using the proper server ID and password.

[jrmunday]

This probably means your /etc/vmware/snmp.xml is corrupt;

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1000529

Use "vi" to recreate the xml file and not a regular text editor.

[MichealBlack]

I already found that KB and renamed the snmp.xml file and created a net one per the KB. However, tha service mgmt-vmware restart command does not run on this server and an /sbin/services.sh restart restarts everything but there is still no snmp.

[jrmunday]

Has the host been rebooted since recreating the XML file?

[MichealBlack]

Yes and still no change. It is like there is no snmp code there at all.

[jrmunday]

I would either log a support call to VMware (if you have support) or take a sledgehammer approach and rebuild the host.

[MichealBlack]

I am thinking the sledgehammer approach. I have the IT 8# sledge behind my office door. Sometimes hardware only responds to threats of violence!

I found that the new snmp.xml file was corupted when I downloaded a copy via sftp. I created another copy in a programing file editor and uploaded it but again, no changr. I am going to just let it sit for the weekend and look at it again on Monday.  I may have to update to ESXi 5.1, but I want to keep all the blades in a cluster on the same version and I only get one or two chances to do upgrades in a year as these are in production 24/7. This cluster, however, is only 2 blades and currently has only 1 VM so I can do and upgrade on it next Monday.

Thanks for all the help. Hopefully 5.1 will fix things.

[jrmunday]

vSphere 5.1 introduces a new set of challenges (such as single sign on), so I would plan this carefully. We have taken the decision not to roll out 5.1 until all the initial bugs are ironed out and we have planned the infrastructure appropriately ... I reckon we'll only start this in a few months time.

Hope you get it sorted!

[MichealBlack]

Thank you all for the assistance. I learned a lot about how SNMP works with ESXi 5. The solution ended up being to reinstall ESXi 5 Update 1 using the HP image which has all the necessary stuff in it. I did a complete reinstall, not an upgrade, and was able to setup the networking and use the VMware vSphere CLI to setup the SNMP and Cacti is now happily graphing away.

[jrmunday]

Good news! I guess learning something new is the silver lining :smileyhappy:

Cheers,

Jon

[jrmunday]

Have you got the VMware vSphere CLI installed? If yes, can you run and post the results of the command below;

vicfg-snmp.pl --server myesxihost --username root --password mypassword --show

You should see something like this;

===============================

Current SNMP agent settings:

Enabled  : 1

UDP port : 161

Communities :

public

anothercommunity

Notification targets :

mysnmptarget1.fqdn@162/public

mysnmptarget2.fqdn@162/anothercommunity

Options :

EnvEventSource=sensors

===============================

You can send a test trap as follows;

vicfg-snmp.pl --server myesxihost --username root --password mypassword--test

It would probably be useful if you could add an additional target to see if the trap is received there (perhaps HP-SIM if you have it) ... this will narrow down if the issue is with the host SNMP configuration or your Cacti setup.

[MichealBlack]

I do have the CLI installed on my workstation but when I enter that command I gust get a >> prompt and nothing else.

The same thing happens when I run the command from the vCenter server installation of the CLI.

[Argyle]

Can you verify that the snmp service have started correctly on the esxi host.

Via SSH or console run:

#esxcli network ip connection list

You should see 0.0.0.0:161 under Local Address if its started correctly and is listening on port 161.

Proto  Recv Q  Send Q  Local Address       Foreign Address    State        World ID  World Name

-----  ------  ------  ------------------  -----------------  -----------  --------  ---------------

udp     0      0       0.0.0.0:161         0.0.0.0:0                       20178     hostd-worker

[MichealBlack]

I am not seeing any udp .161 entries at all. I even rebooted the host and still do not see any snmp entry. The /etc/vmware/snmp.xmp file looks OK. I do not see an entry for snmp in the /etc/vmware/service/service.xml file.

[amatteo78]

Hi,

I have same problem with ESXi 5.1, but i can't reinstall server.

I setup snmp, it start but after 1/2 request from other server with snmpwalk, service crash and stop.

I try delete snmp.xml and re-create by vi but nothing. Need reboot host after re-creare snmp.xml ??

Thanks

M.