Peer table full for sfcbd

[supahted]

Hi,

I am currently testing ESXi 4 to evaluate an upgrade of our ESXi 3.5 hosts to ESXi 4. The tests hosts areHP BL460c G6 blades running ESXi 4 build 175625 connected to a HP EVA 6000 storage array. The ESXi 4 host seems to run fine but i noticed the following kernel warnings in the system log:

Jul 29 19:53:49 Hostd: 2009-07-29 19:53:49.287 604C6B90 verbose 'DvsManager' PersistAllDvsInfo called

Jul 29 19:53:56 vmkernel: 0:06:03:33.862 cpu12:98154)VMK_PCI: 728: device 006:00.0 capType 16 capIndex 76

Jul 29 19:53:56 vmkernel: 0:06:03:33.862 cpu12:98154)VMK_PCI: 728: device 006:00.1 capType 16 capIndex 76

Jul 29 19:53:56 vmkernel: 0:06:03:33.863 cpu12:98154)WARNING: UserLinux: 1710: UNIMPLEMENTED! write-back of mmap regions unsupported

Jul 29 19:53:56 vmkernel: 0:06:03:33.863 cpu12:98154)VMK_PCI: 728: device 006:00.0 capType 16 capIndex 76

Jul 29 19:53:56 vmkernel: 0:06:03:33.874 cpu12:98154)VMK_PCI: 728: device 006:00.1 capType 16 capIndex 76

Jul 29 19:53:57 sfcb[98827]: INTERNAL StorelibManager::createDefaultSelfCheckSettings - failed to get TopLevelSystem

Jul 29 19:53:57 vmkernel: 0:06:03:34.837 cpu2:5535)WARNING: UserThread: 402: Peer table full for sfcbd

Jul 29 19:53:57 vmkernel: 0:06:03:34.837 cpu2:5535)WARNING: World: vm 99478: 1111: init fn user failed with: Out of resources!

Jul 29 19:53:57 Hostd: 2009-07-29 19:53:57.941 603C2B90 info 'ha-license-manager' Update: 78774 minutes left in eval period

Jul 29 19:53:57 Hostd: 2009-07-29 19:53:57.942 604C6B90 verbose 'ha-license-manager' Load: Loading existing file: /etc/vmware/license.cfg

Jul 29 19:53:57 Hostd: 2009-07-29 19:53:57.954 604C6B90 verbose 'ha-license-manager' Load: Loading existing file: /etc/vmware/license.cfg

Jul 29 19:53:59 vmkernel: 0:06:03:37.052 cpu5:5535)WARNING: UserThread: 402: Peer table full for sfcbd

Jul 29 19:53:59 vmkernel: 0:06:03:37.052 cpu5:5535)WARNING: World: vm 99641: 1111: init fn user failed with: Out of resources!

Jul 29 19:54:01 cimslp: Found 40 profiles in namespace root/interop

Jul 29 19:54:02 vmkernel: 0:06:03:39.483 cpu5:5535)WARNING: UserThread: 402: Peer table full for sfcbd

Jul 29 19:54:02 vmkernel: 0:06:03:39.483 cpu5:5535)WARNING: World: vm 100180: 1111: init fn user failed with: Out of resources!

Jul 29 19:54:04 vmkernel: 0:06:03:41.707 cpu7:5535)WARNING: UserThread: 402: Peer table full for sfcbd

Jul 29 19:54:04 vmkernel: 0:06:03:41.707 cpu7:5535)WARNING: World: vm 100316: 1111: init fn user failed with: Out of resources!

Jul 29 19:54:06 vmkernel: 0:06:03:43.931 cpu5:5535)WARNING: UserThread: 402: Peer table full for sfcbd

Jul 29 19:54:06 vmkernel: 0:06:03:43.931 cpu5:5535)WARNING: World: vm 100454: 1111: init fn user failed with: Out of resources!

Jul 29 19:54:08 cimslp: Found 40 profiles in namespace root/interop

Jul 29 19:54:08 cimslp: SLP data collection finished

Jul 29 19:54:15 Hostd: 2009-07-29 19:54:15.321 60485B90 verbose 'DvsTracker' FetchSwitches: added 0 items

Jul 29 19:54:15 Hostd: 2009-07-29 19:54:15.321 60485B90 verbose 'DvsTracker' FetchDVPortgroups: added 0 items

Jul 29 19:54:29 Hostd: 2009-07-29 19:54:29.404 60340B90 verbose 'Cimsvc' Ticket issued for CIMOM version 1.0, user root

Jul 29 19:54:44 Hostd: 2009-07-29 19:54:44.029 603C2B90 verbose 'ResourcePool ha-root-pool' Root pool capacity changed from 17477MHz/32099MB to 17477MHz/32097MB

Jul 29 19:55:27 vmkernel: 0:06:05:05.033 cpu0:98456)WARNING: UserLinux: 1710: UNIMPLEMENTED! write-back of mmap regions unsupported

Jul 29 19:55:27 vmkernel: 0:06:05:05.033 cpu0:98456)WARNING: UserCartel: 1820: Fork not supported for multithreaded parents.

I know that the sfcb stands for Small-Footprint CIM Broker. ESXi 3.5 update 4 has a problem with the CIM agents that results in errors that look like the above errors (see ). However the knowledge base article mentioned is does affect ESXi 3.5 but not ESXi 4.0.

Is anybody else experiencing these warnings on ESXi 4?

Regards,

Ted Steenvoorden

[EricTam]

Hi Ted

Yes, I am experiencing the same in ESXi v4, BL460c G6, currently is in test, therefore, no SAN attached to it yet. Also, I am using HP version of VMWare.

I have a support ticket with VM. Here is the response.

As I understand it from the case notes, the issue we are seeing is " Service Console filled with warnings - "Peer table full for sfcbd""
For the errors related to Service console -
1. Stop the following three services: sfcbd-watchdog, openwsmand and slpd:
# /etc/init.d/sfcbd-watchdog stop
Stopping sfcbd
# /etc/init.d/wsmand stop
Stopping openwsmand
# /etc/init.d/slpd stop
Stopping slpd
2. Check which services are configured to start at boot:
# chkconfig --list
The output appears similar to:
sfcbd-watchdog on
wsmand on
slpd on
sfcbd on
ntpd off
hostd on
vmware-aam on
vmware-vpxa on
Note: Whether ntpd or vmware-aam are running or not depends on the configuration of the system and these do not need to be changed.
3. Disable the following four services starting at boot: sfcbd-watchdog, wsmand, slpd, sfcbd.
# chkconfig sfcbd-watchdog off
# chkconfig wsmand off
# chkconfig slpd off
# chkconfig sfcbd off
4. Verify the services were reconfigured to not start at boot:
# chkconfig --list
The output appears similar to:
sfcbd-watchdog off
wsmand off
slpd off
sfcbd off
ntpd off
hostd on
vmware-aam on
vmware-vpxa on
This should be persistent after reboot.
5. Run backup.sh 0 to save the config to the usb:
# backup.sh 0

I will check with VM as I do not want to turn off CIM.

[supahted]

Hi Erik,

I also contacted VMware and have posted the results on my blog. You can find the article here:

Let me know if you have some new information!

Regards,

Ted

blog:

[EricTam]

Latest update from VM that message can be ignore, see below

I also checked with my Senior engineer about the messages - "Peer table full for sfcbd". As per the discussion if you are using HP CIM these messages are ignorable. This ideally means that the vmkernel is sluggish in cleaning up the worlds. There has been requests to hide these messages and the engineering is already aware of this. Hence I guess we should be fine.

[bulletprooffool]

There is a bug with the ESX CIM agents - we had it with a 3.5i box, running the HP CIM agents.

You can either stop the service running . . . or just bear with it and set some alerts to let you know if there are any issues. (perhaps set strong automated DRS rules to protect yourself)

I am not sure, but I believe the latest revision of firmware should have resolved this, as I have stopped seeing these issues.

[mr_vaughn]

I have the same thing happening to a clients HP DL360 G6.

All latest firmware since Sep 6th

[mblake4u]

I have a similar problem:

vmkernel: 260:01:05:32.487 cpu10:70913510)WARNING: UserThread: 406: Peer table full for sfcbd

vmkernel: 260:01:05:32.487 cpu10:70913510)WARNING: World: vm 69455488: 910: init fn user failed with: Out of resources!

vmkernel: 260:01:05:32.487 cpu10:70913510)WARNING: World: vm 69455488: 1775: WorldInit failed: trying to cleanup.

using ESXi 3.5.0 build 123629 (which I think it update 2 if I'm not mistaken). I don't have the Dell CIM bundle installed on this server, and what I usually do when I see this is restart the services '/sbin/services.sh restart'. There's always the following error:

watchdog-openwsmand: PID file /var/run/vmware/watchdog-openwsmand.PID not found

watchdog-openwsmand: Unable to terminate watchdog: Can't find process

and a subsequent restart doesn't produce this error (until I've received the vmkernel errors again in the syslog). Some of our servers are now on 4.1.0 with the Dell CIM bundle and we don't get the error anymore, but occasionally get:

Nov 9 14:55:33 cimslp: --- HTTP-Daemon no longer active. Deregistering service with slp

Nov 9 14:55:33 cimslp: --- Error deregistering service with slp (-22) ... it will now timeout

errors. Has anybody else noticed these ones?