VMware vSphere

 View Only
  • 1.  esxi connect to localhost failed

    Posted Mar 27, 2013 01:05 PM

    I have a 10 esxi host 5.1 vsphere farm and one of my hosts 'out of the blue' lost all its performance metrics.. i.e. reported 0% mem/cpu etc.  I then attempted to restart management agent from cli and got a bunch of "connect to localhost failed" errors.  Host is now disconnected from vCenter and I cannot reconnect.  Performing a "test management network" from esxi system config console it will resolve everything except local hostname.

    I have verified:

    - all running VM's are fine

    - esxi hostname still in DNS

    - I can ping/putty to esxi host

    - DNS and all other network settings are still configured correctly on host

    Below is what I get when attempt the management agent restart:



    ~ #
    ~ #
    ~ # ./sbin/services.sh restart
    Running vmware-fdm stop
    Stopping vmware-fdm:success
    Running xorg stop
    Running wsman stop
    Stopping openwsmand
    Openwsmand is not running.
    Running sfcbd stop
    This operation is not supported.
    Please use /etc/init.d/sfcbd-watchdog stop
    Running snmpd stop
    root: snmpd is not running.
    Running sfcbd-watchdog stop
    sh: bad number
    sh: you need to specify whom to kill
    sfcbd is not running.
    Connect to localhost failed: Connection failure
    Connect to localhost failed: Connection failure
    Running vpxa stop
    vpxa is not running
    Connect to localhost failed: Connection failure
    Running vobd stop
    vobd is not running
    Running dcbd stop
    dcbd is not running
    Running smartd stop
    smartd is not running
    Running cdp stop
    cdp is not running
    Running lacp stop
    lacp is not running
    Running memscrubd stop
    memscrubd is not running
    Running slpd stop
    Stopping slpd
    Running hostd stop
    hostd is not running.
    Running sensord stop
    sensord is not running
    Running storageRM stop
    storageRM is not running
    Running lbtd stop
    net-lbt is not running
    Running vprobed stop
    vprobed is not running
    Running rhttpproxy stop
    rhttpproxy is not running.
    Running usbarbitrator stop
    usbarbitrator is not running
    Running SSH stop
    SSH login disabled
    VobUserLib_Init failed with -1
    Connect to localhost failed: Connection failure
    Errors:
    Invalid operation requested: This ruleset is required and connot be disabled
    Running DCUI stop
    Disabling DCUI logins
    VobUserLib_Init failed with -1
    Running ESXShell stop
    ESXi shell login disabled
    VobUserLib_Init failed with -1
    Running ntpd stop
    Stopping ntpd
    Connect to localhost failed: Connection failure
    Running ntpd restart
    Connect to localhost failed: Connection failure
    Starting ntpd
    Running ESXShell restart
    ESXi shell login enabled
    VobUserLib_Init failed with -1
    Running DCUI restart
    Enabling DCUI login: runlevel =
    VobUserLib_Init failed with -1
    Running SSH restart
    Connect to localhost failed: Connection failure
    SSH login enabled
    VobUserLib_Init failed with -1
    Running usbarbitrator restart
    ~ # ./sbin/services.sh restart
    Running vmware-fdm stop
    Stopping vmware-fdm:success
    Running xorg stop
    Running wsman stop
    Stopping openwsmand
    Openwsmand is not running.
    Running sfcbd stop
    This operation is not supported.
    Please use /etc/init.d/sfcbd-watchdog stop
    Running snmpd stop
    root: snmpd is not running.
    Running sfcbd-watchdog stop
    sh: bad number
    sh: you need to specify whom to kill
    sfcbd is not running.
    Connect to localhost failed: Connection failure
    Connect to localhost failed: Connection failure


    Thanks in advance,

    Jaime



  • 2.  RE: esxi connect to localhost failed

    Posted Mar 27, 2013 04:25 PM

    Can you run "stat -f /" to check you free inodes?



  • 3.  RE: esxi connect to localhost failed

    Posted Mar 27, 2013 04:40 PM


    ~ # stat -f /
      File: "/"
        ID: 100000000 Namelen: 127     Type: visorfs
    Block size: 4096
    Blocks: Total: 370937     Free: 253870     Available: 253870
    Inodes: Total: 524288     Free: 521311



  • 4.  RE: esxi connect to localhost failed

    Posted Mar 27, 2013 05:04 PM

    That looks Ok, I was wondering if it was the same issue that I had recently;

    http://communities.vmware.com/message/2180248



  • 5.  RE: esxi connect to localhost failed

    Posted Apr 01, 2013 08:18 PM

    This issue turned out to be due to scsi reservations.  VMWare support found lines similar to this in vmkernel.log:

    2013-03-28T17:16:21.532Z cpu14:10076)FS3: 1250: vol 'VM05', lock at 62275584: [Req mode: 1] Checking liveness:

    2013-03-28T17:16:21.532Z cpu14:10076)[type 10c00001 offset 62275584 v 3658, hb offset 3223552
    gen 11, mode 1, owner 511e4014-d3346516-baac-xxxxxxxxxxxx mtime 381885

    Mac address x'd out.  We found the esxi host that contained the NIC with that MAC and rebooted it, this cleared up the reservations from the logs, at which point I was also able to re-connect the orphaned host.



  • 6.  RE: esxi connect to localhost failed

    Posted Apr 02, 2013 07:17 AM

    Glad you've resolved it, thanks for sharing the fix.