View Only
  • 1.  VSAN Health - 503 Error

    Posted Oct 09, 2015 05:15 AM

    I'm seeing an error when trying to load the VSAN health tab, no buttons show and I see a 503 error.

    I've checked the permissions on the cert files and they look ok - the van health log file is showing the error below, this is a clean Venter install (6.0, upgrade to U1) and a new VSAN install - I can't find anything online in regards to these errors.

    2015-10-09T05:09:43.880Z CRITICAL vsan-health[MainThread] [VsanHealthServer::UncaughtExcpetionHandler] Traceback (most recent call last):

       File "/usr/lib/vmware-vpx/vsan-health/VsanHealthServer.py", line 354, in <module>

        Initialize(options=gCmdOptions, remainingOptions=gCmdRemainingOptions)

       File "/usr/lib/vmware-vpx/vsan-health/VsanHealthServer.py", line 305, in Initialize


       File "/usr/lib/vmware-vpx/vsan-health/VsanHealthServer.py", line 316, in ImportTypesAndManagedObjects

        import pyMoVsan

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/__init__.py", line 38, in <module>

        __import__(name, globals(), locals(), [])

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanVcClusterHealthSystemImpl.py", line 4134, in <module>


       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanVcClusterHealthSystemImpl.py", line 847, in __init__


       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanEventUtil.py", line 621, in registerHealthAlarms

        cls.registerAlarm(content, content.rootFolder, cls.eventIds, enable=True)

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanEventUtil.py", line 569, in registerAlarm

        unregisterEventIds = cls._getUnRegisterTestId(content, mo, eventIds)

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanEventUtil.py", line 542, in _getUnRegisterTestId

        if alarm.info.expression is None or \

       File "/usr/lib/vmware-vpx/pyJack/pyVmomi/VmomiSupport.py", line 537, in __call__

        return self.f(*args, **kwargs)

       File "/usr/lib/vmware-vpx/pyJack/pyVmomi/VmomiSupport.py", line 360, in _InvokeAccessor

        return self._stub.InvokeAccessor(self, info)

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanPyVmomiProfiler.py", line 95, in InvokeAccessor

        out = self._InvokeAccessor(mo, info)

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanPyVmomiProfiler.py", line 103, in _InvokeAccessor

        return self._InvokeMethod(mo, info, (prop,))

       File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanPyVmomiProfiler.py", line 73, in _InvokeMethod

        obj = ds.Deserialize(resp, info.result)

       File "/usr/lib/vmware-vpx/pyJack/pyVmomi/SoapAdapter.py", line 749, in Deserialize


       File "/usr/lib/vmware-vpx/pyJack/pyVmomi/SoapAdapter.py", line 648, in EndElementHandler

        raise TypeError(data)

    TypeError: vCenter

  • 2.  RE: VSAN Health - 503 Error

    Posted Oct 09, 2015 07:51 AM

    This is the solution for this issue.

    Can someone guide me through a vCenter server windows and not an appliance?



    This is a known issue affecting
    VMware Virtual SAN
    VMware vCenter Server Appliance

    Currently, there is no resolution.
    To work around this issue, correct the permissions for the certificate file:

    1. Log in to the vSphere vCenter Server Appliance using SSH.
    2. Run this command to enable access to the Bash shell:

      shell.set --enabled true

    3. Type shell and press Enter.
    4. Correct the group permissions, by running these commands:

      cd /etc/vmware-vpx/ssl
      chgrp cis rui.* vcsoluser.*
      chmod g+r rui.* vcsoluser.*

    5. After performing the preceding steps, the /var/log/vmware/eam/eam.log file,on the vCenter Server Appliance may report an error similar to:

      [YYYY-MM-DD]T[HH:MM:SS] | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

      To resolve this issue, see After replacing the vCenter Server certificates in VMware vSphere 6.0, the ESX Agent Manager solution user fails to log in (2112577).

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

    Additional Information

  • 3.  RE: VSAN Health - 503 Error

    Posted Oct 12, 2015 12:26 AM

    This didn't fix the issue for me - my permissions already matched what was required. I ended up setting up a brand new venter as it was a new environment so not too difficult, this worked straight away. Interestingly I've just hit the same error again after restarting the vCenter appliance to fix an error with vmware replication installation - on the health check I'm seeing the same 503 Errors:

    The vCenter server is not responding, please check your setup (statusCode 503)

    Unexpected status code: 503

    Unexpected status code: 503

    Cannot get the health service instance.

    I also noticed on the vCenter appliance that the health check is looking at /vsanHealth/ - which in my case shows:

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7f5cfd51ebf0] _serverNamespace = /vsanHealth _isRedirect = false _port = 8006)

  • 4.  RE: VSAN Health - 503 Error

    Posted Oct 12, 2015 08:39 AM

    I'm thinking this is some sort of SSL issue - I thought I'd get the health info from RVC but am getting an error trying to run that as well (as below), I've opened a support case so will see how that goes.

    /localhost> ls

    Errno::EPIPE: Broken pipe

    /usr/lib64/ruby/1.9.1/openssl/buffering.rb:235:in `syswrite'

    /usr/lib64/ruby/1.9.1/openssl/buffering.rb:235:in `do_write'

    /usr/lib64/ruby/1.9.1/openssl/buffering.rb:249:in `write'

    /usr/lib64/ruby/1.9.1/net/protocol.rb:191:in `write0'

    /usr/lib64/ruby/1.9.1/net/protocol.rb:167:in `block in write'

    /usr/lib64/ruby/1.9.1/net/protocol.rb:182:in `writing'

    /usr/lib64/ruby/1.9.1/net/protocol.rb:166:in `write'

    /usr/lib64/ruby/1.9.1/net/http.rb:1739:in `send_request_with_body'

    /usr/lib64/ruby/1.9.1/net/http.rb:1724:in `exec'

    /usr/lib64/ruby/1.9.1/net/http.rb:1189:in `transport_request'

    /usr/lib64/ruby/1.9.1/net/http.rb:1177:in `request'

    /usr/lib64/ruby/1.9.1/net/http.rb:1125:in `request_post'

    /opt/vmware/rvc/gems/rbvmomi-1.7.0/lib/rbvmomi/trivial_soap.rb:90:in `block in request'

    <internal:prelude>:10:in `synchronize'

    /opt/vmware/rvc/gems/rbvmomi-1.7.0/lib/rbvmomi/trivial_soap.rb:88:in `request'

    /opt/vmware/rvc/gems/rbvmomi-1.7.0/lib/rbvmomi/connection.rb:87:in `call'

    /opt/vmware/rvc/gems/rbvmomi-1.7.0/lib/rbvmomi/basic_types.rb:205:in `_call'

    /opt/vmware/rvc/gems/rbvmomi-1.7.0/lib/rbvmomi/basic_types.rb:74:in `block (2 levels) in init'

    /opt/vmware/rvc/lib/rvc/util.rb:213:in `collect_children'

    /opt/vmware/rvc/lib/rvc/extensions/Folder.rb:27:in `children'

    /opt/vmware/rvc/lib/rvc/vim.rb:31:in `children'

    /opt/vmware/rvc/lib/rvc/inventory.rb:76:in `rvc_children'

    /opt/vmware/rvc/lib/rvc/modules/basic.rb:144:in `ls'

    /opt/vmware/rvc/lib/rvc/command.rb:42:in `invoke'

    /opt/vmware/rvc/lib/rvc/shell.rb:129:in `eval_command'

    /opt/vmware/rvc/lib/rvc/shell.rb:73:in `eval_input'

    /opt/vmware/rvc/bin/rvc:178:in `<main>'

  • 5.  RE: VSAN Health - 503 Error

    Posted Oct 19, 2015 01:22 PM


    Any feedback from Vmware regarding the support case?



  • 6.  RE: VSAN Health - 503 Error

    Posted Oct 20, 2015 05:02 AM

    I closed the case in the end, when running through checks I noticed we had a typo in a reverse DNS entry for the VCSA appliance. Fixing this - and rebuilding everything from scratch with the correct DNS solved the problem - is not stable and working as expected.. thanks.

  • 7.  RE: VSAN Health - 503 Error

    Posted Oct 20, 2015 11:19 AM

    To clarify, "is now stable" and not "is not stable".  Correct?  Thank you, Zach.

  • 8.  RE: VSAN Health - 503 Error

    Posted Oct 20, 2015 09:50 PM

    Sorry - "Now" stable :smileyhappy:

  • 9.  RE: VSAN Health - 503 Error

    Posted Apr 07, 2016 08:30 PM

    For anyone searching the web for the elusive VSAN Health Check Plugin 503 error, note another possible cause discussed in this thread:

    Re: Problem with VSAN Health Check windows server (Unexpected status code: 503)

    Dell OpenManage integration seems to break VSAN Health Check, in my experience it was true for VSAN 6.1 atop vCenter on Windows and vCenter Appliance.  Unregistering OpenManage from the affected vCenters in addition to verifying the CA certificates via 3 KBs I noted in my reply to the above discussion resolved the issue for us.