VMware vSphere

 View Only
Back to discussions
Expand all | Collapse all

VCSA 6.5 Web GUI Error

  • 1.  VCSA 6.5 Web GUI Error

    Posted Nov 22, 2016 01:55 AM

    I have successfully upgraded my VCSA 6 to 6.5 - after the upgrade I was able to login and use the vsphere web client without issue.

    I then upgraded my single ESXi 6 host to 6.5, which was successful.  After booting up the VCSA, after logging into the Vsphere client I get the following error:-

    Could not connect to one or more vCenter Server systems:

    https://FQDN:443/sdk

    At this point the client is unusable.  I have tried stopping and restarting all services via SSH, and rebooting the VCSA, however nothing has helped.

    I have also tried following the instructions here “Failed to verify the SSL certificate for one or more vCenter Server Systems” error in the vSphere Web Client (2050273) … - However I only see 1 Service ID, so there doesn't appear to be a duplicate entry..

    Many thanks



  • 2.  RE: VCSA 6.5 Web GUI Error

    Broadcom Employee
    Posted Nov 22, 2016 04:10 AM

    Can you attach the sceenshot of the error? Are you able to login the webclient? and if possible attach the vcsupport bundle.



  • 3.  RE: VCSA 6.5 Web GUI Error

    Posted Nov 22, 2016 01:56 PM

    Hi - This is a screenshot of the error.

    I am able to login to the ESXi host web GUI.   This issue occurs with the vSphere web client.

    Many thanks



  • 4.  RE: VCSA 6.5 Web GUI Error

    Posted Nov 22, 2016 06:11 PM

    Hi,

    Disable Ad-Block to your Browser.

    Thanks,



  • 5.  RE: VCSA 6.5 Web GUI Error

    Posted Nov 30, 2016 11:40 PM

    Hello all,

    To add to this thread -- I experienced the exact same issue yesterday for no obvious reason. I was using the vCSA (Connected to several VMs), then out of nowhere it dropped. The appliance is online and supposedly it is "Good" and "Healthy". I have rebooted the appliance, but no avail.

    This is the output from cli:

    Command> service-control --status --all

    StartPending:

    vmware-sps vmware-updatemgr

    Running:

    applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

    Stopped:

    pschealth vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-psc-client vmware-rbd-watchdog vmware-vcha vmware-vpxd

    I can see that there are some services that shouldn't be in a stopped state, so I ran below:

    Command> service-control --start --all

    Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True

    2016-11-30T23:14:40.820Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']

    2016-11-30T23:14:40.824Z   Done running command

    2016-11-30T23:14:40.826Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:40.826Z   Running command: ['/sbin/service', u'lwsmd', 'status']

    2016-11-30T23:14:40.846Z   Done running command

    Successfully started service lwsmd

    2016-11-30T23:14:40.849Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']

    2016-11-30T23:14:40.854Z   Done running command

    2016-11-30T23:14:40.856Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:40.856Z   Running command: ['/sbin/service', u'vmafdd', 'status']

    2016-11-30T23:14:40.875Z   Done running command

    Successfully started service vmafdd

    2016-11-30T23:14:40.878Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']

    2016-11-30T23:14:40.883Z   Done running command

    2016-11-30T23:14:40.885Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:40.885Z   Running command: ['/sbin/service', u'vmdird', 'status']

    2016-11-30T23:14:40.904Z   Done running command

    Successfully started service vmdird

    2016-11-30T23:14:40.907Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']

    2016-11-30T23:14:40.912Z   Done running command

    2016-11-30T23:14:40.914Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:40.914Z   Running command: ['/sbin/service', u'vmcad', 'status']

    2016-11-30T23:14:40.933Z   Done running command

    Successfully started service vmcad

    2016-11-30T23:14:40.936Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']

    2016-11-30T23:14:40.941Z   Done running command

    2016-11-30T23:14:40.943Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:40.943Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']

    2016-11-30T23:14:40.962Z   Done running command

    Successfully started service vmware-sts-idmd

    2016-11-30T23:14:40.964Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']

    2016-11-30T23:14:40.969Z   Done running command

    2016-11-30T23:14:40.971Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:40.971Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']

    2016-11-30T23:14:40.991Z   Done running command

    Successfully started service vmware-stsd

    2016-11-30T23:14:40.994Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']

    2016-11-30T23:14:40.998Z   Done running command

    2016-11-30T23:14:41.000Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:41.001Z   Running command: ['/sbin/service', u'vmdnsd', 'status']

    2016-11-30T23:14:41.023Z   Done running command

    Successfully started service vmdnsd

    2016-11-30T23:14:41.025Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']

    2016-11-30T23:14:41.030Z   Done running command

    2016-11-30T23:14:41.032Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2016-11-30T23:14:41.032Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']

    2016-11-30T23:14:41.052Z   Done running command

    Successfully started service vmware-psc-client

    Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start pschealth services. Error: Service crashed while starting

    /var/log/vmware/psc-client/catalina.2016-12-01.log:

    01-Dec-2016 09:24:53.831 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-bio-7081"]

    01-Dec-2016 09:24:53.844 WARNING [main] com.springsource.tcserver.serviceability.rmi.JmxSocketListener.init Not initializing due to port being set to -1

    01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina

    01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Pivotal tc Runtime 3.1.4.SR1/8.0.33.A.RELEASE

    01-Dec-2016 09:24:53.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war

    01-Dec-2016 09:24:54.233 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

    01-Dec-2016 09:24:54.288 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war has finished in 369 ms

    01-Dec-2016 09:24:54.291 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war

    01-Dec-2016 09:24:54.392 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

    01-Dec-2016 09:24:54.394 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war has finished in 102 ms

    01-Dec-2016 09:24:54.399 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/psc.war

    /var/log/vmware/psc-client/localhost.2016-11-30.log

    30-Nov-2016 03:30:25.252 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

    30-Nov-2016 03:30:25.494 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

    30-Nov-2016 14:30:55.483 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

    30-Nov-2016 14:30:55.577 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

    30-Nov-2016 14:31:00.159 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

    30-Nov-2016 15:06:20.403 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

    30-Nov-2016 15:06:20.579 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

    30-Nov-2016 15:06:25.827 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

    30-Nov-2016 04:15:17.420 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

    30-Nov-2016 04:15:17.458 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

    30-Nov-2016 04:16:25.727 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

    30-Nov-2016 04:16:26.596 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

    30-Nov-2016 04:17:00.499 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

    30-Nov-2016 22:24:30.373 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

    30-Nov-2016 22:24:30.544 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

    30-Nov-2016 22:25:14.320 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

    30-Nov-2016 22:25:14.412 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

    30-Nov-2016 22:25:19.200 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

    /var/log/vmware/psc-client/psc-client.log | grep -i WARN:

    [2016-11-30 22:25:17.621] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter

    [2016-11-30 22:25:17.622] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionListener

    [2016-11-30 22:25:17.623] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionObjectFilter

    [2016-11-30 22:25:17.940] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider

    [2016-11-30 22:25:17.941] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider

    [2016-11-30 22:25:17.943] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.PropertyProviderBean

    [2016-11-30 22:25:17.944] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.DataProviderAdapter

    [2016-11-30 22:25:17.948] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.provider.ExpressionProvider

    [2016-11-30 22:25:18.523] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.vim.security.sso.SsoServiceProvider

    [2016-11-30 22:25:18.645] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.mvc.objectnavigator.interpreter.NavigationRequestInterpreter

    [2016-11-30 22:25:19.057] [WARN ] localhost-startStop-1 org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy - Retained attributes is deprecated. Override the extractAttributes() method instead.

    [2016-11-30 22:25:47.705] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter

    [2016-11-30 22:25:47.707] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

    [2016-11-30 22:25:47.716] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

    [2016-11-30 22:25:47.751] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.impl.JeePluginLoader - Directory/usr/lib/vmware-psc-client/webapps/ROOT is not a plugin (no plugin.xml file).

    [2016-11-30 22:25:47.876] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

    [2016-11-30 22:25:47.878] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

    [2016-11-30 22:25:47.893] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

    [2016-11-30 22:25:47.895] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

    /var/log/vmware/vpxd/vpxd.log | grep -i ERROR:

    2016-11-30T23:14:41.713Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file  : /etc/motd)

    2016-11-30T23:14:41.717Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument

    2016-11-30T23:14:43.900Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: FileIO error: Could not find file  : /var/lib/vmware/hpMetadataCache.xml

    2016-11-30T23:14:44.521Z error vpxd[7F2FE8923700] [Originator@6876 sub=vmomi.soapStub[5]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

    2016-11-30T23:14:44.648Z error vpxd[7F2FE31E3700] [Originator@6876 sub=vmomi.soapStub[7]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

    2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for embeddedEsx version e.x.p: vpx-upgrade-eesx-1-linux-*

    2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for esx version e.x.p: vpx-upgrade-esx-7-linux-*

    2016-11-30T23:14:44.992Z error vpxd[7F2FE12A5700] [Originator@6876 sub=vmomi.soapStub[9]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

    2016-11-30T23:14:45.014Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

    2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"

    2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

    2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

    2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

    2016-11-30T23:14:45.147Z error vpxd[7F2FE32E5700] [Originator@6876 sub=vmomi.soapStub[10]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

    2016-11-30T23:14:45.404Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: FileIO error: Permission denied for file  : /etc/vmware-vpx/vpxd.cfg.tmp

    2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.

    2016-11-30T23:14:45.735Z error vpxd[7F2FE1428700] [Originator@6876 sub=vmomi.soapStub[17]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

    2016-11-30T23:14:45.833Z error vpxd[7F2FE3EFD700] [Originator@6876 sub=vmomi.soapStub[19]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

    /var/log/vmware/vpxd/vpxd.log | grep -i WARN:

    2016-11-30T23:14:41.709Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-mob-pipe. Removing...

    2016-11-30T23:14:43.385Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] Init [LocalizationManagerMo::Init()] took 1666 ms

    2016-11-30T23:14:43.506Z warning vpxd[7F2FE8186700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:44.369Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-webserver-pipe. Removing...

    2016-11-30T23:14:44.377Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension com.vmware.vim.stats.report

    2016-11-30T23:14:44.378Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension com.vmware.vim.vcha

    2016-11-30T23:14:44.390Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension hostdiag

    2016-11-30T23:14:44.521Z warning vpxd[7F2FE8923700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:44.648Z warning vpxd[7F2FE31E3700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:44.801Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 3217 ms

    2016-11-30T23:14:44.992Z warning vpxd[7F2FE12A5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:45.119Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')

    2016-11-30T23:14:45.147Z warning vpxd[7F2FE32E5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:45.187Z warning vpxd[7F2FE34E9700] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')

    2016-11-30T23:14:45.347Z warning vpxd[7F2FE13A7700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.

    2016-11-30T23:14:45.735Z warning vpxd[7F2FE1428700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:45.833Z warning vpxd[7F2FE3EFD700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

    2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

    2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

    2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

    2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

    2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

    2016-11-30T23:14:51.549Z warning vpxd[7F2FE2040700] [Originator@6876 sub=VpxProfiler opID=StatsTruncateExpiredPartitions-618fb492] StatsTruncateExpiredPartitions [TotalTime] took 6230 ms

    2016-11-30T23:14:53.264Z warning vpxd[7F2FE2AD5700] [Originator@6876 sub=OMM opID=HostSync-host-34-11775afc-SWI-65bb0468] Cannot download vm config of [vim.VirtualMachine:vm-1166,MikroTikCHR-6.37.2]: Received SOAP response fault from [<cs p:00007f2fcc58f600, TCP:esxi02.domain.local:443>]: downloadVMXConfig

    Checked disk space as well (Just incase):

         

    FilesystemSizeUsedAvailUse%Mounted on
    devtmpfs4.9G04.9G0%/dev
    tmpfs4.9G12K4.9G1%/dev/shm
    tmpfs4.9G700K4.9G1%/run
    tmpfs4.9G04.9G0%/sys/fs/cgroup
    /dev/sda311G4.0G6.1G40%/
    tmpfs4.9G11M4.9G1%/tmp
    /dev/mapper/netdump_vg-netdump985M1.3M932M1%/storage/netdump
    /dev/mapper/autodeploy_vg-autodeploy9.8G23M9.2G1%/storage/autodeploy
    /dev/mapper/core_vg-core25G1.4G22G6%/storage/core
    /dev/mapper/log_vg-log9.8G241M9.0G3%/storage/log
    /dev/sda1120M28M87M24%/boot
    /dev/mapper/imagebuilder_vg-imagebuilder9.8G23M9.2G1%/storage/imagebuilder
    /dev/mapper/dblog_vg-dblog15G134M14G1%/storage/dblog
    /dev/mapper/seat_vg-seat9.8G75M9.2G1%/storage/seat
    /dev/mapper/db_vg-db9.8G130M9.1G2%/storage/db
    /dev/mapper/updatemgr_vg-updatemgr99G64M94G1%/storage/updatemgr

    This is a fresh 6.5 lab environment as well.

    Thanks in advance!



  • 6.  RE: VCSA 6.5 Web GUI Error

    Posted Dec 14, 2016 10:09 PM

    We have exactly the same situation. Dropped out out of the blue. Also had 2 out of 3 hosts on ESXi 6.5.  Both ESXi hosts upgraded from 6.0 to 6.5.

    Also we migrated from VCSA 6.0 to 6.5.  Rebooted before and all was fine.

    VPXD logs say something interessting:

    ...

    2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

    2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Execution elapsed time: 0 ms

    2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Statement diagnostic data from driver is 23505:0:7:ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

    --> Error while executing the query

    ...

    2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from 23505:7

    2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110

    2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

    --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

    2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] DoHostSync:host-9 [DoHostSyncTime] took 2670 ms

    2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] InvtHostSyncLRO::StartWork [HostSyncTime] took 2670 ms

    2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=vpxCommon opID=HostSync-host-9-1d836128] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...

    2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

    --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

    2016-12-14T21:53:03.765Z panic vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128]

    -->

    --> Panic: Unrecoverable VmRootError. Panic!

    --> Backtrace:

    --> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-4602587, tag: vpxd, cpu: x86_64, os: linux, buildType: release

    --> backtrace[00] libvmacore.so[0x002B9EA8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)

    --> backtrace[01] libvmacore.so[0x001B4E90]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

    --> backtrace[02] libvmacore.so[0x002B7E5B]

    --> backtrace[03] libvmacore.so[0x002B7F36]: Vmacore::PanicExit(char const*)

    --> backtrace[04] vpxd[0x00722E78]

    --> backtrace[05] vpxd[0x00738629]

    --> backtrace[06] vpxd[0x0073A837]

    --> backtrace[07] vpxd[0x0073FB68]

    --> backtrace[08] libvmacore.so[0x0023ED6F]

    --> backtrace[09] libvmacore.so[0x0023F056]

    --> backtrace[10] libvmacore.so[0x002BF67B]

    --> backtrace[11] libpthread.so.0[0x00007454]

    --> backtrace[12] libc.so.6[0x000E8E2D]

    --> [backtrace end]

    What is going on?

    Has anybody a suggestion?



  • 7.  RE: VCSA 6.5 Web GUI Error

    Posted Dec 15, 2016 01:06 AM

    In the end I ended up creating a new vCenter VM and removing the old one.

    Clearly things go amiss during the upgrade, even though the installer states the process is successful.

    Regards



  • 8.  RE: VCSA 6.5 Web GUI Error

    Posted Dec 15, 2016 08:39 AM

    There isn't enough information in the log snippet. The issue looks similar to vCenter – VdbODBCError – unique constraint pk_vpx_vm_virtual_device – fix | vMan where you can see possible workaround



  • 9.  RE: VCSA 6.5 Web GUI Error



  • 10.  RE: VCSA 6.5 Web GUI Error

    Posted Jun 10, 2019 03:18 PM

    vpxd is crashing because of duplicate id's  in VCDB. Connect to VCDB and remove the duplicate one's.

    /var/log/vmware/vpostgres/postgresql.log would provide the duplicate id.

    Ex:-

    postgresql.log

    UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"

    UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists.

    DELETE FROM vc.vpx_vm_virtual_device where id='361' and device_key='3002'; 

    replace the id with the one's from logs.

    Regards,

    AJ



  • 11.  RE: VCSA 6.5 Web GUI Error

    Posted Dec 15, 2016 05:30 AM

    Hi ,

    I also faced the same issue .

    The error can be cleared , if you remove  the certificate it won't pop up the error , but vcenter inventory was not showing there .

    I got this issue after a week from the installation , one week it was smooth .


    You may check the NTP configuration  and try to migrate to another esxi host and check .



  • 12.  RE: VCSA 6.5 Web GUI Error

    Posted Feb 25, 2017 04:31 PM

    I have done some more digging into this issue, and it seems to crop up if I upgrade the VCSA first and then update my hosts afterwards.   I recently performed an update of VCSA 6.5 to 6.5a and rebooted and the web GUI worked without issue.  Once I upgraded my ESXi hosts from 6.5 to 6.5a the problem occurred.

    Something seems to break with respect to communication between the VCSA and the hosts after upgrading the hosts. 



  • 13.  RE: VCSA 6.5 Web GUI Error

    Posted Apr 18, 2017 11:41 AM

    Hi All,

    I have exact same problem. Two vCenter servers in linked mode with external PSCs. One day suddenly one of Vcenter servers stopped to respond. The issue that we have is not related to doubled DB entry as other suggested that.In logs I see exact same entries as misan.

    Additionally I have checked time on both VCs and PSCs and replication between PSCs, all looks good. I have tried to re-generate certificate on PSC where affected VCs is connecting to, but this has failed. I have also performed upgrade to VCSA 6.5c build number 5400, no luck, still cannot start vpxd service. Hopefully this is a test lab, but I would like to avoid reinstallilng it as it runs dvswitch with bunch of VMs connected to it and Primary NSX controller for Cross-vCenter implementation. Hope that someone from VMware will take a look on this. At this point I do not recommend to go into production with VCSA 6.5.



  • 14.  RE: VCSA 6.5 Web GUI Error

    Posted Jun 23, 2017 07:59 PM

    Hi All,

    today we had the same issue as described above.

    To remember:

    Could not connect to one or more vCenter Server systems:

    https://FQDN:443/sdk

    On VCSA Shell:

    service-control --status vmware-vpxd shows "stopped"

    service-control --start vmware-vpxd starts the service starts for a couple of seconds and stops again

    We contacted VMware Support and they analyzed the core dump logs (/var/core) and could figure out a problem with a single VM in our environment.

    There was a problem with a network adapter on that VM (DB entry NULL)

    Solution that worked for us:

    Find VM

    /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres (Login to postgres)

    select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name=''; (get affected VM)

    select * from vpx_entity where id = '<YOURID>'; (get affected VM Name)

    Fix VM / vpxd (for our specific issue with database issue)

    Remove Network Adapter, save config, readd Network Adapter

    service-control --start vmware-vpxd

    You should now be able to login to your vCenter.

    So the NULL Value in the database seem to be a really rare issue. In case of "Could not connect to.....:" just Find VM unregister it from vCenter and Reregister.

    Hope this helps somebody!

    Regards,

    Gabriel



  • 15.  RE: VCSA 6.5 Web GUI Error

    Posted Jul 11, 2017 07:54 PM

    Had the same issue as you although I didn't interpret solution very clearly. You state to remove the network adapter from the VM which I wrongfully assumed you had to do from the database instead of the local webui/management of the esxi server. I went ahead and removed the problem VM from the inventory of the esxi server and a restart of vpxd on VCSA had everything back up and running instantly.

    What broke this for me I believe is that I copied a VM from Workstation 12 onto a datastore and registered that VM via VCSA. Shortly after starting the problem VM, VCSA did freak out and have this problem. Now I'm in the process of using vCenter Converter with hopes I don't have to repeat this!



  • 16.  RE: VCSA 6.5 Web GUI Error

    Posted Jul 14, 2017 04:37 PM

    That worked for me.  Thanks!



  • 17.  RE: VCSA 6.5 Web GUI Error

    Posted Jul 18, 2017 11:14 AM

    This works for me to.

    Thanks for writing this solution.

    Regards

    Pascal



  • 18.  RE: VCSA 6.5 Web GUI Error

    Posted Aug 09, 2017 04:14 PM

    Gabriel

    Thanks a million for posting this, (shame the original poster didn't give you the points)

    We had this on our semi production enviroment and you saved me a call to support.

    In my case I think this came from modifing the vm from the vmrc client through the html5 web

    A couple of heads ups

    When runing the select command the ip address is that of the host, so it's easy to find where the vms is located

    VCDB=# select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name='';

    id  |                                  local_file_name                                                                               |  ip_address

    -----+-----------------------------------------------------------------------------------+--------------

    608 | /vmfs/volumes/5953bbbb7-05cbbb38-82f4-00bbbfafb56/NTvmXXXX/NTvmXXXX.vmx    | 10.111.210.90

    (1 row)

    It wasn't possible to modify the vm, so we created a local vSwitch and port group and the vm assigned itself to that

    thanks again

    Russ



  • 19.  RE: VCSA 6.5 Web GUI Error

    Posted Apr 12, 2018 11:18 PM

    This solved my problem as well

    I was still waiting for VMWare support to come back to me and found this page

    It seems that vCenter is a little delicate if it is completely stopped with some bad data

    The cause of it for me was an ip address conflict between a VM guest and VM Host on different site to the vCenter server



  • 20.  RE: VCSA 6.5 Web GUI Error

    Posted Aug 17, 2017 08:41 PM

    TLDR; If you have any v5.x Host Profiles, delete them.

    We recently upgraded our v5.5 Windows vCenter with embedded SSO to a v6.5 vCSA with an external PSC. Everything seemed stable until we updated our ESXi hosts to v6.5, at which point interacting with the hosts in the Flash client (at least this seemed to be the most reliable trigger) would cause the vpxd service to crash.

    None of the fixes in this thread worked for us, but a VMware tech stopped by today and we informed her that we were having this problem. She immediately told us to delete our v5.x Host Profiles, as they had recently seen this problem at another site. We had one v5.x Host Profile, and deleted it. After some testing, deleting the Host Profile appears to have resolved our vpxd service crashes.



  • 21.  RE: VCSA 6.5 Web GUI Error

    Posted Aug 31, 2018 10:50 AM

    A colleague got this solution for at least improving web client performance.

    Manually increasing the heap memory on vCenter Server components in vCenter 6.x (2150757)

    https://kb.vmware.com/s/article/2150757

    hope it reduces the frustration of using the web client.



  • 22.  RE: VCSA 6.5 Web GUI Error

    Posted Jan 06, 2022 12:00 AM

    If Ad-Block is running in your browser, disable it as I encountered an error loading the vCenter UI - Disable Ad-Block resolved it.