vCenter 6 fails to restart after IP change

[darkdragon001]

I changed the vCenter 6 IP from within web client.

Since I had problems connecting to the new IP, I restarted the server via vSphere Client directly on the host using VMware Tools.

After the restart, I can't connect to vCenter any more. Trying to connect via https shows me the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f4158068bf0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Then I directly used the console, enabled ssh and bash and logged in there.

I tried to check the current IP settings via the script used in version 5.5 which still exists (/opt/vmware/share/vami/vami_config_net) but fails with multiple errors.

Then, I tried to reset the certificates via certificate manager (/usr/lib/vmware-vmca/bin/certificate-manager), but I could not login via SSO (seems that SSO service could also not be started).

Then, I had a look in the log files, and noticed a problem with vmware-vpxd:

VC SSL certificate does not exist, it will be generated by vpxd

This did not help either. I tried to manually start the service, with the same result.

How can I fix my vCenter?

[NutanixCanuck]

Same thing happened to me today

[NutanixCanuck]

I was able to start vmware-vxpd from the CLI but still no luck

emon:                                                                                                                                               unused

Checking for service rpcbind                                                                                                                                                       running

                                                                                                                                                                                   unknown

Checking for service sendmail:                                                                                                                                                     unused

/dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A

/dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A                                                                                                                                         running

/usr/sbin/FOO not installed

/bin/snmpd is not running

                                                                                                                                                                                   unknown

                                                                                                                                                                                   unknown

Checking for service sshd                                                                                                                                                          running

Checking for stunnel (SSL tunnel):                                                                                                                                                 unused

Checking for service syslog:                                                                                                                                                       running

Checking for service uuidd                                                                                                                                                         unused

running (standalone: 5311)

running (standalone: 5367)

Warning: vmci status: unimplemented

running (standalone: 5417)

VMware CIS License Service is running: PID:10693, Wrapper:STARTED, Java:STARTED

VMware Component Manager is running: PID:10134, Wrapper:STARTED, Java:STARTED

VMware ESX Agent Manager is not running.

VMware Inventory Service is not running.

VMware Message Bus Config Service is not running.

VMware ESXi dump collector is not running

VMware Performance Charts Service is not running.

Checking for /usr/bin/rbd-watchdog-linux:                                                                                                                                          unused

VMware HTTP Reverse Proxy is running.

VMware Service Control Agent is running: PID:10937, Wrapper:STARTED, Java:STARTED

VMware vSphere Profile-Driven Storage Service is running: PID:18707, Wrapper:STARTED, Java:STARTED

ensure environment variables are set

Checking for VMware STS IDM Server ...                                                                                                                                             running

ensure environment variables are set

Checking for service vmware-stsd                                                                                                                                                   running

syslog is running, PID: 11999

VMware Common Logging Service is running: PID:11339, Wrapper:STARTED, Java:STARTED

vmtoolsd is running

VGAuth daemon .

vAPI Endpoint is running: PID:11563, Wrapper:STARTED, Java:STARTED

VMware Content Library Service is running: PID:19194, Wrapper:STARTED, Java:STARTED

Last login: Wed May 13 12:29:51 UTC 2015 on pts/0

pg_ctl: server is running (PID: 12178)

/opt/vmware/vpostgres/9.3/bin/postgres "-D" "/storage/db/vpostgres"

VMware vCenter workflow manager is not running.

vmware-vpxd is running

VMware vService Manager is running: PID:27602, Wrapper:STARTED, Java:STARTED

/usr/java/jre-vmware/bin/vmware-vws is running.

Warning: vsock status: unimplemented

VMware vSphere Web Client is running: PID:27919, Wrapper:STARTED, Java:STARTED

Checking for service xinetd:                                                                                                                                                       unused

Checking for ypbind:                                     

[PerAtOdenSoluti]

Hi,

I have the exact same issue as you had. Vxpd is startable from cli but inventory is not...

Did you solve your issue and if so, how?

Crossing my fingers I found someone with answers...

/Per

[AlexeySav]

try to change ip in DNS and /etc/hosts on vcenter. ping vcenter FQDN from shell must work.

[hullt]

Did you happen to try option 4?  Regenerate a new VMCA Root Cert?

[rcstevensonaz]

As a reference to others, Tim's comment refers to Option 4 when you run /usr/lib/vmware-vmca/bin/certificate-manager

But, it is asking for a valid SSO password to perform certificate operations.  Can anyone explain what that is (it is not the root password).

[narenotpp]

valid SSO administrator password is the password associated with account administrator@<sso domain>.

Note: Default <sso dmain> is vsphere.local, but since you are dealing with vcsa6, at the time of deployment that name can be customized

[rcstevensonaz]

Thanks for clarifying.  In my case, the root and the [e-mail] are the same password.

Is there any issue with special characters (e.g., "(", ")", and "^")?

[darkdragon001]

I only noticed issues with passwords more than 20 characters length.

[malabelle]

Hi, I have this problem and when I try to Option 7 or 4 (revert or regenerate) it asks for the password...

But it does not recognize the password.. Is there a default one?

Any hint?

[malabelle]

2015-09-10T18:42:07.590Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2015-09-10T18:42:07.591Z warning vpxd[7F766BBA8700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

2015-09-10T18:42:07.591Z error vpxd[7F766BBA8700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:07.591Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:07.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Will attempt to connect again in 10 seconds.

2015-09-10T18:42:17.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2015-09-10T18:42:17.592Z warning vpxd[7F766BB27700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

2015-09-10T18:42:17.592Z error vpxd[7F766BB27700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...

2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore15SystemExceptionE(Host not found (authoritative)).

2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90015 ms

2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

-->

--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-2776510, tag: vpxd

--> backtrace[00] libvmacore.so[0x003C00C4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)

--> backtrace[01] libvmacore.so[0x001EDBD3]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

--> backtrace[02] libvmacore.so[0x001988DD]: Vmacore::Throwable::Throwable(std::string const&)

--> backtrace[03] vpxd[0x00BD075E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)

[malabelle]

i tried

service-control --start --all

Stdout = Starting VMware Inventory Service...

Waiting for VMware Inventory Service...................................................................................................

WARNING: VMware Inventory Service may have failed to start.

Stderr =

2015-09-10T19:33:24.061Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-invsvc, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-invsvc"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-invsvc'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-invsvc, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-invsvc"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-invsvc'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

[tomsjo]

I too have the same problem.

Been trying to figure this one out for a few days. Even if i regenerate the certs in the psc (i use external psc) it will not play ball, still hangs on starting vpxd, then reboots.

These are the last lines in my vpxd.log

2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=componentManagerUtil] AcquireToken for solution ([e-mail]): SsoException: Unexpected SOAP fault: ns0:MessageExpired; request failed.

2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=licenseServiceDiscovery] Failed to discover license service endpoint: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

2015-05-28T14:55:30.752Z warning vpxd[7F37346B97A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 6690 ms

2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Main] [VpxdMain] Failed to initialize N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...

Any ideas?

[admin]

darkdragon001‌, NutanixCanuck‌, tomsjo‌

I ran into this issue as well, but I was able to find the answer in the documentation. vCenter 6.0 doesn't support changing the 'FQDN' setting. When you first setup the VM\Appliance it suggests to use a FQDN, but a static IP address is an okay alternative. Long story short, if you use a FQDN, then you can change the IP or use DHCP, but you cannot change the FQDN. If you used an IP address, then you cannot change the IP address nor change the FQDN setting. (If you use DHCP, then you need to make certain that the assignment is static.)

This shortcoming is best documented in the 'Installing vCenter Server on a Windows Virtual Machine or Physical Server' section of the documentation.

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-A9CDCCC4-A9E9-44B0-ABED-4ECB83C432A0.html

Important

The name that you type is encoded in the SSL certificate of the system. The components communicate with each other by using this name. The system name must be either a static IP address or a fully qualified domain name (FQDN). Make sure that the system name does not change. You cannot change the system name after the installation completes.

And, there are subtle warnings in the 'Deploying the vCenter Server Appliance' section.

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-86FD9381-88B0-40F1-B516-6CC782ED9A14.html

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-E278E0F2-8983-4246-9903-50C4022F1273.html

On the Network Settings page, set up the network settings.

The IP address or the FQDN of the appliance is used as a system name. It is recommended to use an FQDN. However, if you want to use an IP address, use static IP address allocation for the appliance, because IP addresses allocated by DHCP might change.

...

If you use an IP address as a system name, you cannot change the IP address and update the DNS settings after deployment.

The issue with the script (/usr/lib/vmware-vmca/bin/certificate-manager) is not working is because the FQDN\system name has changed, so it cannot find the SSO server.

I just redeployed the VC, but I would imagine that you may be able to recover by going back to the original FQDN, or IP. Maybe reverting a snapshot would work too.

[darkdragon001]

Thanks welchg for the valuable hints to the documentation.

Maybe my problems is a combination of multiple issues: I used a hostname as FQDN, submitted my own 'external' certificates and then changed the IP address.

[gray13]

Hi,

I just met the exactly same issue...

my solution is simple...

i restart the AD server.

actually i shutdown the AD to check whether i can use the standalone VCSA6.. but the applicance is running fine but when i login to the webclient hit the same error.

So u can try to reconfigure your AD and try to re-add your VCSA6 item in the AD...

btw.. anyone tried RHEL7 LDAP to replace windows AD? can give some advice?

this AD eat a lot of memory and not that useful if i dont install mssql... but can not shut down at all...

hope it helps ..

[kegman21]

There doesn't seem to be many sources at this point with the 'failed to connect to endpoint' error and while my issue was not caused by changing an IP I received the same error when a common time could not be found for multiple components (AD, PSC, VC) within my lab. 

I recently moved the domain's PDC emulator to a new server that turned out to have  inaccurate time which eventually spread throughout the domain... once I corrected the time, rebooting the VCSA VC corrected my 'failed to connect to endpoint' error (VCSA PSC was on a separate VM which started successfully without common time). 

Definitely not suggesting it will solve everyone's problem, but wanted to share as it could be worth validating that time is common for the AD domain, PSC, and VC when getting the below error.

[mohdhanifk]

Hi,

Even we faced similar issue and had to log case with support team.

[welnet]

I ran into this issue today when changing the ip-address of the vcenter appliance. It seems the old ip address is not changed in the hosts file on the appliance. to correct this login to appliance management on port 5480 and enable ssh login & bash access. login with ssh and drop into the shell, edit the /etc/hosts file with vi and change the ip-address. After a reboot of the VM and restarting all services in the shell everything was working again.

[gjbrown]

This worked for me, thanks for the help.

[Matt Brown]

This process worked for me. Thanks for the tip about the hosts file.

[Portwajn1337]

Thanks, this solved my issue too.

Exact steps for editing hosts file are laid out in Talon256 posts (no.23 in this thread) so kudos go to you both.

[beth22]

Please check if Platform Service Controller services are started up and try to start vCenter Server services again?

And which vCenter Server deployment has been deployed? Embedded or External PSC?

[HaniIbrahim]

login to appliance management on port 5480 and enable ssh login & bash access. login with ssh and drop into the shell, edit the /etc/hosts file with vi and change the ip address

[Talon256]

I got this error after changing the vCenter's IP address

1. I used #6 https://rlevchenko.com/2016/03/24/vcenter-503-service-unavailable/ to enable the shell.
2. ssh into the vcenter server
3. Command> shell
4. vi /etc/hosts changed the IP there
5. service-control --stop --all
6. service-control --start --all

That got me fixed

[Portwajn1337]

Thanks a lot for a step-by-step guide, much appreciated! This resolved my issue, too.

Please note, that I did not restart all services, just vSphere web client service as in rlevchenko.com guide (cited), and it didn't work apparently, thus I rebooted entire appliance. The vsphere-client didn't work at first, then showed that is 'preparing itself' and only after about 5-7 mins from appliance start the website was available; so be patient.

[ManSolongines]

I had the same problem, saw the same error messages as reported by all. However, in my case, it was a time drift problem on the ESXi host that created the problem. To fix it, I had to:

1) ssh into the ESXi host

2) using the instructions at: How to Change the ESXi System Time & HW Clock on the CLI  I changed the system clock.

3) restarted the VCSA from vSphere client and all is working fine now.

So as FYI for those that may face a timing issue, you may see the same error message.