vCenter

 View Only
Expand all | Collapse all

vCenter 6 fails to restart after IP change

NutanixCanuck

NutanixCanuckMay 13, 2015 06:42 PM

  • 1.  vCenter 6 fails to restart after IP change

    Posted May 12, 2015 04:27 PM

    I changed the vCenter 6 IP from within web client.

    Since I had problems connecting to the new IP, I restarted the server via vSphere Client directly on the host using VMware Tools.

    After the restart, I can't connect to vCenter any more. Trying to connect via https shows me the following error:

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f4158068bf0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

    Then I directly used the console, enabled ssh and bash and logged in there.

    I tried to check the current IP settings via the script used in version 5.5 which still exists (/opt/vmware/share/vami/vami_config_net) but fails with multiple errors.

    Then, I tried to reset the certificates via certificate manager (/usr/lib/vmware-vmca/bin/certificate-manager), but I could not login via SSO (seems that SSO service could also not be started).

    Then, I had a look in the log files, and noticed a problem with vmware-vpxd:

    VC SSL certificate does not exist, it will be generated by vpxd

    This did not help either. I tried to manually start the service, with the same result.

    How can I fix my vCenter?



  • 2.  RE: vCenter 6 fails to restart after IP change

    Posted May 13, 2015 06:42 PM

    Same thing happened to me today



  • 3.  RE: vCenter 6 fails to restart after IP change

    Posted May 13, 2015 08:39 PM

    I was able to start vmware-vxpd from the CLI but still no luck

    emon:                                                                                                                                               unused

    Checking for service rpcbind                                                                                                                                                       running

                                                                                                                                                                                       unknown

    Checking for service sendmail:                                                                                                                                                     unused

    /dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A

    /dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A                                                                                                                                         running

    /usr/sbin/FOO not installed

    /bin/snmpd is not running

                                                                                                                                                                                       unknown

                                                                                                                                                                                       unknown

    Checking for service sshd                                                                                                                                                          running

    Checking for stunnel (SSL tunnel):                                                                                                                                                 unused

    Checking for service syslog:                                                                                                                                                       running

    Checking for service uuidd                                                                                                                                                         unused

    running (standalone: 5311)

    running (standalone: 5367)

    Warning: vmci status: unimplemented

    running (standalone: 5417)

    VMware CIS License Service is running: PID:10693, Wrapper:STARTED, Java:STARTED

    VMware Component Manager is running: PID:10134, Wrapper:STARTED, Java:STARTED

    VMware ESX Agent Manager is not running.

    VMware Inventory Service is not running.

    VMware Message Bus Config Service is not running.

    VMware ESXi dump collector is not running

    VMware Performance Charts Service is not running.

    Checking for /usr/bin/rbd-watchdog-linux:                                                                                                                                          unused

    VMware HTTP Reverse Proxy is running.

    VMware Service Control Agent is running: PID:10937, Wrapper:STARTED, Java:STARTED

    VMware vSphere Profile-Driven Storage Service is running: PID:18707, Wrapper:STARTED, Java:STARTED

    ensure environment variables are set

    Checking for VMware STS IDM Server ...                                                                                                                                             running

    ensure environment variables are set

    Checking for service vmware-stsd                                                                                                                                                   running

    syslog is running, PID: 11999

    VMware Common Logging Service is running: PID:11339, Wrapper:STARTED, Java:STARTED

    vmtoolsd is running

    VGAuth daemon .

    vAPI Endpoint is running: PID:11563, Wrapper:STARTED, Java:STARTED

    VMware Content Library Service is running: PID:19194, Wrapper:STARTED, Java:STARTED

    Last login: Wed May 13 12:29:51 UTC 2015 on pts/0

    pg_ctl: server is running (PID: 12178)

    /opt/vmware/vpostgres/9.3/bin/postgres "-D" "/storage/db/vpostgres"

    VMware vCenter workflow manager is not running.

    vmware-vpxd is running

    VMware vService Manager is running: PID:27602, Wrapper:STARTED, Java:STARTED

    /usr/java/jre-vmware/bin/vmware-vws is running.

    Warning: vsock status: unimplemented

    VMware vSphere Web Client is running: PID:27919, Wrapper:STARTED, Java:STARTED

    Checking for service xinetd:                                                                                                                                                       unused

    Checking for ypbind:                                     



  • 4.  RE: vCenter 6 fails to restart after IP change

    Posted Oct 06, 2015 04:19 PM

    Hi,

    I have the exact same issue as you had. Vxpd is startable from cli but inventory is not...

    Did you solve your issue and if so, how?

    Crossing my fingers I found someone with answers...

    /Per



  • 5.  RE: vCenter 6 fails to restart after IP change

    Posted Dec 08, 2015 01:43 PM

    try to change ip in DNS and /etc/hosts on vcenter. ping vcenter FQDN from shell must work.



  • 6.  RE: vCenter 6 fails to restart after IP change

    Posted May 21, 2015 11:27 AM

    Did you happen to try option 4?  Regenerate a new VMCA Root Cert?



  • 7.  RE: vCenter 6 fails to restart after IP change

    Posted May 24, 2015 01:32 AM

    As a reference to others, Tim's comment refers to Option 4 when you run /usr/lib/vmware-vmca/bin/certificate-manager

    But, it is asking for a valid SSO password to perform certificate operations.  Can anyone explain what that is (it is not the root password).



  • 8.  RE: vCenter 6 fails to restart after IP change

    Posted May 24, 2015 01:39 AM

    valid SSO administrator password is the password associated with account administrator@<sso domain>.

    Note: Default <sso dmain> is vsphere.local, but since you are dealing with vcsa6, at the time of deployment that name can be customized



  • 9.  RE: vCenter 6 fails to restart after IP change

    Posted May 24, 2015 01:50 AM

    Thanks for clarifying.  In my case, the root and the administrator@vsphere.local are the same password.

    Is there any issue with special characters (e.g., "(", ")", and "^")?



  • 10.  RE: vCenter 6 fails to restart after IP change

    Posted May 26, 2015 08:21 AM

    I only noticed issues with passwords more than 20 characters length.



  • 11.  RE: vCenter 6 fails to restart after IP change

    Posted Sep 10, 2015 07:08 PM

    Hi, I have this problem and when I try to Option 7 or 4 (revert or regenerate) it asks for the password...

    But it does not recognize the password.. Is there a default one?

    Any hint?



  • 12.  RE: vCenter 6 fails to restart after IP change

    Posted Sep 10, 2015 07:13 PM

    2015-09-10T18:42:07.590Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

    2015-09-10T18:42:07.591Z warning vpxd[7F766BBA8700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

    2015-09-10T18:42:07.591Z error vpxd[7F766BBA8700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

    2015-09-10T18:42:07.591Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

    2015-09-10T18:42:07.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Will attempt to connect again in 10 seconds.

    2015-09-10T18:42:17.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

    2015-09-10T18:42:17.592Z warning vpxd[7F766BB27700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

    2015-09-10T18:42:17.592Z error vpxd[7F766BB27700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

    2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

    2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...

    2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore15SystemExceptionE(Host not found (authoritative)).

    2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90015 ms

    2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

    --> Backtrace:

    -->

    --> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-2776510, tag: vpxd

    --> backtrace[00] libvmacore.so[0x003C00C4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)

    --> backtrace[01] libvmacore.so[0x001EDBD3]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

    --> backtrace[02] libvmacore.so[0x001988DD]: Vmacore::Throwable::Throwable(std::string const&)

    --> backtrace[03] vpxd[0x00BD075E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)



  • 13.  RE: vCenter 6 fails to restart after IP change

    Posted Sep 10, 2015 07:34 PM

    i tried

    service-control --start --all

    Stdout = Starting VMware Inventory Service...

    Waiting for VMware Inventory Service...................................................................................................

    WARNING: VMware Inventory Service may have failed to start.

    Stderr =

    2015-09-10T19:33:24.061Z   {

        "resolution": null,

        "detail": [

            {

                "args": [

                    "Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "

                ],

                "id": "install.ciscommon.command.errinvoke",

                "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",

                "translatable": "An error occurred while invoking external command : '%(0)s'"

            }

        ],

        "componentKey": null,

        "problemId": null

    }

    ERROR:root:Unable to start service vmware-invsvc, Exception: {

        "resolution": null,

        "detail": [

            {

                "args": [

                    "vmware-invsvc"

                ],

                "id": "install.ciscommon.service.failstart",

                "localized": "An error occurred while starting service 'vmware-invsvc'",

                "translatable": "An error occurred while starting service '%(0)s'"

            }

        ],

        "componentKey": null,

        "problemId": null

    }

    Unable to start service vmware-invsvc, Exception: {

        "resolution": null,

        "detail": [

            {

                "args": [

                    "vmware-invsvc"

                ],

                "id": "install.ciscommon.service.failstart",

                "localized": "An error occurred while starting service 'vmware-invsvc'",

                "translatable": "An error occurred while starting service '%(0)s'"

            }

        ],

        "componentKey": null,

        "problemId": null

    }



  • 14.  RE: vCenter 6 fails to restart after IP change

    Posted May 28, 2015 01:08 PM

    I too have the same problem.

    Been trying to figure this one out for a few days. Even if i regenerate the certs in the psc (i use external psc) it will not play ball, still hangs on starting vpxd, then reboots.

    These are the last lines in my vpxd.log

    2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=componentManagerUtil] AcquireToken for solution (vpxd-e0ba3305-3310-4e62-9e36-0339ae5c0291@vsphere.local): SsoException: Unexpected SOAP fault: ns0:MessageExpired; request failed.

    2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=licenseServiceDiscovery] Failed to discover license service endpoint: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

    2015-05-28T14:55:30.752Z warning vpxd[7F37346B97A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 6690 ms

    2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Main] [VpxdMain] Failed to initialize N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

    2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...

    Any ideas?



  • 15.  RE: vCenter 6 fails to restart after IP change

    Posted Jun 26, 2015 04:24 PM

    darkdragon001‌, NutanixCanuck‌, tomsjo

    I ran into this issue as well, but I was able to find the answer in the documentation. vCenter 6.0 doesn't support changing the 'FQDN' setting. When you first setup the VM\Appliance it suggests to use a FQDN, but a static IP address is an okay alternative. Long story short, if you use a FQDN, then you can change the IP or use DHCP, but you cannot change the FQDN. If you used an IP address, then you cannot change the IP address nor change the FQDN setting. (If you use DHCP, then you need to make certain that the assignment is static.)

    This shortcoming is best documented in the 'Installing vCenter Server on a Windows Virtual Machine or Physical Server' section of the documentation.

    http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-A9CDCCC4-A9E9-44B0-ABED-4ECB83C432A0.html

    Important

    The name that you type is encoded in the SSL certificate of the system. The components communicate with each other by using this name. The system name must be either a static IP address or a fully qualified domain name (FQDN). Make sure that the system name does not change. You cannot change the system name after the installation completes.

    And, there are subtle warnings in the 'Deploying the vCenter Server Appliance' section.

    http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-86FD9381-88B0-40F1-B516-6CC782ED9A14.html

    http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-E278E0F2-8983-4246-9903-50C4022F1273.html

    On the Network Settings page, set up the network settings.

    The IP address or the FQDN of the appliance is used as a system name. It is recommended to use an FQDN. However, if you want to use an IP address, use static IP address allocation for the appliance, because IP addresses allocated by DHCP might change.

    ...

    If you use an IP address as a system name, you cannot change the IP address and update the DNS settings after deployment.

    The issue with the script (/usr/lib/vmware-vmca/bin/certificate-manager) is not working is because the FQDN\system name has changed, so it cannot find the SSO server.

    I just redeployed the VC, but I would imagine that you may be able to recover by going back to the original FQDN, or IP. Maybe reverting a snapshot would work too.



  • 16.  RE: vCenter 6 fails to restart after IP change

    Posted Jun 29, 2015 10:55 AM

    Thanks welchg for the valuable hints to the documentation.

    Maybe my problems is a combination of multiple issues: I used a hostname as FQDN, submitted my own 'external' certificates and then changed the IP address.



  • 17.  RE: vCenter 6 fails to restart after IP change

    Posted Jul 14, 2015 05:13 PM

    Hi,

    I just met the exactly same issue...

    my solution is simple...

    i restart the AD server.

    actually i shutdown the AD to check whether i can use the standalone VCSA6.. but the applicance is running fine but when i login to the webclient hit the same error.

    So u can try to reconfigure your AD and try to re-add your VCSA6 item in the AD...

    btw.. anyone tried RHEL7 LDAP to replace windows AD? can give some advice?

    this AD eat a lot of memory and not that useful if i dont install mssql... but can not shut down at all...

    hope it helps ..



  • 18.  RE: vCenter 6 fails to restart after IP change

    Posted Sep 20, 2015 10:23 PM

    There doesn't seem to be many sources at this point with the 'failed to connect to endpoint' error and while my issue was not caused by changing an IP I received the same error when a common time could not be found for multiple components (AD, PSC, VC) within my lab. 

    I recently moved the domain's PDC emulator to a new server that turned out to have  inaccurate time which eventually spread throughout the domain... once I corrected the time, rebooting the VCSA VC corrected my 'failed to connect to endpoint' error (VCSA PSC was on a separate VM which started successfully without common time). 

    Definitely not suggesting it will solve everyone's problem, but wanted to share as it could be worth validating that time is common for the AD domain, PSC, and VC when getting the below error.



  • 19.  RE: vCenter 6 fails to restart after IP change

    Posted Oct 07, 2015 03:04 PM

    Hi,

    Even we faced similar issue and had to log case with support team.



  • 20.  RE: vCenter 6 fails to restart after IP change

    Posted Oct 30, 2015 01:53 PM

    I ran into this issue today when changing the ip-address of the vcenter appliance. It seems the old ip address is not changed in the hosts file on the appliance. to correct this login to appliance management on port 5480 and enable ssh login & bash access. login with ssh and drop into the shell, edit the /etc/hosts file with vi and change the ip-address. After a reboot of the VM and restarting all services in the shell everything was working again.



  • 21.  RE: vCenter 6 fails to restart after IP change

    Posted Dec 15, 2015 02:23 PM

    This worked for me, thanks for the help.



  • 22.  RE: vCenter 6 fails to restart after IP change

    Posted Jun 24, 2016 09:39 PM

    This process worked for me. Thanks for the tip about the hosts file.



  • 23.  RE: vCenter 6 fails to restart after IP change

    Posted Nov 09, 2016 09:56 PM

    Thanks, this solved my issue too.

    Exact steps for editing hosts file are laid out in Talon256 posts (no.23 in this thread) so kudos go to you both.



  • 24.  RE: vCenter 6 fails to restart after IP change

    Posted Nov 14, 2015 05:30 AM

    Please check if Platform Service Controller services are started up and try to start vCenter Server services again?

    And which vCenter Server deployment has been deployed? Embedded or External PSC?



  • 25.  RE: vCenter 6 fails to restart after IP change

    Posted Jan 29, 2016 02:48 PM

    login to appliance management on port 5480 and enable ssh login & bash access. login with ssh and drop into the shell, edit the /etc/hosts file with vi and change the ip address



  • 26.  RE: vCenter 6 fails to restart after IP change
    Best Answer

    Posted Jun 06, 2016 02:28 PM

    I got this error after changing the vCenter's IP address

    1. I used #6 https://rlevchenko.com/2016/03/24/vcenter-503-service-unavailable/ to enable the shell.
    2. ssh into the vcenter server
    3. Command> shell
    4. vi /etc/hosts changed the IP there
    5. service-control --stop --all
    6. service-control --start --all

    That got me fixed



  • 27.  RE: vCenter 6 fails to restart after IP change

    Posted Nov 09, 2016 09:59 PM

    Thanks a lot for a step-by-step guide, much appreciated! This resolved my issue, too.

    Please note, that I did not restart all services, just vSphere web client service as in rlevchenko.com guide (cited), and it didn't work apparently, thus I rebooted entire appliance. The vsphere-client didn't work at first, then showed that is 'preparing itself' and only after about 5-7 mins from appliance start the website was available; so be patient.



  • 28.  RE: vCenter 6 fails to restart after IP change

    Posted Jan 13, 2017 05:38 PM

    I had the same problem, saw the same error messages as reported by all. However, in my case, it was a time drift problem on the ESXi host that created the problem. To fix it, I had to:

    1) ssh into the ESXi host

    2) using the instructions at: How to Change the ESXi System Time & HW Clock on the CLI  I changed the system clock.

    3) restarted the VCSA from vSphere client and all is working fine now.

    So as FYI for those that may face a timing issue, you may see the same error message.



  • 29.  RE: vCenter 6 fails to restart after IP change

    Posted Jun 21, 2016 05:31 PM

    ** make sure the vCenter hostname only has what ever was originally entered during installation **

    Late, but hopefully useful for others.  I ran into this today from similar activities (changed IP, name, etc.)  After realizing there's no change flexibility, I reinstalled with certain, permanent values to prevent this from happening again.  This appliance runs alone in a v6 host (HP blade.)  All worked well, even rebooted a couple of times to ensure it'd remain functioning.  Then I made an atypical network change, after-witch my vCenter broke.  The change was at the HP C7000 chassis @ layer 2.

    Initially networks were provided to the blades via shared up-link sets (SUS) containing ~ 3 vlans over the first 2 LOM's (on-board NIC's) --> vSwitch0 with vlan support.

    Change..., configured same 2 LOM's to "tunnel" the vlans to the host --> vSwitch0.  Both configurations use virtual switch tagging (VST) and should have no effect.  After everything came up and I realized the vCenter was borked, simple troubleshooting proved the network config was solid... I could ping interfaces on all 3 vlans (one which was on the vCenter appliance.)  Pulled hair out, scoured I-net, and searched logs.  Happened to notice the appliance's hostname was populated with its FQDN, generally no big deal - but I'm certain I entered it with only the hostname (just built the dang thing for the ##?? time) So;

    I manually edited the hostname from the console back to just the hostname and rebooted... everything came back functioning as designed.  I'm going to keep this thing in dev a bit longer... not sure it's ready for more than playing with at this point, to fragile with little information available for troubleshooting. 



  • 30.  RE: vCenter 6 fails to restart after IP change

    Posted Jun 24, 2016 11:58 AM

    dear experts;

    i changed nothing...not an IP etc....but it still shows this error....gave a static IP because need ESXi with vCenter for a home lab setup.....

    Pls can someone from VMWare post whether this is a known bug or something.....have even heard that it effects ESXi 6U2 also....

    Regards



  • 31.  RE: vCenter 6 fails to restart after IP change