VMwareVCMSDS unable to start, unable to connect to vCenter server

[JimmyShoes]

I have a vCenter server problem.  It was working fine last week, this week I could not connect.

I think it is a problem with VMwareVCMSDS, because in the vpx-xx.log files it was complaining about starting this LDAP service.  I have rejoined the server to the domain, checked credentials and granted full to all administrators, even Everyone on the folder that contains the .DIT file.  None of these have resolved the issue.

I see also VMware Virtual Center Management Webserver service is not starting.

Some of my errors include:

Event ID 1000

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found.

Event ID 7023

The VMwareVCMSDS service terminated with the following error:

%%-1073741823

Event ID 7024

The VMware VirtualCenter Server service terminated with service-specific error The system cannot find the file specified..

SQL services seem to be running OK.

Version:  VMware VirtualCenter 5.1.0 build-880146

Any suggestions?  I'm running out of ideas.

Thank you

[Ramesh Dhanaraj]

could you please check vpxd log ? is there any latest update in IE and java?

[JimmyShoes]

I didn't update any Windows or Java updates recently.   Last I looked at the server it was working (about a week ago) and Windows Update was off.

When people say vpxd.log - I assume they mean the vpxd-###.log files I see?  Here is the contents of the latest one:

Thank you!

Section for VMware VirtualCenter, pid=3096, version=5.1.0, build=880146, option=Release

------ In-memory logs start --------

mem> 2013-12-10T03:22:19.429-08:00 [03320 info 'Default'] Process attached

mem> 2013-12-10T03:22:19.523-08:00 [03356 info 'Default'] Thread attached

mem> 2013-12-10T03:22:19.523-08:00 [03204 info 'Default'] Thread attached

mem> 2013-12-10T03:22:19.523-08:00 [03440 info 'Default'] Thread attached

mem> 2013-12-10T03:22:19.523-08:00 [03440 info 'Hooks'] Hooks Initialized

------ In-memory logs end   --------

2013-12-10T03:22:21.278-08:00 [03440 info 'Default'] Logging uses fast path: true

2013-12-10T03:22:21.340-08:00 [03440 info 'Libs'] VThreadBase detected multiple threads.

2013-12-10T03:22:21.403-08:00 [03440 info 'Default'] Handling bora/lib logs with VmaCore facilities

2013-12-10T03:22:21.403-08:00 [03440 info 'Default'] Initialized channel manager

2013-12-10T03:22:22.296-08:00 [03440 info 'Default'] Current working directory: C:\Windows\system32

2013-12-10T03:22:22.296-08:00 [03440 info 'Default'] ThreadPool windowsStackImmediateCommit = true

2013-12-10T03:22:22.296-08:00 [03440 info 'ThreadPool'] Thread enlisted

2013-12-10T03:22:22.296-08:00 [03440 info 'Default'] Log path: C:\ProgramData\VMware\VMware VirtualCenter\Logs

2013-12-10T03:22:22.296-08:00 [03440 info 'Default'] Initializing SSL

2013-12-10T03:22:23.346-08:00 [03440 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000

2013-12-10T03:22:23.346-08:00 [02604 info 'Default'] Thread attached

2013-12-10T03:22:23.346-08:00 [02604 info 'ThreadPool'] Thread enlisted

2013-12-10T03:22:23.346-08:00 [03440 info 'ThreadPool'] Thread enlisted

2013-12-10T03:22:23.424-08:00 [02844 info 'Default'] Thread attached

2013-12-10T03:22:23.424-08:00 [03440 info 'Default'] Starting VMware VirtualCenter 5.1.0 build-880146

2013-12-10T03:22:23.424-08:00 [03440 info 'Default'] Log directory: C:\ProgramData\VMware\VMware VirtualCenter\Logs.

2013-12-10T03:22:23.424-08:00 [03440 info 'vpxdvpxdMain'] Account name: SYSTEM

2013-12-10T03:22:23.424-08:00 [03440 info 'win32vpxdOsLayer_win32'] Total virtual memory available for the process 8589934464 KB

2013-12-10T03:22:23.440-08:00 [03440 info 'win32vpxdOsLayer_win32'] [VpxOsLayer] Enabled low-frag process heap.

-->

2013-12-10T03:22:23.440-08:00 [03440 info 'win32vpxdOsLayer_win32'] [VpxOsLayer] Enabled low-frag crt heap.

-->

2013-12-10T03:22:23.440-08:00 [02708 info 'Default'] Thread attached

2013-12-10T03:22:23.440-08:00 [02708 info 'ThreadPool'] Thread enlisted

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 640 max LROs

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 12 reserved internal LROs

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 12 reserved blocker LROs

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 12 reserved short LROs

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 8 reserved long LROs

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 600-second completed task lifetime

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 60-second minimum completed task lifetime

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 200 maximum completed tasks

2013-12-10T03:22:23.440-08:00 [03440 info 'commonvpxLro'] [VpxLRO] 600-second unregistered task lifetime

2013-12-10T03:22:23.440-08:00 [03136 info 'Default'] Thread attached

2013-12-10T03:22:23.440-08:00 [03136 info 'ThreadPool'] Thread enlisted

2013-12-10T03:22:23.440-08:00 [03440 info 'utilvpxdVdb'] Registry Item DB 5 value is '50'

2013-12-10T03:22:23.440-08:00 [03440 info 'utilvpxdVdb'] [VpxdVdb::SetDBType] Logging in to DSN: VMware vCenter with username

2013-12-10T03:22:23.455-08:00 [03160 info 'Default'] Thread attached

2013-12-10T03:22:23.565-08:00 [03160 info 'Default'] Thread detached

2013-12-10T03:22:23.581-08:00 [01284 info 'Default'] Thread attached

2013-12-10T03:22:23.722-08:00 [03440 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB

2013-12-10T03:22:23.847-08:00 [03440 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::GetSchemaFilename]: sqlDir: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\sql\

2013-12-10T03:22:23.910-08:00 [03440 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::CheckSchema]: schema file: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\sql\VCDB_mssql.sql

2013-12-10T03:22:23.957-08:00 [03440 info 'win32vpxLdapBackup_win32'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running

2013-12-10T03:22:23.957-08:00 [03784 info 'Default'] Thread attached

2013-12-10T03:22:24.019-08:00 [03784 info 'ThreadPool'] Thread enlisted

2013-12-10T03:22:24.019-08:00 [03440 info 'win32vpxLdapBackup_win32'] [LdapBackup] Attempting to start service ADAM_VMwareVCMSDS...

2013-12-10T03:22:24.098-08:00 [03440 info 'win32vpxLdapBackup_win32'] [LdapBackup] Service stopped, starting

2013-12-10T03:22:35.503-08:00 [02708 warning 'VpxProfiler' opID=SWI-8fee79e2] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:22:47.558-08:00 [02708 warning 'VpxProfiler' opID=SWI-3bfff25b] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:22:59.588-08:00 [02708 warning 'VpxProfiler' opID=SWI-8634cbf2] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:23:11.619-08:00 [02708 warning 'VpxProfiler' opID=SWI-26e3168d] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:23:23.649-08:00 [02708 warning 'VpxProfiler' opID=SWI-10f01905] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:23:26.696-08:00 [03356 info 'Default'] Thread detached

2013-12-10T03:23:35.726-08:00 [02708 warning 'VpxProfiler' opID=SWI-8f503d2] VpxUtil_InvokeWithOpId [TotalTime] took 12059 ms

2013-12-10T03:23:47.913-08:00 [02708 warning 'VpxProfiler' opID=SWI-94ce49b0] VpxUtil_InvokeWithOpId [TotalTime] took 12168 ms

2013-12-10T03:23:49.835-08:00 [02896 info 'Default'] Thread attached

2013-12-10T03:23:49.944-08:00 [02928 info 'Default'] Thread attached

2013-12-10T03:24:00.084-08:00 [02708 warning 'VpxProfiler' opID=SWI-95b6156a] VpxUtil_InvokeWithOpId [TotalTime] took 12152 ms

2013-12-10T03:24:12.270-08:00 [02708 warning 'VpxProfiler' opID=SWI-d16e9b4d] VpxUtil_InvokeWithOpId [TotalTime] took 12168 ms

2013-12-10T03:24:24.660-08:00 [02708 warning 'VpxProfiler' opID=SWI-2c7332ac] VpxUtil_InvokeWithOpId [TotalTime] took 12371 ms

2013-12-10T03:24:37.050-08:00 [02708 warning 'VpxProfiler' opID=SWI-c4f74deb] VpxUtil_InvokeWithOpId [TotalTime] took 12371 ms

2013-12-10T03:24:49.283-08:00 [02708 warning 'VpxProfiler' opID=SWI-982113c6] VpxUtil_InvokeWithOpId [TotalTime] took 12215 ms

2013-12-10T03:24:56.939-08:00 [02896 info 'Default'] Thread detached

2013-12-10T03:24:57.048-08:00 [02928 info 'Default'] Thread detached

2013-12-10T03:25:01.314-08:00 [02708 warning 'VpxProfiler' opID=SWI-d31b0816] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:25:13.391-08:00 [02708 warning 'VpxProfiler' opID=SWI-86df7b3c] VpxUtil_InvokeWithOpId [TotalTime] took 12059 ms

2013-12-10T03:25:20.062-08:00 [01888 info 'Default'] Thread attached

2013-12-10T03:25:20.062-08:00 [03340 info 'Default'] Thread attached

2013-12-10T03:25:25.421-08:00 [02708 warning 'VpxProfiler' opID=SWI-ec398969] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:25:37.514-08:00 [02708 warning 'VpxProfiler' opID=SWI-12dca753] VpxUtil_InvokeWithOpId [TotalTime] took 12074 ms

2013-12-10T03:25:49.544-08:00 [02708 warning 'VpxProfiler' opID=SWI-8b30e0e7] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:26:01.575-08:00 [02708 warning 'VpxProfiler' opID=SWI-729c8014] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:26:13.605-08:00 [02708 warning 'VpxProfiler' opID=SWI-1e25a95e] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:26:25.635-08:00 [02708 warning 'VpxProfiler' opID=SWI-b18df31c] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T03:26:27.166-08:00 [01888 info 'Default'] Thread detached

2013-12-10T03:26:27.166-08:00 [03204 info 'Default'] Thread detached

2013-12-10T03:26:37.665-08:00 [02708 warning 'VpxProfiler' opID=SWI-c6b6abec] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

--several similar entries as above removed for brevity--

2013-12-10T04:03:16.379-08:00 [03136 warning 'VpxProfiler' opID=SWI-a854910a] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T04:03:23.383-08:00 [03440 error 'win32vpxLdapBackup_win32'] [LdapBackup] Timed out waiting for service.

2013-12-10T04:03:24.694-08:00 [03292 info 'Default'] Thread attached

2013-12-10T04:03:26.800-08:00 [03440 error 'commonvpxLdap'] [LDAP Client] Failed to connect to LDAP: 0x81 (Cannot contact the LDAP server.)

2013-12-10T04:03:27.315-08:00 [03440 error 'utilvpxdLdap'] [VpxdLdap] Failed to create LDAP client

2013-12-10T04:03:27.471-08:00 [03440 warning 'VpxProfiler'] Vpxd::ServerApp::Init [VpxdLdap::Init()] took 2462554 ms

2013-12-10T04:03:27.985-08:00 [03440 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: VpxdLdap::Init()

--> Backtrace:

--> backtrace[00] rip 000000018018a8ca

--> backtrace[01] rip 0000000180102f28

--> backtrace[02] rip 000000018010423e

--> backtrace[03] rip 000000018008e00b

--> backtrace[04] rip 0000000000495c2c

--> backtrace[05] rip 00000000004b6512

--> backtrace[06] rip 00000001401a9e58

--> backtrace[07] rip 00000001401a859c

--> backtrace[08] rip 00000001403ca78b

--> backtrace[09] rip 000007feff60a82d

--> backtrace[10] rip 000000007727652d

--> backtrace[11] rip 00000000773ac521

-->

2013-12-10T04:03:28.391-08:00 [03136 warning 'VpxProfiler' opID=SWI-5aee2457] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T04:03:28.641-08:00 [03440 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 2464239 ms

2013-12-10T04:03:28.812-08:00 [03440 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

2013-12-10T04:03:29.530-08:00 [03440 info 'vpxdvpxdSupportManager'] Wrote uptime information

2013-12-10T04:03:40.404-08:00 [02708 warning 'VpxProfiler' opID=SWI-b1db912f] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T04:03:52.416-08:00 [02708 warning 'VpxProfiler' opID=SWI-3e2e79ad] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T04:04:04.818-08:00 [02708 warning 'VpxProfiler' opID=SWI-4c8226f] VpxUtil_InvokeWithOpId [TotalTime] took 12402 ms

2013-12-10T04:04:16.831-08:00 [02708 warning 'VpxProfiler' opID=SWI-1cadf190] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2013-12-10T04:04:29.296-08:00 [02708 warning 'VpxProfiler' opID=SWI-a371d0f6] VpxUtil_InvokeWithOpId [TotalTime] took 12464 ms

2013-12-10T04:04:29.405-08:00 [03440 info 'Default'] Forcing shutdown of VMware VirtualCenter now

[Ramesh Dhanaraj]

hi Jimmy, I hope you have rejoined the vcenter installed OS server to the same domain or different domain. log in to the SSO ( https://:9443/vsphere-client ) with Admin username password (by default  admin@system-domain ) and delete the old domain and re-add the new domain and sync. once it is join restart the server and try again. make sure your os can resolve FDQN of your domain server and the AD account is working fine for your os server level ( if u joined the os to domain or else add the dns suffix to the server.)

[JimmyShoes]

Hello - I didn't do anything like this in vCenter, no.  So I just logged in now (as domain admin) and it authenticated and logged me in OK.  I see no resources or hosts though.   Where is this SSO you speak of?  I do see Sign-On and Discover under Administration - it is blank below this title.  vCenter shows blank.

I am hoping I can recover and re-inventory everything like you said!

Thank you for your help

[Ramesh Dhanaraj]

when u login with SSO user admin(admin@system-domain) , by default you cannot access vcenter from there. this user account only detecated for SSO.

[JimmyShoes]

OK, that's the account I thought you recommended I log in with in your above post.   Anyway, given my several posts & updates, any idea what I could do next?

The services are not starting.  I'm OK with reinstalling vCenter, but not sure how to get back my inventory of VMs - what is the best course of action in your opinion?

When trying to start the virtual center service it stalls out with an event id 1000.

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Failed to intialize VMware VirtualCenter. Shutting down...

the message resource is present but the message is not found in the string/message table

Thanks

[JimmyShoes]

Trying a reinstall of vCenter, SSO and other components on the vCenter server.   It "upgrades" the install but still fails to start the services.   Any ideas on what would affect all services like this (fails to start SSO service, vCenter service, and others - just sits there forever).  All services are set to Local System.

Update:  just to clarify - the services not starting up are only two - the primary virtual center service and the virtualcenter management web services.  All others have started.

Thanks

[Ramesh Dhanaraj]

while you, trying to start the services, what error message your getting in event viewer? have u recently update IE or java?

[JimmyShoes]

Event ID posted in the first email, same one happening now.

Event ID 1000

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found.

Event ID 7023

The VMwareVCMSDS service terminated with the following error:

%%-1073741823

Event ID 7024

The VMware VirtualCenter Server service terminated with service-specific error The system cannot find the file specified..

Thanks!

[JimmyShoes]

Also at the top of vCenter I see this:

Could not connect to one or more vCenter Server systems:

https://vcenter.domain.com:443/sdk

[JimmyShoes]

I re-logged in as admin@system-domain and then saw some information under Sign-On, regarding domains.

I have an LDAP connection to AD entry in under Identity Sources.  I tested this connection successfully - should not have to delete this if it's working?

Below I see VCENTER (local domain apparently) and "System-Domain".   I do not see my domain listed here, but I see no way to add, only delete.  Should I delete VCENTER and System-Domain?   By "Actions" ( see a spinning gear - assume this is the services trying to start up, it just keeps running).   Under Actions I also see "No available actions".

What do you recommend from here?

Thank you!

Update:  I added my domain and now it shoes up below.  Rebooted the vCenter server but it's still grinding away trying to start the services, with no success.   The drop down menu where you select hte vCenter server is blank - because it hasn't started the services and recognized itself.