vCenter Server Service Starts Then Stops

[KGoch]

All of the sudden the vCenter Server service has stopped on the vCenter Server box. When we start the service, it immediately stops. We have tried restarting the box as well, but still get the same issue. Any ideas?

[vmroyale]

Hello.

These errors are usually database related. Is your database up and running?

Good Luck!

[JoeLyons]

Agreed, SQL is the 1st port of call

Check your ODBC connection is working.

SQL account is available and working - password is correct.

SQL service is up and running.

Joe

Remember to back EVERYTHING up before you change ANYTHING

and consider awarding points if answers where helpful to you.

[Troy_Clavell]

I agree DB problem. Possibly out of disk space. Check you event logs on your vCenter Host, it may point you to the exact problem.

[KGoch]

The ODBC connection is good. The DBA is checking the SQl cluster now.

In event viewer I see these two logs repeating every 2 minutes.

Log 1:

The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Log directory: C:\Documents and Settings\Default User\Local Settings\Application Data\VMware\vpx..

Log 2:

The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Starting VMware VirtualCenter 2.5.0 build-147633.

Any ideas?

[Troy_Clavell]

you don't happen to have IIS running on your vCenter Host do you? If so, stop it.

[KGoch]

IIS is not installed on the vCenter box.

[JoeLyons]

Since you mention a cluster can we assume you are usnig MSSQL 2005 clustered on another set of boxes?

Could the cluster have failed over and then you had the issue?

If so is there a permissions difference on the other node with the account?

Can you get the VC DB failed back to the original node?

or is there anything else in the VC box's event viewer?

Joe

Remember to back EVERYTHING up before you change ANYTHING

and consider awarding points if answers where helpful to you.

[KGoch]

Oops, sorry if I made it sound like its clustered. The vCenter box is not clustered. We are running with only one vCenter box.

(We will probably be clustering after this incident is resolved to prevent snafus like this)

[JoeLyons]

Understood, but you mentioed your DBA was checking the "SQL cluster" which is what I was refering to,

Is SQL on the local VC server aswell?

Remember to back EVERYTHING up before you change ANYTHING

and consider awarding points if answers where helpful to you.

[KGoch]

Oh, sorry. Yes SQL is clustered. There are no issues with the SQL cluster as all of our other systems relying on DBs in the cluster are running fine. So nothing has failed over there. I am still waiting to hear back if the DB itself is running/connecting. But the SQL system as a whole is running fine.

[vmroyale]

All of the sudden the vCenter Server service has stopped on the vCenter Server box.

Has anything changed on this system lately? Windows Updates?

Anything interesting in the vpxd logs?

[JoeLyons]

It's still worth checking the points in my previous post as VC is not cluster aware like other apps. so the cluster may have failed over, other apps are runing fine but VC needs a service restart to create the DB connection again.

If the server you failed over to has different security settings it would explain the error in event viewer, hence the " can you get it failed back" query to test.

Joe

Remember to back EVERYTHING up before you change ANYTHING

and consider awarding points if answers where helpful to you.

[KGoch]

In looking at the vpxd logs they all are "recent" the oldest one goes back to 11:51a today. THere are a ton of dmp files though with the first one being modified at 2:28am last night. At 2:28am last night is when the first of the two repeating vCenter errors started occuring. Oddly enough at 2:25am the box was restarted. Talking to engineers now to figure out why they restarted it.

This logged occured about 15 minutes before reboot and is the only suspcious log of the night.

The configuration information of the performance library "C:\WINDOWS\system32\perfts.dll" for the "TermService" service does not match the trusted performance library information stored in the registry. The functions in this library will not be treated as trusted.

[KGoch]

Any issues with just doing a reinstall, rather than troubleshooting reinstall.

[Troy_Clavell]

you can see this http://support.microsoft.com/kb/932813 for the error.

However, at this point if your DB is healthy, you may want to do a reinstall of vCenter. Just backup your DB before you start.

[KGoch]

Did an uninstall then a reinstall of vCenter Server and no dice. Still the same issue. Service stops as soon as starts.

[KGoch]

Also,

I saw this thread:

http://communities.vmware.com/thread/36941;jsessionid=54C6CD54F6437E64E4F0B23762607A41

Even though Im not getting the same second error message as that user, I went to check my log files on the SQL DB and they were set to restricted growth and had hit the restriction, so I changed the settings to allow growth, and the restarted the vCenter box. Still getting the same thing.

[Troy_Clavell]

you are 100% sure that the account you use for your ODBC settings has db_owner on the virtual center DB within SQL?

[KGoch]

Yes. The ODBC settings and SQL owner settings have never been changed. And just to test I did change the ODBC login from what should be there, and sure enough the service wouldnt even start. It errored out. Change the ODBC login back to what it should be results in the error I currently face. Also there is nothing of interest in the System logs, just some errors about printers.

[KGoch]

Since my DB is on a different server and the ODBC connection is setup to run as SQL Native Client, should there be a SQl Native Client service running?

[Troy_Clavell]

Since my DB is on a different server and the ODBC connection is setup to run as SQL Native Client, should there be a SQl Native Client service running?

no, the sql native client is really just a driver.

I did find this, don't know if it will work, but as long as you have good DB backup, it may be worth a shot.

http://usefulglyphs.wordpress.com/2009/11/24/vmware-virtualcenter-server-service-wont-start/

[KGoch]

That article kind of described the issue, but we actually do use sa as the designated login. I noticed this in the vpxd log just now when I tried to start the service. This is the very end of the log and attached is the whole log.

Win32 exception: Access Violation (0xc0000005)

Write (1) at address 0x000094

eip: 0x17669fb esp: 0x6e8f680 ebp: 0x6e8f680

eax: 0x6e8f694 ebx: 00000000 ecx: 00000000 edx: 0x6e8f601 edi: 0x711fb54 esi: 0x6e8f730

Exception: Win32 exception

Backtrace:

backtrace[00] eip 0x0128edf6 ?GenerateCoreDump@System@Vmacore@@YAXXZ

backtrace[01] eip 0x011d6d3a ?CreateBacktrace@SystemFactoryImpl@System@Vmacore@@UAEXAAV?$Ref@VBacktrace@System@Vmacore@@@3@@Z

backtrace[02] eip 0x011b185e ??0Throwable@Vmacore@@QAE@ABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z

backtrace[03] eip 0x011afe77 ?PanicExit@Vmacore@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z

backtrace[04] eip 0x01293487 ?GetThisThread@System@Vmacore@@YAPAVThread@12@XZ

backtrace[05] eip 0x7c359c97 __CxxLongjmpUnwind

backtrace[06] eip 0x7c359629 __CxxExceptionFilter

backtrace[07] eip 0x7c3598ad __CxxExceptionFilter

backtrace[08] eip 0x7c35994a __CxxExceptionFilter

backtrace[09] eip 0x7c359b9d __CxxFrameHandler

backtrace[10] eip 0x7c828772 RtlRaiseStatus

backtrace[11] eip 0x7c828743 RtlRaiseStatus

backtrace[12] eip 0x7c82857e KiUserExceptionDispatcher

backtrace[13] eip 0x0092e793 (no symbol)

backtrace[14] eip 0x018c9a39 ??1ByUsername@EventFilterSpec@Event@Vim@@UAE@XZ

backtrace[15] eip 0x013c3d99 ?_GetType@MoRef@Vmomi@@UBEPAVType@2@XZ

backtrace[16] eip 0x013e6be6 ?_RecordPropertyChange@PropertyProviderMixin@Vmomi@@QAE_NABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PAVAny@2@1@Z

backtrace[17] eip 0x013e7dae ?_GetChanges@PropertyProviderMixin@Vmomi@@UAE_J_JABVPropertySet@2@_NAAV?$Ref@V?$DataArray@VChange@PropertyCollector@Core@Vmomi@@@Vmomi@@@Vmacore@@AAV?$Ref@V?$DataArray@VMissingProperty@PropertyCollector@Core@Vmomi@@@Vmomi@@@5@@Z

backtrace[18] eip 0x01407ba8 ?GetPropCollectorLogger@Vmomi@@YAPAVLogger@Service@Vmacore@@XZ

backtrace[19] eip 0x0140ac37 ?GetPropCollectorLogger@Vmomi@@YAPAVLogger@Service@Vmacore@@XZ

backtrace[20] eip 0x0140b6ac ?GetPropCollectorLogger@Vmomi@@YAPAVLogger@Service@Vmacore@@XZ

backtrace[21] eip 0x0140c03c ?GetPropCollectorLogger@Vmomi@@YAPAVLogger@Service@Vmacore@@XZ

backtrace[22] eip 0x0140c69f ?GetPropCollectorLogger@Vmomi@@YAPAVLogger@Service@Vmacore@@XZ

backtrace[23] eip 0x0129699a ?IsEnlisted@System@Vmacore@@YA_NXZ

backtrace[24] eip 0x012972ec ?IsEnlisted@System@Vmacore@@YA_NXZ

backtrace[25] eip 0x01292b0e ?GetThreadId@System@Vmacore@@YA_JXZ

backtrace[26] eip 0x7c349565 _endthreadex

backtrace[27] eip 0x77e6482f GetModuleHandleA

CoreDump: Writing minidump to C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd-7716-6364.dmp

[Troy_Clavell]

I found this

2009-12-16 14:57:48.136 'App' 7604 error detected IP VS_VM:10.255.241.30 conflict between vm Hostsms (moId:vm-7581) nic:GuestInfo.net[0].ipAddress[0] and entity (moId:vm-4301)

2009-12-16 14:57:48.183 'App' 7604 error detected IP VS_VM:10.255.241.80 conflict between vm hostocs04 (moId:vm-14119) nic:GuestInfo.net[0].ipAddress[0] and entity (moId:vm-4441)

have you tried a reboot of your vCenter Host? Have you made any IP changes recently? You may also want to open an SR and get support the vpxd logs

[KGoch]

Yes its been rebooted several times. Those are VMs on the ESX hosts that are having IP conflicts.

I have submitted an SR to Support. While we have a support contract I dont like making SR's for things I can potentially fix myself. Thanks for the help we'll see how it goes and Ill update this thread when it gets resolved.

[Troy_Clavell]

wish I could be of more help, but i'm out of suggestions. Let me know what VMware Support says.

[KGoch]

After talking with VMware support they determined that something was causing a conflict with the service and therefore keeping it from starting up. We scoured all the services on the box, and started moving our search outward. We discovered that when we turned off a recently installed monitoring appliance that the service started almost instantly. So we have left the appliance off and are actively working with the vendor to see what happened.

Thanks for all your help!

[jb12345]

Can you post the monitoring software that was causing the conflict? It may be useful for others to know in the future.

[Troy_Clavell]

also, check you system logs in event view as well... They may have information.

[HughBorg707]

I've seen issues similar to this when another service is running on the box that is using the same ports as vCenter. For instance if you try to install the MS VMM management software and reboot the box, the vCenter service will never be able to start.

As soon as I removed the MS software it went right back to normal. (not slamming MS just that their software used the same ports and for some reason the MS software always won the contest...hmmm)

Regards

[VMJake]

Just to add to this....

I fixed this same problem on my Vcenter Server (which is running 4.1).  Turns out our networking engineer installed software called PRTG Network monitor for the purpose of monitoring switch and firewall traffic via SNMP.  This software has about 3 services associated with it and runs a web interface.  Shutting this software down resolved the port conflict and restored Vcenter to an operational state in my case.  From my research, this problem is caused by two possible things:

1.  Another software is using the same ports as Vcenter and therefore conflicting.  The solution is to shut the software down, or change it's ports.

2.  The other possible cause is if the SQL database is not running, or it is running but has bad information in it.  If this is the case, see this link:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008114