Restarting the VC database instance

[tfluegge]

Is there a proper way to restart the VC database server? The reason I ask is because our DBA restarted our VC server and it cause the VMware VirtualCenter Server service to "terminate unexpectedly" (eventid 7034).

At first I figured...easy enough, I'll just restart the service. It started as normal, with no errors, but then I started noticing some wierd VMotion errors. I couldn't VMotion VM's that were turned off or turned on. I was getting "The operation is not allowed in the current state". I then started poking around the forums and someone said to restart the "mgmt-vmware" service on the host. That gave me the ability to VMotion VM's that were turned off but not on.

I had to power off the VM and then was able to VMotion it but was not able to VMotion it back (while on) to the original host until I rebooted the host....the mgmt-vmware service restart wasn't enough.

Thank god this was a test environment. I couldn't imagine having to have to power down 10 - 15 VM's to reboot the host just to get VMotion working again.

So...Long story, short.....Is there a proper way to restart the VC database server to avoid this type of problems?

[Jaime Gonzales]

The error you were getting, "The operation is not allowed in the current state", is likely due to a lock on the VMX configuration file owned by one host.

You need to use Putty on the host where the VM is listed.

Use ps -efw and look for the parent process ID of the VMX config file.

Use kill -9 PID# of VMX config file in use.

As for restarting VC db server, i'm not sure. I've had to reboot our VC server before but never had problems w/ VMotion like that. I've had that type of vmotion problem before but only when powering up/powering down VM guests.

Good luck!

[tfluegge]

I seen this message before too. I had to change permissions on the VMX file (CHMOD 777) to make them go away but that wasn't the case this time.

As for killing the PID of the vmx file in use....Correct me if I'm wrong but killing the VMX PID will "turn off" that VM. I wanted to fix the problem without having to shutdown vm's. I didn't have many vm's running so I ended up shutting them down (gracefully) to VMotion them then rebooted the host.

Should the VC Server service terminate unexpectedly when rebooting the VC DB server? This is what I think started my problems.

Thanks for the response....

[Jaime Gonzales]

If the VC Server service stops unexpectedly, you can try this at the commandline:

service vmware-vpxa start

OR if you think it might be hung:

service vmware-vpxa restart

Yes, killing the PID will turn the VM guest off. Sometimes that is all you can do in order to get the VM guest back up and running quickly. The questions is do you shutdown and VMotion all guests on one quirky host (affecting all) or zap the PID of one and get it back up and running (affecting few)? The correct answer is "it depends". B/c only you know your environment and the SLA's for your DC.

[tfluegge]

It was my VirtualCenter Server service on my Server 2003 server that crashed. Not on my ESX host.

I am going to do some testing with this and see if I can replicate the problems I encountered. This is a test environment and I would like to figure out what happened before it happens again.

Thanks again!

[Jaime Gonzales]

Where does your VC db reside? Locally installed or on a SQL db server somewhere?

I had a problem the other night where my VC service would not stay stared on my VC server. The problem was a full VC db on our SQL farm. It could not stay started as there were stored procedures trying to run at service startup that would not get inserted.

The vmotion problem you had could very well be remedied by the command I mentioned before. If the vpx agent is hosed in any way, vmotion/managemnt of vm guests will be flaky until that host is rebooted.

Just a thought!

[tfluegge]

The VC Server and Licensing server is on it own system and the DB is on a different system (MS SQL 2k).

I'll shutdown the SQL instance and see if it crashes my VC Server service. I'll let you know the outcome. I need to talk to our DBA's too. I don't know what else they did when they downed the SQL server for maintainance.