VMware vSphere hang after power up few guest systems

[adyk723]

Upgraded the test VMware environment this weekend from 3.5 U3 to vSphere 4. The upgrade completed without any problem. But the vSphere server always hangs when powering up the 3rd guest servers. (All guest system are Windows 2003 32 bit.) We first thought one of the guest system might be the problem because there is not problem with only 2 guest systems powered on. But there is no problem when we started that VM first. The host server only hangs when we started the third guest server.

We first believed the upgrade went wrong. So we backed up the guest systems and start the vSphere install from scratch. After the install and restore the servers, same problem. When we run with the evaluation license, host server hang during the first system boot up. There is no BSOD, the server remain in the F1 and F11 screen. No key stoke allow on the server.

Has anyone seen this problem before?

Here is my configuration;

Dell PowerEdge SC1430 server

Dell CERC 1.5/6CH SATA RAID card

(3) 500 GB SATA hard drives.

(2) Intel 1000 pro NIC

(1) BroadCom NetXtreme NIC

Upgraded and installed from the ISO file downloaded this weekend. Used both vSphere license and evaluation mode

If you need some information/log from the server, please just ask.

[DSTAVERT]

Is it on the HCL? There are some things dropped from the HCL with 4. Download the CPU compatibility ISO from http://www.vmware.com/download/shared_utilities.html

[adyk723]

The system has the Intel Xeon E5310. CPU has the EM64T and Intel® Virtualization Technology. VT is enabled in the BIOS. That should be all is needed in the processor. I also see the processor in the HCL list in the VMware site. I am hope I don't need a new server to run vSphere...

[DSTAVERT]

Since it isn't on the HCL run the Dell diagnostics and check using the compatibility CD. Eliminate those things as problems.

[RParker]

I'd say it was a hardware issue, specifically bad memory. If your VM's are the same, and you boot 1, boot 2, then it hangs on 3.. that indicates memory parity errors or something is trying to allocate memory at the spot to bring up that 3rd VM.

Did you let it sit there for a LONG time skip it and try to boot other VM's? Or did you stop there and try again?

I would download the diagnostics CD from Dell and run a FULL memory test on that box, it seems like a hardware problem. Maybe even swap the memory sticks just in case.

[adyk723]

I have had these memeory sticks since the system got here. They have no problem running ESX 3.5 U3 with 6 guest running at the same time. But on the safe side, I will run the memory diagnostic from the boot once I get into office tomorrow. Too bad that I don't have another set of memory to test.

For the test I have done. I started two VMs and ran overnight, the server and the VMs ran without any problem. Once I started the thrid one, the host hangs.

[Jinto Antony]

Please check the following things

1,Did you reboot the machine after the upgrade

2,Please check the DELL DRAC Logs for any kind of hardware errors

3,Please run memory diag given by DELL

4, What is the Host machine memory? ESX4 need almost 5GB for system+console

[adyk723]

Thanks for the reply. Here are the responses for your question.

1. The ESX 4 was re-installed. The server was rebooted after the clean install.

2. The server does not have a Dell Remote Access Card. The server was purchased for Dev/Test environment. The RAC was not included.

3. Ran the memory test from the Dell Open Manage disk and everything turns out clean.

4. The server has 5 GB of memory. Do you think more memory might fix the problem?

More diagnostic update:

After the server hang, I can still ping the server. When I try to use Alt-F1 to enter the console, the Username prompt still comes up and I can enter the username. But I can not enter password. This is different behavior then before. The only difference is I checked the console after it hangs overnight this time.

[LucasAlbers]

could you have a bad hard disk, and therefore a corrupt install?

Boot the system off a linux burn in cd, and verify the cpu/memory/motherboard works, by running a burnin test for a day.

I run 6 vm's on esxi 4 with 4GB and it appears to run happily, my point being that esxi needs less memory than esx, which can be critical in a low memory system.

[adyk723]

We have installed ESX3.5 U4 back into the system. (Co-workers aren't that happy with their test/dev system down for too long.) The ESX 3.5 U4 is running with 5 guests without any problem. Now I am guessing one of the hardware in the server is not compatible with ESX4. And I am betting on the Dell CERC SATA controller card. Besides the Dell SC1430 server, which I don't think the motherboard can cause this, the SATA controller card is the only thing that is NOT in the HCL. Need to find another controller card and give that a try.

I don't believe the server has a bad hardware. Especially ESX3.5 U4 runs without any problem...

Andy

[admin]

Although it's hard to say definitively without specific error codes (such as a "PSOD" in vmkernel,

or a virtual machine monitor panic message), this has at least some of the hallmarks of a specific bug

(PR 460968) that is fixed with the just-released (or very soon, if not quite yet) ESX 4.0 patch 02.

It is consistent with working OK on ESX 3.5 but not ESX 4, and the bug definitely only hits when using

32-bit guests. (Of course this isn't to say it always hits with that combination, or it wouldn't have been a

subtle bug only recently nailed down.)

-Ari

[adyk723]

Thanks for the information. I will keep that in mind.

The problem might be from the RAID card. I have update the server to the DELL PERC 5i SAS RAID card and everything is fine after. But I am planning to build a second vSphere system for test environment. I will test the old card with the patch once the parts for the test server has arrived.

Thanks,

Andy

[Danno9]

This is an old thread, but I found it on search for issues with ESXi 4.0u1

and the CERC 1.5/6ch RAID controller. Figured I'd post

an answer in case others are having the same issues and using the same

search. I had a lot of SCSI errors in the logs pointing to an issue

with the aacraid driver. This is commonly accepted as a non-working

controller for ESXi 4.0 and I just left it out of the server and used the

onboard SATA controller.

With the introduction of ESXi 4.1, it WORKS!!! On a whim, I

re-installed the CERC 1.5/6ch card (Adaptec 2610SA) and loaded a clean

install of ESXi 4.1 on the SATA drive connected to the onboard SATA

controller. Rebooted and it recognized the CERC 1.5/6ch controller and

the RAID-5 array I had made with 5 drives using the BIOS utility. I was

able to format a VMFS datastore and used the entire array for iSCSI

storage with OpenFiler (this was where it always hung and crashed on the ESXi 4.0). Some benchmark testing shows the CERC

controller being slightly slower than the single SATA drive connected

to the onboard controller. However, after turning on write-caching, it

was about 10-15% faster.

Not super performance by any means, but I'm glad I was able to finally

use this card for creating a 2-TB volume for backing my VMs..

[igorle1]

My case is similar:

• Hypervisor: ESXi 4.1 (run on USB flash)
• Controller: Dell CERC 1.5 6-CH /w 01 WD 320GB SATA

I could not run more than 3VMs. After power on 4th VM, host has become unserponsive. From Performance tab, I have learn that if disk latency is more than 25ms in 3 minute, host status became time-out.

Solution is not to put more than 3VMs on one disk. I think so