Slow Server Boot

[DMSIT]

Needless to say we have been running ESX for about a year now and have not been overly impressed thus far with the performance. When we orignally installed ESX in our environment it was to get a hands on and expereince with a proven Virtual technology on a budget. Needless to say something is limiting us from getting the full affect of what VMWARE is supposed to be able to provide for us and hopefully you will be able to provide some feedback . Currently we are running about 10 Virtual Servers across a cluster that includes 2 ESX hosts. The VM servers are as follows....

Server 1 Exchange Email Archiving using Enterprise Vault - Windows 2003 R2 4GB of RAM allocated 4 CPU's allocated

Server 2 SMS 2003 with SQL 2005, Acronis Snap Deploy, File Server, Spector 360 (none of these applications are getting heavily utilized) - Windows Server 2003 2 CPU's 4GB of RAM

Server 3 Windows XP Workstation for a couple security applications - 1 CPU 2GB of RAM

Server 4 Enterprise Vault Discovery Accelartor (Again we are not utilizing or running any reports just sits there in case we are in need) 1 CPU 2GB of RAM

Server 5 P2V of our SQL Server for testing... Only connected to Virtual Network. only utilized at console and very lightly Server 2003 4GB of RAM 4 CPU's

Server 7 Another Development Server not being utilized is currently just running Server 2003 with 1 CPU and 2GB of RAM

Server 8 Domain Controller Server 2003 4GB of RAM and 2 CPU's

Server 9 Server 2003 R2 64-bit Running SQL (fresh install no DB's not connected to network) 8GB of RAM 2 CPU's

Server 10 Server 2003 R2 64-Bit (fresh build to see if it performs any better then Server 9 not currently running)

The 2 ESX servers are running ESX 3.5 and are fully patched running on DL580 G4 Hardware. The DL580's have 4 Dual Core 3.4ghz processors with 16GB of RAM. We are connected to an ISCSI Storage Area Network that is running 9 SATA drives. We are utilizing hardware ISCSI with dual port QLOGIC HBA QLE4062C cards.

We have a wide range of performance concerns. The first being that a few of our Virtual Servers take anywhere from 10 minutes to 20 minutes to boot. When coming up from a power off this is sometimes increased to 2-3 minutes. But on a restart we are talking 10-20 minutes. The Server sits at the Windows 2003 Splash Screen for the entire time. Once it gets past the splash screen the server Applys Settings rather rapidly and allows me to logon... Once this server is up it performs fairly well. Most of our servers take an abnormally long time to start for some reason... I would expect this if we were starting all of our servers at the same time however this problem normally occurs when starting just 1 server. When the startup process is occuring we notice that the CPU spikes to 100% and stays there until the system is booted. Memory sometimes spikes to 100% and other times is a rollercoster going up and down... We have resource pools setup for the servers that appear to take longer then others but this does not seem to help. When all servers are running our ESX servers memory usage sits at about 10-12GB per ESX host give or take a couple gig.

Now we are looking at upgrading our virtual environment to receive better performance out of our Virtual Environment however we want to make sure that we are allocating the money in all of the right places. It is being proposed to us that we upgrade our ESX servers to 48GB of RAM in each of our ESX hosts. I know this cant hurt and I thought it would allow us for some future growth. However we are being informed this additional RAM really only brings our environment up to where it needs to be to handle our existing servers. Seriously 96GB of RAM between 10 Servers? That just does not seem accurate? All the money invested and we are only getting a 5:1 compression ration and in all reality a 3:1 ratio (3 virtuals to 1 physical) if this is the case why virtualize?

So with this all said this is what I would like from any of you...

Your suggestions on improving performance if you are able to provide any with the information I provided. Should we get away from ISCSI and upgrade to Fiber channel? Do we need to upgrade our SAN disks from SATA disks to FC disks? RAM and Processor upgrade? If we do this what type of compression ratio should we expect after upgrading. Clearly cost justifying for a 5:1 ratio is very difficult!

Secondly any suggestions or tips for the slow boot times.

[khughes]

From reading your server listings and your hardware I would say you have some major CPU contension. Do you really nee 4 vCPU's in your exchange server? I would bet if you logged into the console of your ESX servers and ran esxtop your cpu ready % would be through the roof. To me it seems like the virtual systems are more like physical boxes where you throw enough resources at them they'll work better, where in a virtual environment they don't need all of the CPU's and all the RAM, sometimes its better if it is less....

My suggestion is to run esxtop on your servers, watch it for a while and see if you have multiple servers going with heavy cpu ready %'s (like 10-20+)

• Kyle

[RParker]

> Needless to say we have been running ESX for about a year now and have not been overly impressed thus far with the performance.

3 peformance points.

1) your VM's will take longer to boot. 4 CPU in a VM is taking longer, because the IO is ALL virtualized. This will be the same for ANY VM product you run (it's the nature of performance. NO ONE ever said you will get the same performance as a physical host. Your VM's are WAY over committed. Try reducing the CPU, you should see better performance. For one thing you are running dual core. You realize that a single VM in your cluster is capable of peging that ESX server, right? Well theres the first hurdle.

2) iSCSI isn't any better than NFS. Try switching to NFS mounts instead, iSCSI uses additional CPU overhead on an already overburdened cluster.

3). SATA drives. Now, SATA is good for data and storage but NOT for performance. Dont' be misled by the 3Gb/s BURST speed.. that sounds good, but SATA drives cannot meet the demand for command queuing like SCSI / SAS drives. That's your biggest problem. You really should be running these VM's on SAS drives.

So your environment doesn't follow ANY of the best practices for ESX environment, so before you go and blame VM Ware for your "slow performance" you should read up more on performance and understand the technology better. You went the cheap route, and you are getting what you paid for. No Quad Core (Dual core instead) No SAS (SATA instead) and VM's spiking your box because you are maxing out the CPU in a VM. Reduce those Core loads, and you should see increase, but until you get rid of those SATA drives, you aren't going to notice ANY significant improvement.

You can't run Exchange on a Dual Core box on SATA, that's pretty much the worst part of it, considering the other 9 servers are sharing that lack of IO. You should segrate your IO also. Exchange needs it's OWN drive or LUN, nothing should share IO with Exchange.

Did you read any performance tips for virtualizing Exchange or ESX? It doesn't appear you did, you threw caution to the wind.. and now you are paying the price. You should redesign your entire cluster.

[khughes]

What I didn't explain earlier and what rparker was beating around the bush with was the way ESX handles the CPU scheduling. In a physical world, you have a single server for a single process, example: there is an exchange server and thats it, so you would have a duel duel-core box with 4GB ram and the more RAM the better performance usually, even though it would only use about 10% of the box's processing power.

When it comes down to a virtual setup, less is usually more. In a nutshell ESX is just a buffer between your virtual machines and your hardware, so it does all the scheduling of resources. When it comes to the CPU, every virtual cpu you assign to a virtual machine = 1 core basically. So in your setup you have 4 Dual-Core CPUs which = 8 physical cores. When a virtual machine needs to access the resources, esx looks at the virtual machine's "hardware" and sees how much ram and how many cpu's it has. We'll use your exchange server for an example, it has 4 vCPU's assigned so ESX says for every process i have to run for the exchange VM, I need to have 4 physical cores available to run that process, regardless if it is a simple program like calculator or its processing a large cleanup process within exchange, whenever it needs to do something it has to have 4 physical cores free. This can cause an issue performance wise if your host gets bog'd down at the CPU level because if you have one VM that has 4 vCPU's, then another couple with 2 vCPU's and then a few others with 1 vCPU, you can see you dont' have the physical cores to cover all your virtually assigned ones.

So if you think of it like a 8 lane highway (8 physical cores) you have a VM that takes up 4 lanes, then another that takes up 2 then your other two that take up 1 lane, your last 2 vCPU box can't go anywhere because all the lanes are blocked so it has to wait for 2 lanes to become open before it can move forward. When this happens your CPU Ready% rises, which is why you should look at the ESXtop utility within vmware. I would be close to bet money that not even vmware runs their exchange with 4 vCPUs assigned to it, and if they don't run their exchange on 4 vCPU's neither should you.

If I was you I would take a serious look at your CPU usage at a per-vm-level and see if you're really using all that CPU power that you gave it? I'll tell you right now that the only boxes we truely need 2 vCPU's assigned to are our Citrix servers. All of our other servers that were on dual cpus before are down to a single 1 vCPU. Best practices when you do your conversions, not sure who did yours, was that you start low, with 1 vCPU and then add if you find you need another cpu added. As for the SATA vs SAS yes you'll get better performance for sure out of SAS but should first worry about getting your virtual CPUs under control. Then I would look at the amount of RAM you have assigned to each VM vs. How much you are actually using. If you have been up for over 6 months that should give you a good baseline to look at past stats.

Its probably the biggest lession and hardest thing to grasp when jumping from physical to virtual, less is more and throwing resources at a virtual machine will almost always just make it run slower.

• Kyle

[DMSIT]

So at what point do you guys recommend allocating another CPU to the virtual machine? I have a VM that has 2 CPU's allocated to it now and we keep receiving alarms on the CPU.

[khughes]

You have continuous alarms on a VM that is running 2 vCPUs? What type of server is this? What is it running. Overall if a server is that power intensive sometimes its not made for a virtual environment.

• Kyle

[DMSIT]

Apparently our environment was not scoped for true performance. Yes this is correct I have a Virtual machine that keeps throwing alarms for CPU. The server is running SMS 2003 with SQL. However we are not utilizing SMS to any of its potential.

[DMSIT]

Please let me apologize that I came off as blaming VMWARE. That is the last thing that I want I truly feel VMWARE is a great product. I just wanted a better understanding of how to gain the best performance out of our Virtual Environment and make sure that we spend money in the right places to get the best performance. Clearly when this was scoped it was not scoped accurately and we do not want this to happen again.

Lastly we are not running Exchange virtually. We are running Email Archiving for Exchange. Enterprise Vault Requires its own server. Also we are not running all 10 Servers of the 10 listed we are running about 5 of them.

Thanks for the information!