IIS performance 50% slower on a VM?

[bsti]

I've got a cluster of IIS Servers that we host our web application on. Recently, we converted several of our web servers from physical machines to virtual machines. By converted, I mean we built a template from scratch and cloned it to create the rest of our Web VMs.

The server work, but there is a definite and noticable difference between our VMs and our physical machines. In examining the IIS logs, we found that the time taken to process the request for a certain page we're measuring normally takes about 600 ms to process on a physical server.

On the new VMs, it's closer to 900 ms, which is about 50% slower.

The physical machines are Windows server 2003 R2 Std x86 servers with 4 GB of memory and 2 quad-core 3Ghz processors. Our VMs are quad-processor VMs with 8 GB of RAM running Windows Server 2003 x86 Std R2.

Our first thought was the difference in the processor count, with 8 being in the physical machines vs 4 in the VMs (IIS is a multi-thread-friendly application). To address this, we built a new Windows 2003 Enterprise R2 Server with 8 vCPUs and put in in the mix. This new server runs with lower CPU utilization, but the page render time is still the same.

Our cluster is new and is not being heavily utilized. CPU contention is not an issue. I've even tested by moving one of the web server VMs to it's own host for a day and saw no change. Our web server admin insists none of these measures have changed anything.

Are there any other settings or options I should look into tweaking to improve this deficiency? My guess is he's seeing expected overhead from CPU virtualization, but everything I'm reading says we should not see this big of a performance hit.

Thanks!

[bsti]

I misspoke above. Our VMs only have 4 GB of RAM each.

[golddiggie]

What are you using for storage for the VM's to reside upon? RAID controller, drive type, speed, count, etc.? How are the virtual drives provisioned? Size of the C drive? Is C thick or thin provisioned?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[bsti]

It's a NetApp SAN. NetApp uses RAID DP, and we have all 300 GB 15K SCSI RPM disks. There are 14 disks in the RAID group.

I have two data stores for OSs, one for vswap, and one for "other" data. The only ones of concern to the web cluster is one of the OS volumes and the vSwap store. I use thin provisioning on all of the stores (except the vswap store).

Disk activity on the Web servers on the whole is VERY low, so I've no looked in that direction as to it being the cause.

[golddiggie]

I'm going through the process now, and have noticed some improvment on some of my VM's (not had the time to test deeper, so it's more initial reaction), of inflating the C drives to thick provisioned. I'm leaving D (or other disks) as thin provisioned for now.

Try testing that out on one of the VM's to see if it helps improve your performance issues. I do know that officially, there's no set rule for when to use thick or thin, it's more a "try it and decide" type deal. I would pull the VM's down to 4 vCPU's max again (if you haven't already).

How do you have the SAN connecting to the hosts? How many pNIC's are in the vSwitch for the VM traffic?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[bsti]

Thanks for the response.

We use FCP and have 2 FC connections from each host to the SAN. I use ALUA and Round Robin path selection policies for fault tolerance and proper load balancing. I've verified it is using all available FCP paths.

We do not use iSCSI or NFS.

Our focus right now is on CPU and Network speeds. We noticed CPU utilization and processor queue lengths were higher on the 4-cpu VMs, so we tried the 8-cpu box. The 8-cpu box has better (lower) utilization and queue length, but still processes the page more slowly.

I'll try dropping back down to 4 cpus and thickening the disk just to try it.

You appear to think it's disk. Not that I know at this point, but what is steering you in that direction?

[golddiggie]

Most is a process of elimination. If you've already confirmed several items are good, or not the issue, then the list keeps shrinking.

What are the actual host servers you're running? Are these brand new hosts, or were they used for other things previously? Are you running ESX 4 u1 on them? Have you checked for post update 1 updates for the hosts? When you installed ESX 4 on the hosts, did you accept the defaults for paritioning and such or modify the settings?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[bsti]

I appreciate the feedback, thanks.

I think we've pretty much ruled disk out as the cause. Disk % time is less than 1%, Disk queue never reaches 1, and bytes/s is less than 10 kb for the most part. It was definitiely worth a look though.

The host servers are brand new HP Proliant DL380 G6 servers. All identical. All newly loaded with VmWare ESX4 U1 Build 208167.

I modified the default partitions per the following guidelines:

http://www.yellow-bricks.com/2009/05/27/partitioning-your-esx-host-part-ii/

/ 5120 MB

Swap 1600 MB

/var 4096 MB

/home 2048 MB

/opt 2048 MB

/tmp 2048 MB

[golddiggie]

What processors are you running in those hosts? Is ESX installed on disk or to flash media? If disk, how is that configured? Is the performance the same on all the hosts, or just some/one of them? Did you check to make sure the VT setting in the bios was enabled/correct? How are the vSwiches configured for the VM's? What does the physical network configuration look like?

It could be a performance delta in the thick/thin provisioning of the VM's drives. Once you've done the C drive, see how it responds. How much free space do the C drives have? Do the VM's have secondary drives where the software/web sites are installed?

Whenever I build a Windows VM (2003 or 2008, all releases) I use one drive (C) for just the OS. Any/all software is installed onto a secondary drive (usually D, renaming the cd drive to something else). I keep the C drive as clean, and lean, as possible.

How are the memory and CPU resources set for the VM's? I typically use 0 reservation and unlimited there. If the processors support hyperthreading, and you have it enabled, what is the Advanced CPU setting at (under the Resources tab, in the VM Properties)?

If everything looks correct, I'd reach out to VMware and/or HP to find out if there's a setting that needs to be changed to make these systems perform like they did when physical...

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[bsti]

Processors are Intel Xeon X5560 2.8 Ghz processors.

ESX is installed on Local disk (RAID 1, 2 disks).

Performance is the same regardless of host.

We enable the No-Execute Memory protection and INtel(R) Virtualization Technology options on all VMWare host servers, though these options are turned ON by default with the new G6 servers.

Good recommendation. Our Web admin guys leaves the OS isolated on the C, and puts the IIS web app on the E: drive. On a physical server, these two partitions were on separate physical disks. In the VMWare environment, it really doesn't matter (It's all on a 14-disk RAID DP array).

I set no special reservation and do not limit resources on any VM at this time.

We do have hyperthreading enabled. The CPU/MMU Virtualization setting is set to "Automatic" (the default).

I opened a case with VMWare, but thought I'd post here too since the community seems rather knowledgable.

Thanks again for the feedback.

[golddiggie]

I would always follow what's proven to be 'best practices' when it comes to Windows Servers. It doesn't matter if they are physical or virtual since Windows doesn't notice any difference. I prefer to isolate the OS and then have the secondary disk for applications (and files) since far too often the amount of space is misgauged and you now need XXGB more on the drive. With that being a single, C, drive, it's a pain to expand (up until some of the new releases of Windows). It's far easier to expand the non-OS/boot volume after adding space to it.

As recommended above, disable HT on one host, but I would even split the drives up like was done before. On win2k3, a 20GB C drive is usually good, with 2008, a 40GB (the default) C drive is a good baseline. Make the D/E drive large enough to hold the software and any other files (plus room to grow a bit).

Just for giggles, try making a VM with thick provisioned drives for both volumes.

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[golddiggie]

What were the servers you were using before? Especially, which processors, speed, and count (socket and core)? How about memory configurations on both old and new (just for grins)?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[bsti]

The physical servers were HP Proliant DL 360. 2 quad-core 3ghz processors and 4 GB RAM.

Two tests I'll try Monday:

- Turning off Hyperthreading

- Thick provisioning the disk.

Thanks again for all the suggestions.

[golddiggie]

Already seeing improved performance on the VM's that I've converted/inflated the C drive to thick provisioning. Also noticed a difference in Linux VM's when comparing thin to thick provisioned disks. Since I don't have enough room in my datastore to thick provision all my vdisks, I'm going to just do the C drives (for Windows) and only in Linux systems with small enough vdisks (<40GB). I'll test again when I move to a NAS to see if it makes sense to keep this method, or to change it. For now, thick boot drive, thin secondary drive, seems to offer the best of both worlds.

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[bsti]

An Update:

I ran the same set of tests with the disk thick provisioned. This made no difference in my environment. This doesn't surprise me too much because our web servers attempt almost no disk activity.

However, disabling hyperthreading actually helped. It shaved about 100 ms off the time to render the page. This gain becomes smaller as the VM becomes more loaded, but it did make a measurable difference. The VMs still perform noticably slower than their physcial counterparts, but this was helpful.

[RParker]

I think we've pretty much ruled disk out as the cause.

Your environment is almost exactly identical to ours. WE are running Netapp 3070. Everything else about you is exactly the same as ours, I see the same thing. I hope you can resolve it.

Disk seems to drag, even though the performance counters don't show this.

[joshp]

If hyperthreading is enabled on the ESX host; try disabling it. Hyperthreading on an ESX host can have a negative impact on VM performance. After you disable hyperthreading on the host try testing your IIS VM again (you might even want to test the VM with a single vCPU).

VCP 3, 4

www.vstable.com

[RParker]

An earlier post by this individual stipulated that he will disable hyperthreading....

[bsti]

Right... I'm the indivdual.

I'm just confirming that I did indeed run the test with Hyperthreading off and it did make a difference, though I would not say it resolved the issue entirely.

I'm waiting for a callback from Tech support, but if there are any further suggestions to try let me know. Thanks!

[TimOudin]

If you are still running with 4 vCPU then I'd highly suggest you take the suggestion joshp offered and reduce that vCPU count to either 2 or 1 (with HAL modification of course!). I've seen a case like yours and have resolved it by reduction of vCPU count...nobody believed it would work. Read this for a thorough explanation of CPU scheduling in vSphere.

Traditionally disabling hyperthreading wasencouraged though this has changed with vSphere, I believe the reference article also covers the logic built in to discourage coscheduling on the same core with hyperthreading enabled. I still do not enable it though I also do not overcommit my CPU very heavily and run largely uniprocessor for efficiency.

The physical and virtual disk suggestions are all great, just make sure you're not creating your own problem with CPU READY time. Using vscsiStats, if classic ESX, will help you determine if you have a storage issue or not. http://communities.vmware.com/docs/DOC-10095|d-DOC-10095

Good luck!

Tim Oudin

[a_p_]

A couple of days ago I came across this VMware KB article Poor virtual machine application performance on HP hardware

It's worth a try.

André

[golddiggie]

Things like that make me like HP servers less and less all the time... HP has great switches, but servers... not so much...

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[Josh26]

Just to take a different tact..

Is it a networking issue?

I would try installing the latest VMWare Tools, and adding a NIC using the vmxnet3 driver (if not already done) to replace the default NIC.

[bsti]

First off, thanks for all of the helpful suggestions.

In our vCPU count study, so far we're finding the following:

- More vCPUs reduces CPU utilization and processor queue length on the VM.

- In this regard, more vCPUs seem better

- However, the time to process the page does not seem any better with more vCPUs. So I'm unconvinced more vCPUs = better performance.

- Hyperthreading off seems to have shaved some time off the page render time. We will know more conclusively soon since we turned it off across the cluster and will study all of the web server VMs for a similar performance benefit. This will verify the 100ms or so we saved was not a fluke of some sort.

My next tests will follow your suggestions and try 2 vCPUs with hyperthreading on and off.

One vCPU is hard sell (IIS is known to perform better on multithread-friendly systems). Plus, we'd have to rebuild the VM. I will try it it nothing else shows promise.

The link about poor HP Hardware performance was very interesting. I'm not going to take the bait on the "HP Hardware sucks" debate. To each his own, but I will definitely check into the BIOS settings mentioned on the article and see if that makes any difference.

I'll also try the VMNet NIC suggestion. We are seeing somewhat high output queue lengths on the NIC on the VM.

[TimOudin]

Warning: Moderately Off Topic...

Please do report your findings, this information can be valuable to other people later on!

I know the uniprocessor is a tough one to sell but we find interesting things doing it on occasion and you do not need to rebuild the servers from a Windows point of view, the HAL can be modified quite easily. If you have the time it's an interesting test to run. There are factors in virtualization that drive greater efficiency with reduction in multi-processor virtual machines. The articles on ESX co-scheduling is enlightening though the latest one is a tough read. I have personally applied this concept of scale out growth successfully (on HP hardware!), most recently in a JBoss web application environment. Insure that when you test you test all virtual machines on an ESX server. Also, as you appear to be doing, measure application/infrastructure response times and not reported resource utilization within the virtual machine as it's not necessarily representative of reality.

If you haven't already seen the Performance Best Practices on VMware vSphere 4.0 whitepaper, it's worth a read and speaks on (and against) virtual SMP quite a bit. http://www.vmware.com/resources/techresources/10041

Josh26 brings up a good point. Also, insure you read up on performance of the pvSCSI adapter before you consider trying it.

Cheers

Tim Oudin

[RParker]

One vCPU is hard sell (IIS is known to perform better on multithread-friendly systems).

Ah Ha! Now you almost got the picture. Virtual CPU are not REAL CPU, therefore the capabilities are SOFTWARE driven NOT hardware. The Hypervisor(s) will get there one day, but right now the layer that tasks the CPU are managed by SOFTWARE.

So you just said the magic word, multithread. Start a VM, look at the capabilities of the processor using like an Intel CPUID, even though the underpinings SAY it's an Intel XXX it's not TRULY an Intel XXX, it's synthetic. It's software, so the capabilities are MUCH different than REAL stand alone hardware that you would expect. So yes IIS WILL work better on multi-thread PHYSICAL cores, but the vCPU don't have ALL the same capabilities..

That's what we try to tell people. You are seeing the same thing, adding vCPU lowers the usage, but does NOT improve performance. So once again you only found the conclusion that we have trying to explain for years, that adding MORE vCPU does NOT equal better performance. It takes people to actually figure it out for themselves, so hours of testing have proved what we said from the beginning (which would have saved you ALL the trouble) 1 vCPU will perform equally as well, you just needed to see it for yourself. You said 2 vCPU, but that 1 vCPU will probably perform just as well, and for those of us that have ALREADY figured this out, we have tried to explain this, but alas you need to see with your own eyes, so hopefully NOW you can use this information and know the difference.

[bsti]

I'm sure your thought on that is correct in many cases. I'm not going to argue it as I'm not exactly a VMWare expert.

What I can contribute however is what my tests have yielded. In our specific case (IIS running our custom web application), more vCPUs help in general because it lowers CPU utilization and queueing. This hasnt' fixed our specific issue, but it does conttribute to the overall CPU health of the VM and make it generally more usable. Again, this is so in this specific case.

In many (most?) other cases, your supposition is likely more correct. I think what you are saying makes perfect sense.

Basically, I just want to point out that this is obviously not a "one size fits all" scenario. We're not done testing yet. I still intend to test with 2 and 1 vCPU and note the results. I will share what I find.

My web guy intends to comment with some more details on IIS and how it works with multiple threads and such. He's more an expert than I and you may find what he shares to be useful.

[PConlan]

Well, it isn't quite that simple. Classic ASP on IIS by default will create 25 ASP worker threads per processor. Our physical servers have 8 processors, so for one ASP application, it will create a max of 200 threads it can use to process requests. IIS and ASP don't know the difference between physical and virtual processors. It simply looks at how many processors the OS has, and creates threads based on that number.

If we drop that down to 1 vCPU on the virtual machine, that application will only have 25 ASP worker threads. This will result in ASP requests getting queued while they wait for a thread to get freed up.

Now, that number of 25 can be increased, but the recommendation is to not go over 100. If we go down to one vCPU, which we will test, we will have only have half the number of ASP worker threads available as when we had 8 processors.

Simply going to one vCPU is not going to perform as well as multiple processors without tuning the application settings, and even then you would have to go beyond Microsoft recommendations to allow the same maximum number of requests to be processed concurrently.

Also we see much higher guest processor queue lengths with only 4 vCPU's as opposed to 8 vCPU's. Maybe it doesn't mean the same thing on a virtual machine as it does on a physical one, but the OS doesn't tune itself because it knows it is on a virtual machine. How it deals with processor queues is going to be the same. As far as the OS is concerned, it thinks it needs to wait on the processor. Again, maybe there is some tuning that could be done in the OS to allow it to have 1 vCPU and not have a high processor queue length.

[TimOudin]

I believe that RParker and I are both looking at this from an efficiency standpoint and being able to provide scheduling on the physical CPU to the guest VM in a more timely fashion. I understand IIS, or used to, and do agree that performance might be better from an application delivery point with 2 vCPU. The high level communication being made is that the VMware administrator should be highly in tune with the co-scheduling capabilities of your ESX servers and the demands of the virtual machines. Short story, more SMP virtual machines yields less resources for all virtual machines on a host and the issue compounds at an exponential rate as the ratio of SMP virtual machines to physical cores increases. If 8 vCPU virtual machines are required then those HP DL380 will have to be ditched in favor of something with far more sockets. It's all about providing the hypervisor more opportunity to co-schedule the SMP virtual machine.

On the queue depth statement, are you referring specifically to physical servers? You are correct that the performance metrics on a virtual machine don't really tell the full story but in this case I do agree that a queue depth on the guest is what it is, or worse. The virtualization layer causes the guests performance metrics to not really coincide with reality, for example low CPU utilization and low I/O activity on the guest can potentially being a real READY time issue on the host or guest queue depth being compounded by the vCPU queue depth on the hypervisor.

This thread is getting lengthy with theory, enough for now. Let's see those test results. Before modifying those virtual machines esxtop output would be awesome to see.

Tim Oudin

[bsti]

I'm familiar enough with esxtop to run it and see the various stats, but how exactly do I record the output you want to see. Do I just take a screenshot of it while it's running, or is there a command to record the results to a file?

[TimOudin]

The ability to run esxtop in 'batch' mode and redirect to a .csv file exists though a few screens shots of would give some insight. Using the VM view ('v' option IIRC) would help.

http://pubs.vmware.com/vi301/resmgmt/wwhelp/wwhimpl/common/html/wwhelp.htm?context=resmgmt&file=vc_esxtop.14.10.html

Cheers

Tim Oudin

[bsti]

Another update:

We set the HP Power Regulator setting on our ESX host servers that a.p. linked to and immediately noticed a 200-300ms improvement in the page rendering times.

http://h18000.www1.hp.com/products/servers/management/ilo/power-regulator.html

Apparently, this setting is set by default to save power when the CPU is not being heavily-used. We set the setting according to the VMWare document (which essentially causes the CPU to run at full performance at all times). We've been watching the page render times across all of our web servers for almost 24 hours now and they all are showing the improvment. There's still a 100ms or so difference in time in favor or our physical machines, but this is much more acceptable that the 50% performance difference before. This setting makes a HUGE difference!

We still intend on running the 2vcpu vs 1 vcpu vs 4 vcpu tests and posting the results, but I wanted to make it known the above setting rectified our performance issues for the most part.

I appreciate everyone's feedback on this issue as it was very helpful in helping us resolve the issue.

[etieseler]

Curious,

Did you ever run the test on 2 / 1 vCPU?

If so, what results have you found? I've always sided with the arguments of RParker and TimOudin, where you are better served running 1vCPU VMs, if you need more out of it, scale it out with another VM. Hardware is cheap when its virtualized, so adding another IIS server isn't the cost it used to be when its physical, but if you load your hosts up with 2 and 4 vCPU VM's then you are ultimately reducing the total number of VM's you can run, ultimately reducing efficiency.

I agree its a hard concept to sell but you have to kind of think of things differently when you virtualize. Its no longer tied to the issues of the physical world.

I'd be interested to see how it compared to a single 2vCPU server vs 2 1vCPU servers load balanced.

-Ed