Setting CPU affinity to help Apache HTTP server performance

[vanv]

While researching perofrmance tuning options for Apache HTTP server workloads, one of the guys on my team came across these articles: http://testlab.inin.com/compatibilityfiles_external/documents/WebServices%20on%20VMWare%20ESX%20doc.pdf and

http://www.vmware.com/files/pdf/consolidating_webapps_vi3_wp.pdf One article recommends increasing the CPU reservations on each virtual machine up to the point which would basicallly reserve a whole CPU on the host for each virtual machine.  The other article recommends using CPU affinity to assign a CPU on the host to each virtual machine.  After reviewing these articles I am left with a few questions.  First, does anyone know if these recommendations still apply for ESXi 4.1?  Both of these articles were written back in 2008 for versions of ESX 3 and 3.5. I know there were some scheduller changes made between 3 and 4. Second, is anyone using CPU affinity in a real worlld workload that can give some feedback on their experiences? Additionally in the article where they are setting up CPU affinity they are assigning a single vCPU to each virtual machine then seeming assigning it to a single core on a processor. However over on http://www.yellow-bricks.com/2009/04/28/cpu-affinity/ Duncan makes a intresting point that it appears the other two articles fail to discuss.  He points out that VMware recommends in their Resource Management Guide, that when setting CPU affinity you should include an additional CPU for another thread to be scheduled that may need to happen on the virtual machine at the same time.  If you are using CPU affinity are you using a single vCPU or two?  Are you adding in this additional physical CPU for the extra threading headroom?

[VMmatty]

I would start with CPU reservations before you go down the road of setting CPU affinity.  Using CPU affinity limits you from using a lot of other options, namely vMotion and HA.  And then to really set this up properly you need to configure CPU affinity on all guests so that they cannot use the CPU you're reserving for your workload.

As I said I would stay away from affinity unless you've tried all other alternatives and found them to be ineffective.  Even then, consider the limitations of CPU affinity and decide if performance is worth more than things like availability.

Matt

http://www.thelowercasew.com

[vanv]

Thanks for the input Matt.  This is pretty much the approach we have on this.  We have increased the CPU reservations for now but want to explore the pros and cons of setting CPU affinity. On these particualar VM's we are not really concerned about the other funtions like VMotion or HA since we have multiple VM's across datacenters behind a load balancers all doing the same thing. We don't allow them to move hosts.  Performance is the priority.  We can loose a VM and not really notice it.  I had not considered the option that once you affinitize a CPU to a VM it might still get scheduled out to other VMs.  We do have a few other VMs that we would not affinitize so this might be a consideration.  I have also read a little bit about having to worry about dealing with NUMA memory access.  Do you have an thoughts/experience on this?

[VMmatty]

You'll get better performance if your VM uses memory that is local (same NUMA node) to the CPU that it is scheduled on.  Using CPU affinity can cause problems because the NUMA node may not have enough memory left or the affinity you set might cause remote memory access.

You can also set NUMA node affinity to prevent it from accessing remote memory, but this adds another layer of management and complexity to the design.  Also the ability to use local memory depends on how many vCPUs and RAM the VM has along with the configuration (CPU/RAM) of the ESXi host itself.  So it's hard to say how well you can make this work without more information.

Short version - again you can probably overcome this but it will add extra complexity and reduce the flexibility of the ESX host in general.  I would do some testing on this first to see how performance is affected before making these changes to all of your VMs.  And explore all other options for optimizing performance before going down the road of setting affinity.

Oh, and one more thing - welcome to the forums!

Matt

http://www.thelowercasew.com

[vanv]

Thanks again Matt.  I appreciate the welcome.  I actually have been (I have multiple accounts) on the forums for years.  I just don't make time to come here unless I have a problem or question and I almost always find the answer without having to post.  I need to make more time to give back to everyone like yourself here in the forums by participating more.  I am hoping get be able to doing testing after the first of the year.

[rickardnobel]

CPU Affinity is also not possible when running DRS Clusters (option unavailable). I agree that you should avoid setting any affinity, which basically means that you do not trust the Vmkernel vCPU Scheduler to know how to best place the vCPUs based on the available resources.

[vanv]

Rickard in the 7 or 8 years of using VMware I have never ran across a case that made sense to use CPU affinity and as a rule would not recommend it either.  However after running across documentation release by VMware itself making the recommendation to set CPU affinity for this specific type of workload has made me stop and rethink it.  For these particular virtual machines we do not care about the other functionality that VMware gives us such as DRS clusters, we care more about the horizontal scaling capabilities that being able to run multiple virtual machines on a single server gives us. Under heavy load we do seem to be hitting the software limitations mentioned in the article I linked to in my previous post and we would like to be able to squeeze some additional performance out of these virtual machines.   It is not that I don't trust the scheduler it more about making sure we are keeping our options open and not getting tunnel vision that a general best practice recommendation fits all senrioes.  I appreciate your input and would like to hear more about your experiences with using CPU affinity good or bad as I am sure there are things that we are overlooking or have not considered.

[rickardnobel]

Interesting discussion! And it is of course important to try to think outside of old habits.

vanv wrote:

However after running across documentation release by VMware itself making the recommendation to set CPU affinity for this specific type of workload

If you refer to the link in the first post (http://www.vmware.com/files/pdf/consolidating_webapps_vi3_wp.pdf) I think they do not really recommend using affinity, but more that they did it to "better understand and track the CPU usage".

However over on http://www.yellow-bricks.com/2009/04/28/cpu-affinity/ Duncan makes a intresting point that it appears the other two articles fail to discuss.  He points out that VMware recommends in their Resource Management Guide, that when setting CPU affinity you should include an additional CPU for another thread to be scheduled that may need to happen on the virtual machine at the same time.  If you are using CPU affinity are you using a single vCPU or two?  Are you adding in this additional physical CPU for the extra threading headroom?

I think this means that besides that you need a physical core per vCPU to a VM there is also some "invisible" Vmkernel support processes per VM. The Virtual Machine Monitor which is the interface between the VM and the vmkernel and some mouse/keyboard/screen handling. This could need cpu time too and might need more CPUs when using affinity.

However, in your specific case: do you have indications that your VMs is suffering from to little CPU time? Ready time and similar?