Load sharing among cores

[grob115]

Hi, I'm running ESXi 4 on a single chip 4-core Xeon.  On this ESXi, there are 4 VMs provisioned with the following setup:

Guest VMs	Shares	Shares Value	% Shares	vCore
VM1	Normal	1000	14	1
VM2	Normal	1000	14	1
VM3	High	4000	57	2
VM4	Normal	1000	14	1

Now the processes that are running on these 4 VMs can in reality be installed on the server itself under the same OS without ESXi.  Doing so, I know anyone of the processes running on these VMs will have access to any 1 of the 4 physical cores on the phyical CPU.  However, by installing them onto 4 VMs, it appears I'm limiting the physical resource available to only 1 of the physical cores for 3 of the VMs, and 2 of the physical cores for 1 of the VM.

So if VM2, VM3, and VM4 are extremely quiet at the moment and VM1 becomes very busy, processes on VM1 will still have access to only 1 of the 4 physical cores resource and have to wait in line to be processed since there is at most only 1 physical core?  If VM1, VM2, and VM4 are extremely quiet when VM3 becomes very busy, processes on VM3 will have access to only 2 of the 4 physical cores resource so two processes can be processed in parallel?  If settings for "Shares", "Shares Value", and "% Shares" do not affect the number of cores each of the VM have access to, how does it changes things given the setup above?

In the end, if I want VM1 - VM4 to have access to all the physical cores, I should set them each to have 4 vCores?

[jdptechnc]

Titus Cheung wrote:

Hi, I'm running ESXi 4 on a single chip 4-core Xeon.  On this ESXi, there are 4 VMs provisioned with the following setup:

Guest VMs	Shares	Shares Value	% Shares	vCore
VM1	Normal	1000	14	1
VM2	Normal	1000	14	1
VM3	High	4000	57	2
VM4	Normal	1000	14	1

Now the processes that are running on these 4 VMs can in reality be installed on the server itself under the same OS without ESXi.  Doing so, I know anyone of the processes running on these VMs will have access to any 1 of the 4 physical cores on the phyical CPU.  However, by installing them onto 4 VMs, it appears I'm limiting the physical resource available to only 1 of the physical cores for 3 of the VMs, and 2 of the physical cores for 1 of the VM.

So if VM2, VM3, and VM4 are extremely quiet at the moment and VM1 becomes very busy, processes on VM1 will still have access to only 1 of the 4 physical cores resource and have to wait in line to be processed since there is at most only 1 physical core?  If VM1, VM2, and VM4 are extremely quiet when VM3 becomes very busy, processes on VM3 will have access to only 2 of the 4 physical cores resource so two processes can be processed in parallel?  If settings for "Shares", "Shares Value", and "% Shares" do not affect the number of cores each of the VM have access to, how does it changes things given the setup above?

In the end, if I want VM1 - VM4 to have access to all the physical cores, I should set them each to have 4 vCores?

The shares only come into play when the CPU is overcommitted.  As long as there are available processor resources, every VM will have 100% of its allocation available to it at a given time... so if you have a 2000Mhz 6-core processor, VMs 1, 2, and 4 will have 2000Mhz available to each, and VM 3 will have 4000Mhz.  After the point of CPU contention, if VMs are continuing to request physical CPU, then CPU resources will be granted based on their shares.  "High" gets twice as many of the contended CPU cycles as "Normal".

When you allocate a core to a VM, it is true that it will only have access the number of cores that you allocated at any given time, but it will not remain bound to the same physical core necessarily, over time.  When a VM has something to process, the VMkernel decides which core the VM will access to execute that command.  If a VM has multiple cores, the VM will not be able to process an instruction unless it can access the number of cores that it was allocated, simultaneously.  So, unless you really NEED multiple cores, it is not recommended to allocate more than one.

[grob115]

Thanks.  I just also read the following link CPU Virtualization Basics and it says the following.  It appears if I configure all 4 VMs to have 4 cores each, then VM1, VM2, and VM4 will have access to the 4 cores 14% of the time per VM, and VM3 will have access to the 4 cores 57% of the time.

When CPU resources are overcommitted, the ESX/ESXi host time-slices the physical processors across all virtual machines so each virtual machine runs as if it has its specified number of virtual processors. When an ESX/ESXi host runs multiple virtual machines, it allocates to each virtual machine a share of the physical resources. With the default resource allocation settings, all virtual machines associated with the same host receive an equal share of CPU per virtual CPU. This means that a single-processor virtual machines is assigned only half of the resources of a dual-processor virtual machine.

However, your following statement is quite interesting.  While this wouldn't be an issue if I abandon ESXi and just install all the processes running on the 4 VMs on one single OS (ie not using VMware), it does appear to be an issue if I want to use ESXi.  Can you provide a URL where I can read more about this?  I have read through a few docs but don't recall having seen this.

If a VM has multiple cores, the VM will not be able to process an instruction unless it can access the number of cores that it was allocated, simultaneously.  So, unless you really NEED multiple cores, it is not recommended to allocate more than one.

[jdptechnc]

Titus Cheung wrote:

Thanks.  I just also read the following link CPU Virtualization Basics and it says the following.  It appears if I configure all 4 VMs to have 4 cores each, then VM1, VM2, and VM4 will have access to the 4 cores 14% of the time per VM, and VM3 will have access to the 4 cores 57% of the time.

When CPU resources are overcommitted, the ESX/ESXi host time-slices the physical processors across all virtual machines so each virtual machine runs as if it has its specified number of virtual processors. When an ESX/ESXi host runs multiple virtual machines, it allocates to each virtual machine a share of the physical resources. With the default resource allocation settings, all virtual machines associated with the same host receive an equal share of CPU per virtual CPU. This means that a single-processor virtual machines is assigned only half of the resources of a dual-processor virtual machine.

However, your following statement is quite interesting.  While this wouldn't be an issue if I abandon ESXi and just install all the processes running on the 4 VMs on one single OS (ie not using VMware), it does appear to be an issue if I want to use ESXi.  Can you provide a URL where I can read more about this?  I have read through a few docs but don't recall having seen this.

If a VM has multiple cores, the VM will not be able to process an instruction unless it can access the number of cores that it was allocated, simultaneously.  So, unless you really NEED multiple cores, it is not recommended to allocate more than one.

This PDF may help clear up what I was trying to say regarding the VM needing access to all its cores.

http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf

Another reason why not to allocate CPU cores that aren't needed - as you increase vCPU's, the amount of memory overhead ESXi uses to manage the VM increases.

http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.resmgmt.doc%2FGUID-B42C72C1-F8D5-40DC-93D1-FB31849B1114.html

One point about the shares value - the value goes up when you change the relative priority, and also changes as you add cores.  So a VM with one core and normal priority gets 1000 shares, two cores and normal priority - 2000 shares, four cores and normal priority - 4000 shares.  So in your scenario where you have 4 cores allocated to all 4 VM's, your "Normal" VM's get 4000 shares, and your "High" VM gets double that - 8000 shares.  So in that case, VM3 would have at least 40% of the CPU resources, and the other three VM's would have 20% each - if they were all requesting 100% of their allocated CPU during the same cycle.

[Linjo]

I would say its up to the application, does it do multithreading?

If not then it does not matter how many cores or VM:s you are assigning, it will only execute on one core anyway.

If it does then you can benefit from using more cores.

// Linjo