CPU reservations irrelevant - core scheduling more important

[VMwareBL]

I am having a bit of a problem wraping my head around the whole resource reservations when it comes to CPUs and resource pools and what VMware is doing here basing it on Ghz. Memory is straight forward because you have certian amount and can allocated out of that until its used up but CPUs is hard to grasp since only one process can allocate at the time even if only 100Mhz the rest is not used.

The CPU has two areas of concer as i see it "the clock speed known as Ghz" and "core schdule".  Lets say you have 4 core CPU with 4 Ghz for simplicity.  And i create two resource pools one with 3Ghz and other 1Ghz. How if any way does this affect scheduling of the cores? Thos wont make sure that 3 cores are always ready for one pool and one core for the other as i want.  What's the point of the Ghz  reservation unless you have massively high CPU consuming apps running?  When the core is scheduled to execute and is busy executing even 100Mhz the rest of the core capability is wasted and can't be allocated to something else as two processes can't execute at same time! So to me ghz is irrelevant when thinking from that perspective. The part that really counts is core scheduling.  This i found true even in my environment where i has 10% utalization yet system slooooooooow because of core overcommitment. I had 24vCPUs and assigned 38 to VMs. So even though the Ghz was at close to zero the VMs were sky high in "ready' time becasue there was no free core to execute in! Once that had core busy were using only little mghz but might as well have been using everything as its irrelevant. You see my point?  I have over 50 clients and each one of them has 10% CPU utalization yet close to perofrmance limit becasue of core allocation not Ghz!  If i add more VMs ready time sky rockets and performance drops like hell.

Scenario: I want to split the server half for lab and half for production and i set 3Ghz for Production and 1Ghz for Lab using same CPU specs (4Ghz 4 core). Well my lab can still be grinding my production server to a halt if its scheudled accross all cores but not high clock!  Only way around that would be shares i think!  but even then there is no way to guarantee cores for certain pool right?  Like you can't create a Lab environment and make sure it never takes up more then certain amount of cores?

[weinstein5]

Welco me to the Community - the reservation is measure in GHz so that it is easier to understand and allow users to visualize on how much of the pool cycles is being reserved -

The first point is a reservation insures there are going to be many cycles available for the vm/pool - so for your example if you a VM in the 3 GHz pool using 100 MHz pool - the rremainiing 3.9 GHz is available for any vm - including those in the 1 GHz pool - now as the VM starts using more cycles it will be taken away from other VMs until the VM in the 1 GHz pool - and if both VMs are using their reserved and require more cycle the CPU ready time will increase

Shares will definitely a better way to because with reservations you can run in to situations where you will not be able to power a vm up -

[VMwareBL]

Thanks! Take it you noticed its the 1st post :smileyhappy:...

See that's the problem though! Your comment: "so for your example if you a VM in the 3 GHz pool using 100 MHz pool - the rremainiing 3.9 GHz is available for any vm" The remnaining 3.9Ghz is NOT available to any other VM. Not if your 100Mhz is being executed by 4 threads on 4 cores! Then you may have 3.9Ghz of free Ghz but nothing can get to it!

So if you have a VM in lab which has 4 cores and running multi threaded application and using only 400Ghz lest say, you think you have 3Ghz for your other VM except you don't becasue all of the 4 cores are busy executing the 400Ghz by the lab VM.  That's what i mean. The Ghz is nowhere near important as the free core and make sure there is a free one in my opinion and that you can't do unless you ensure that you dont over provision your cores.

Also on another point, why would you even want to limit Ghz! This measure is one which in more basic terms is the amount of instructions your computer can carry out.  Thus why limit it, wouldn't it make more sense to leave it so that whatever thread is exeuting uses what it needs to finish faster and free up the core. Afterall as i said you can have two different threads using 3 Ghz core at same time 1.5Ghz each. Schedulign doesn't work like that nor does the CPU in general.

[chriswahl]

Hertz are a measurement of time. The CPU reservation/limit relationship is how much time the vmkernel scheduler allows the VM to run on a given core with respect to the clock rate of that core. This can be seen with ESXTOP when you look at RDY values for a limited VM.

[VMwareBL]

I don't think so. I disagree with that as CPU Hrz is not a measure of time but cycles and is relevant to other varable. If you trully believe so can you find me an article proving so and i mean that with utmost respect couse i would like to know for sure but I think your wrong in this case. I have read ton of articles to support the argument. Here first one that came up on google.

http://wiki.answers.com/Q/Is_speed_of_CPU_is_measured_in_hertz

Maybe VMware made resource pools and reservations treat it as such but that's not what it is!  And real life scenarios prove otherswise as well.  I have 4Ghz CPU with 4 cores. I have a 2 VMs each with 4 cores (total 8) and one is in pool for 3Ghz other in 1Ghz. The VMs will have to wait on each other i dont care how you set the Ghz reservations.   The whole point behind this is that you can't use this reservation to guarantee perfromance to something. Maybe in ratio to another VM yet but not to guarantee it will not have performance hit!

Like the shares make perfect sense to me as its priority to get scheduled to the CPU but this reservations make none. i'm either missing something or there really is no point.  Its not like memory or storage you have x capacity and you allocate until full. core is capable of 1Ghz lets say and only 1 process can access it wether its using 1mhz or 1ghz so if its scheduled its scehduled. if i reserve 1ghz but grant 4 cores and have application using all 4 cores at low Mhz the other pools is blocked as cores are busy regardless that Ghz are free.

[chriswahl]

Frank has written quite extensively on this subject.

http://frankdenneman.nl/2010/06/reservations-and-cpu-scheduling/

Additionally, there is a rather complete white paper on the subject from vSphere 4 entitled "CPU Scheduler in VMware ESX 4.1"

http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf

Cheers.

[VMwareBL]

see that's my problem, his article doesn't help but adds to the confusion for me...

quote: "Limit: A limit is a mechanism to restrict physical  resource usage of the virtual machine. A limit ensures that the VM will  never receive more CPU cycles than specified, even if extra cycles are  available on the host."

Why would you limit a VM to 500Mhz  on a 1Ghz core when only one thread can access at a time anyway's?

quote: "Now reservations act differently when setting it on a CPU than setting  it on memory. When the virtual machine does not use its CPU cycles,  these CPU cycles are redistributed to other active virtual machines"

if this statement it true then the first quote makes sense as difference is redistributed but again, how can it be when the core is busy though? i reserve 500Mhz on a 1Ghz core and i have 2 cores executing 500mhz each .. that can't be. Maybe one after another but one is waiting while other competes if its executing at same time.

also another thing that comes to mind. if you have 4 cores 1ghz each and you allocate a VM with a single vCPU logic says the most it can ever use is 1Ghz since that underlying CPU hardware capacity and you gave it only 1 CPU? Would that be accurate to say or is it false assumption in VMware and this VM can have 4Ghz though that single vCPU by switching cores?

I hear all the arguments but none of them address my question directly hence why i keep going back to the same question. Simple scenario:

you have a pool with 4 Ghz and 4 core CPU supporting it. You create 2 VMs and one you allocate 3Ghz reservation but you give it 2 cores (i don't even know if that would make sense) then on second VM you give it 4 cores and only 1Ghz reservation. When the second VM is executing its 4 threads (assuming its multi threaded app) the first VM that has 3Ghz reservation has to wait as core is busy regardless that this one is limited to 1Ghz. so its not a "real" reservation but rather higher priority if anything to me.

GUYS THANKS SO MUCH for discussing this with me as i can't seem to find the answer anywhere and i know your just trying to answer even though i might come off "argumentative" i assure you its not i just want to get to the bottom of this so that i can understand it.

[Dwayne Sinclair]

Why would you limit a VM to 500Mhz  on a 1Ghz core when only one thread can access at a time anyway's?

It's an example and the 1Ghz single core is for illistrative purposes only. VMware Hypervisor is virtualizing the physical 1 Core/ 1Ghz CPU and providing a virtual 1Ghz vCPU to multiple VM's. Multiple VM's would each have a 1Ghz vCPU and their respective OS's would be capable of running up to 1Ghz but the Hypervisor limit will restrict CPU utilization to 500Mhz.

Multiple VM's each with a 500Mhz CPU limit would happily share the single physical 1Ghz core assuming all VM's are not running at 100% all at the same time but no one VM could run more than 500Mhz. More than two VM's running simutaniously at 500Mhz would exceed the physical capability of the process and the Hypervisor would now share the physical 1Ghz CPU so 4 VM's running at 100% would each get 250Mhz each.

The reality is -

1. when CPU limits are placed on VM's, the physical CPU's are faster (eg 3Ghz) (Multicore (4/8/12/etc), and support Hyperthreadiing thus the limits are relevent to the VM and not the physical machine. Larger ESXi hypervisor servers can be built to ~3Ghz/40 Cores/512GB RAM+.

2. Irrespective of a VM's memory allocation, limit, and reservation, all VM's dont run at 100% simutaniously.

quote: "Now reservations act differently when setting it on a CPU than setting  it on memory. When the virtual machine does not use its CPU cycles,  these CPU cycles are redistributed to other active virtual machines"

if this statement it true then the first quote makes sense as difference is redistributed but again, how can it be when the core is busy though? i reserve 500Mhz on a 1Ghz core and i have 2 cores executing 500mhz each .. that can't be. Maybe one after another but one is waiting while other competes if its executing at same time.

Reservations are not limits. Reservations as they pertain to the above example ensures the Hypervisor has 500Mhz of CPU processing power for the VM whether it uses it or not.

In larger VM environments (100's-10,000 VM's) and because of the capabilities of multisocket, multicore servers, physical RAM becomes the linmiting factor well before CPU.

From...

http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-resource-management-guide.pdf

vCenter Server or ESXi allows you to power on a virtual machine only if there are enough unreserved resources to satisfy the reservation of the virtual machine. The server guarantees that amount even when the physical server is heavily loaded. The reservation is expressed in concrete units (megahertz or megabytes).

For example, assume you have 2GHz available and specify a reservation of 1GHz for VM1 and 1GHz for VM2. Now each virtual machine is guaranteed to get 1GHz if it needs it. However, if VM1 is using only 500MHz, VM2 can use 1.5GHz.

Reservation defaults to 0. You can specify a reservation if you need to guarantee that the minimum required amounts of CPU or memory are always available for the virtual machine.

[VMwareBL]

First let me thank you for your response.  I understand all that in the context of text but not real life. I just don't see how "CPU reservation" is  areservation and guarantee at all. Maybe if you answer this for me it will help.

if you have 4Ghz CPU with 4 cores lets say. Then i create 4 VMs each with 1Ghz reservation. Thus i am saying i guarantee this VM will have 1Ghz. Then i assign each VM with 4 cores so i exssentialy overallocated my cores and have 12 cores assigned when i have only 4 (assume no hyper threading). At this point how can i guarantee that one of these VMs will have CPU to run on and 1 Ghz if my other VM is running a hyper threaded app executing on all 4 cores but at very low hrz?  It can't. The VM which has 1Ghz reservation will have it CPU ready time shoot up indicating that its waiting and thus performance is impacted thus to me your not reserving CPU and guaranteeing it will  be there.  If the core is busy its irrelevant if its executing at 2hz or 1 ghz as one thread has access and that's it.

Thanks so much!

Added: Also if i have overallocated cores but i want to ensure that one VM is not impacted, i really can't with CPU reservations only wtih shares i can ensuring it will be shceduled with higher priority then other VMs. At least that's what makes sense to me or seems to be like as when i set reservation on a VM in overallocated environment i still have CPOU ready go up and impact on that vm.

[Dwayne Sinclair]

I just don't see how "CPU reservation" is a reservation and guarantee at all. Maybe if you answer this for me it will help.

Depending on if you are viewing your environment from vSphere vCenter or stand alone directly connected to the Hypervisor, Servers and Clusters have a tab called Resource Allocation which details the available resources for Memory and CPU, total capacity, reserved capacity. available capacity.

Its a reservation because new VM's will not start if the environment does not have enough resources to meet the reservation. If 10Ghz CPU is available and 7.5Ghz is reserved, you cannot start a new VM if its reservation is greater than 2.5Ghz irrespective of how much the three existing and one new VM's actually use.

Reservations/limits are normally applied on specific VM's or on Pools so that reservations are spread across all VM's within a pool. Ideally, the CPU and memory capabilities of your virtualized environment will meet your requirements. Limits and Reservations will assist to ensure the outlyers are controlled and guarenteed.

I would recommend readling the link provided as well as "testing" various scenarios using a VM and resource pools on ESXi - ESXi can be virtualized under VMware Workstation or Fusion for personal testing/education.

[VMwareBL]

I totaly understand that! You can't reserver 20Ghz when you have 10Ghz and expect to power up those extra VMs past 10Ghz. However, what does that have to do with ensuring that a VM will have CPU resources to execute on when it needs to?

I understand it have implimation on being able to power up vms - but does the reservation guarantee that VM will have CPU to execute on when it needs to?  For me the answer is no as it all depends if the cores are busy executing threads form another VM which is what i tried to explain in the question above with high overallocation of cores.  If i reserve 1Ghz and i have to execute i am not guaranteed i will be able to right away unde the above scenario right? 

[Dwayne Sinclair]

Does the reservation guarantee that VM will have CPU to execute on when it needs to?  For me the answer is no as it all depends if the cores are busy executing threads form another VM which is what i tried to explain in the question above with high overallocation of cores.  If i reserve 1Ghz and i have to execute i am not guaranteed i will be able to right away unde the above scenario right?

Virtualized environemts will be memory constrained well before CPU constrained. I'm not sure where you are at with your virtualization journey but I want to share with you I have performed many VMware Capacity Planning assessments and what I commonly see is average physical CPU utilization of around 1%-2% and max 5%-10% across physical environments.

The answer to your questions is yes - reservations are honored and cpu will be available to meet the reservation. When the reservation is honored and the VM starts, the reservation will always be available to the VM. Other VM's without reservations will be dynamically constrained (limited) based on the reservation and remaining/available resources.

Statistics on CPU scheduling are maintained by vSphere and certainly in a very small/ very constained environment there could be the possiblity of scheduling delays but real world environemnts are not likely to be constrained in this manner.

As a test, use the Nostalgia VM available in the VA Marketplace as this in an old DOS app which uses all availale CPU you can dyanmically apply a limit to constrain the VM down to whatever CPU limit you like while the machine is running - This is what the Hypervisor is doing realtime in constrained environments.

[VMwareBL]

Actually i am a Lead Consultant with all the certifications in VMware and years of experiance. Problem here is that i am asking one question but other is being answered though so my question might not be clear.

Quote ' I have performed many VMware Capacity Planning assessments and what I commonly see is average physical CPU utilization of around 1%-2% and max 5%-10% across physical environments.'

That's true if your looking at the Ghz performance metrics. Which is EXACTLLY why i have this problem.  However, look at your CPU ready counter! you can have 5% CPU utalization but have MAJOR performance impact if your cores are overallocated!   This is a fact!

quote 'The answer to your questions is yes - reservations are honored and cpu will be available to meet the reservation'

My question was not if reservations are honored, my question was "if a vm wants to execute will there be CPU ready for it" - answer is no if you have overallocation in cores regardless of CPU reservations.

[VMwareBL]

Maybe this will help as its reall world scenario  which in part raised this thread:

A month ago i went to see a new client who reports having major perofmrance issues.  I looked at his VC and noticed he is running lab and production in same cluster (which we can all agree on is not smart).  Anyways his comment was "but i created two resource groups and i have production 80% reservation while i gave lab only 20%" .  Meanwhile he had over 50 VMs in lab and only 20 production! He had allocated over 100 cores on a system that has only fraction of that. When i told him its the CPU, he said (which by the way is what you were saying earlier)  but look at the performance tab for CPU, its not even 10%!  Meanwhile his CPU ready counter was showing 1000's in ESXTOP!

So while the ghz might be 5% it means nothing to your performance if your cores are overallocated and reservation will not guarantee you perfromance.

[Dwayne Sinclair]

Thanks so much for the clarification. Certainly specific workloads are optimized for more/smaller cores vs less/greater cpu so 10Ghz could consist of 4 Cores/2.5Ghz vs 10 Cores/1 Ghz. Reservations and Limits don't address core/workload optimization - this needs to occur at design and implementation. Today, CPU Affinity combined HT Sharing can help to assist but depending on the workload, a cluster that is architecturely optimized to meet the specific workload would help.

Given the example you provided, appropriate CPU resources need to be allocated to respective environents. It's also important to be sure VM's are correctly sized (minimize vCPUs where possible).

I do not know if ESX(i) is running on a NUMA server but I have attached the folowing kb which is a good match for the scenario you detailed - first validate your customer is on current versions of the hypervisor...

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1026063&sliceId=2&docTypeID=DT_KB_1_1&dialogID=380638546&stateId=1%200%20380656302

http://blogs.vmware.com/vsphere/2012/02/vspherenuma-loadbalancing.html

As the VMware environment scales, core optimized worlodads mixed with general purpose workloads is fine given the size provides spread. I would encourage you to get a vSphere Hypervisor futures update from your VMware account team.

Thanks.

[yaza]

i guess i got your point.

"so for your example if you a VM in the 3 GHz pool using 100 MHz pool - the rremainiing 3.9 GHz is available for any vm"

if you have a physical cpu of 3.9GHZ it means 3 900 000 0000 cycles per second so if you have a VM with 100 MHZ reservation which means 100 000 000 cyles per second. so if you do mathematical calculation 100 000 000 cycles can be accomplished by physical cpu in 0.02 seconds. in other words the physical cpu will be given to this vm 20ms in each second for this reason you have cpu ready time.

i am not an expert but i have had the same question in my mind and i couldn't find an answer so this is the idea that seems logic to me

[joshodgers]

I have been doing some testing on CPU reservations and the impact on CPU Ready /CPU scheduling.

The test was very basic, and I plan to do further testing, but here is what I have done so far.

4 VMs, 3 with 1 vCPU and the fouth with 2 vCPUs. (1 ESXi 5 host with 4 Physical cores @ 2.4Ghz)

I then ran CPUBUSY on all VMs, and two instances on the 2vCPU VM (as CPUBUSY) is not multi threaded.

So with all VMs running CPU Busy I had 5 vCPUs running as fast as they would go and therefor CPU contention.

I measured the CPU ready and CPUBUSY performance under various curcumstances and from my initial testing it seems that setting a CPU reservation of <100% of the total MHz of all vCPUs assigned to the 2 vCPU VM does improve performance (but CPU ready remains well above acceptable levels. (In my test 80% Reservation of Mhz had the VM running at >10% CPU ready)

Setting 100% resevration saw an immediate drop in CPU Ready from >10% down to approx 2.5%, which is much better, but not what 100% reservarion implies.

So my initial observation is if you have CPU Ready issues (you should Right Size the VMs first!!!) and your planning on trying to resolve the issue with CPU Reservations, setting anything less than 100% reservation will not result in acceptable CPU ready levels (<2%) depending on the level of contention.

I will continue testing and post Part 1 of my Article on this topic in my blog (below) in the next few days.

http://www.joshodgers.com

[VMwareBL]

In this scenario your over allocating cores by one core only or 25% (4 cores). You assigned 5 cores and have 4 so I wouldn't expect much of an issue. I would test like this, over allocate by double at least and then run the test. You should see high impact. This is not uncommon in real life in SMB sector.

There is no question that issue here and with my client for example was system was not scaled properly but that's sort of the point and mistake I see a lot in real world. People looking at hz performance tab to see if they have cpu issues thinking they don't when they do (hz will always be low unless you running high intensity CPU servers/apps) or assuming that setting reservation "gurantees" (this is the key word here) that VM will have CPU cycles and execution like it does for memory. You ask 9/10 system admins that manage vmware but are not sr. Level and they won't have a clue as to what CPU ready counter is because day to day management and troubleshooting or performance tunning are two separate beasts. That's sort of what I want to clarify and get to the bottom and have done here in approach by first asking the question to make sure I am not missing something. Always best take that approach in my opinion.

I have no doubt that reservations do help performance as they do, but again the topic here is "do they guarantee"

Let me know how your tests work out. Very curious.

[thgreyprnc]

I just stumbled uppon this thread and especially created an VMware account in order to be able to post this reply.

VMwareBL, you are perfectly right.  You are pinpointing something essential and none of the different reply's was able to precisely answer your question.

The point VMwareBL is trying to expose, is that a CPU reservation is very confusing to 90% of admins regarding that it is directly DEPENDING on what the physical processor core's are actually doing at a given time. Meaning that what is prevailing in ANY case, and in order to be able to actually USE the given reservation, is whether the core's are already doing something or not.

Because if all core's of that processor are processing a long calculation from something initiated from a VM within, let's say, a 50Mhz reservated pool AND configured with a number of vCPU's equal to the number of cores of the physical processor AND if at the same moment a request comes from a VM in a 2000Mhz reservated pool, that last VM is going to wait, regardless of his higher reservation.

The higher Ghz reservation just means that, for the same amount of time, the VM in the defined Ghz pool will be able to process the defined amount of cycles for the time the scheduler gives it the hand.

BUT, and here comes what we are talking about, in which case would this apply since when that given VM has the hand on the physical CPU no other VM can access it ? Thus, what sense does it make to cap the number of clock cycles it can use ?

>> If you have a very high clock cycle/Ghz usage VM, why would you want to give it only let's say 1000Mhz out of the 2400 that the core can do ? The remaining 1400Mhz would be lost since for the time the VM is processing on the core, no other VM can access it.

>>> If any calculation has to happen, the goal is that this calculation is processed the fatest possible. Like the title of this thread says it would be more clear to say something like "This ressource pool has an high priority access on the ESXi processor scheduler". Then it makes total sense.

I pain to understand this clock cycle limitation.

[joshodgers]

As discussed, here is the first post on my blog relating to Reservations and CPU scheduling.

http://joshodgers.com/2012/07/22/common-mistake-using-cpu-reservations-to-solve-cpu-ready/