VMware vSphere

Hi All,

I am new to the vmware technology,  Please help me out to understand the Limit  attribute on vm memory setting.

I just refers through some site and made a conclusion that , the limit is the maximum memory a VM can use even though its configured memory is higher than limit. once VM reached the limit , then the balloon driver kicks in and start the paging on VM ,thus esx limit the memory utilization up to limit.

For example , i created a VM with configured memory of 1024MB, and set a reservation to 256MB and limit to 756MB. now when VM is powered on ,

Because of reservation esx always assign 256MB of machine memory to VM, and the gap between reservation max 256MB to Limit max 756MB may be filled with available physical memory or using swap aread of esx. But once vm reached the memory utlization of 756MB, then the ballooon driver comes in to play ,ie the esx will inflate the balloon and clamp theVMs memory utlization on 756MB.

So in short VM cannot user a single page of memory after 756MB.

Here is my Doubt,

I simulated the same scenario , by creating a Linux VM of configured memory 256MB, Reservation 71MB ,and Limit to 118MB. shares are set to low.

But ,when i am analyzing the performance matrix for VM , i can see the consumed memory(machine memory allocated to VM ) go above the limit that up to 165MB, even though i limited at memory at 118MB.

Please help me out for this !!.   i am attaching the outputs for your reference.

Is there any relation with balloon values with the above dubt

Also correct me ,if i am wrong in terms of the concept understandings 

Million thanks in advance. 

Hi

Have a look at this blog posting by Frank Denneman

http://frankdenneman.nl/2010/06/reservations-and-cpu-scheduling/

Also this posting on resource pools seeing as you're looking at limits and reservations

http://www.yellow-bricks.com/2010/02/22/the-resource-pool-priority-pie-paradox/

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

Gregg Robertson

First, I really recommend that you read the Resource Management Guide, it is an excellent guide for understanding how reservations and limits work.

To answer your question, the consumed memory is a combination of the memory consumed by the VM as well as any overhead that the VM may have (every VM has some, you can see this on the summary tab).

Next, the balloon driver only comes into effect when there is memory contention on an ESX host, reaching the limit will not cause the balloon driver to activate.  The balloon driver is also a part of VMware tools so without that installed the driver cannot activate.  When a VM reaches the limit it ESX will discontinue granting the VM memory so the VM will likely lag or start swapping, this is generally bad.

For reservations, ESX does not just assign it reserves memory blocks when the VM boots up that are not shared with any other VMs until the VM is powered down.  ESX will also not power on a virtual machine if it cannot meet its memory or CPU reservation.  Reservations also can come into play during cluster admission during a HA event.  The last thing that a reservation will do is change the size of the ESX VM swap file (.vswp), the vswp file is always (configured memory - reservation)MB, so if you configure for 1024MB and reserve 256MB expect to see a swap file of about 756MB.

Another important thing to note, ESX will only swap as a last effort, and this is good as your performance will suffer in a major way because it swaps indiscriminate of how active the actual memory pages are.

What is the scenario that you are looking to use limits in?  It is considered best practice not to use limits as this could cause unnecessary bad performance inside your virtual machine.  The only situations that I would use limits in is to "jail" a bad virtual machine and usually I use this on CPU only or in the event of a hardware failure where I wanted to cap un-important VMs.  If you just want your VM to use less memory you should just configure it with less memory.

Hi

Welcome !! :-)

You're almost right. The limit is the setting that limits the use of PHYSICAL host memory, but not swap memory.

In your example:

1024MB assigned, 256 MB reservation, 756MB limit.

What happens:

1- At boot time ESX wil check to see if it can allocate 256MB physical host memory to the VM. If that is possible the VM can be started.

2- When the VM starts using more than 756MB, all memory above 756MB will be used from the Swap disk, which is very slow compared to physical RAM.

The ballooning starts when the limit is reached and ESX will tell your Guest OS to move rarely used / unimportant memory pages to local harddisk (swapfile inside the Guest), but if there is still not enough free memory, it will swap to disk.

Keep in mind with reservations, that ESX has to make sure there is 256MB of RAM for that VM in physical memory otherwise it won't start !!! When using a lot of reservations, this could get you into trouble. Say you have 10 VMs, all between 4 and 8GB RAM and for each VM you reserve 2GB of physical RAM, then to be able to start your VMs you need 20GB of free physical memory. Even though they're not using all the RAM they have a reservation for.

With limits you should also be careful since these immediately impact the performance because they will swap to disk all memory above the limit. A common mistake is that admins set the limit equal to the assigned memory. There is no need for that and it will actually impact DRS performance.

So common sense rules:

- Don't use limits unless you really really, I mean really have a valid case for that

- For reservations there are more use cases, but also be very careful when using them.

- With many things in vSphere environments that influence performance (using limits, reservations, shares, etc for cpu / memory / network / disk) it is often that vSphere knows best and you should only use them when you understand the impact.

Tips for good read on severall of these subjects:

- http://www.yellow-bricks.com/

- http://frankdenneman.nl/

and also my own site :-)  http://www.gabesvirtualworld.com/

Have fun with ESX :-)

Gabrie

Hi  Gabrie,

Thanks for your valid information.

As you said , The ESX will allocate memory from SWAP DISK , when vm memory utilization shoot above the limit.

Then in that case, The consumed memory should remain a constant  and the swap utilization needs to rise (which means , the memory will allocate from <vmname>-<uuid>.vswp .

While we checking the performance matrix , it should show ,the consumed should stay constant and as the vm use swap disk, the swap in and swap out graph should show some activity ???

But in my simulation , it is not showing any value.

"A common mistake is that admins set the limit equal to the assigned memory. There is no need for that and it will actually impact DRS performance. "

Could you elaborate more on this? Is the overhead memory included in the limit?

I agree with what you are saying, but I would like just a little more info on why you think this is bad.

I just started working at a new environment and the current admin has done this for EVERY vm (over 500 vm's).

The statement that setting a limit equal to max memory assigned infects performance negatively is 100% correct, however ......  the impact is only on your vCenter server, since there the calculation is done. What happens is that when DRS has to calculate the entitlement of memory resources, there is a difference between "unlimited"  (no limit set) and "4096" where it is equal to the memory assigned. When set to unlimited, calculations are quicker.

I doubt however that with 500VMs you'll notice the impact on vCenter. Problem however is that when your environment grows it suddenly might reach the point where there is an issue and you will have trouble finding the crullpit.

Biggest drawback of setting the max limit is that you easily forget about it. I visit a lot of customers and I often see that a mem limit was set to 4096MB for example with 4096MB assigned. Then the Windows admin asks for more memory and the VMware admin assigns 6144MB but forgets about the 4096MB, so 2048MB will be swapped to disk and the Windows VM suddenly is dead slow.

Plus, there is no reason to set this memory limit, so just don't. Mostly these are set because the admin doesn't understand how the settings work. There are a few powershell scripts around that help you detect and change the settings back to default.

Gabrie

thanks for the explanation. I've always known it to be best practice to not set one unless really needed, but I wanted a little more info on why. Hate to be the new guy and come in and tell the current admin he's doing it all wrong - instead is better to explain why I think we don't need the limits.

One last question - is memory overhead included in the limit?

If you give a VM 1024mb, and limit at 1024mb , and say it has overhead of 100mb, does this leave 924mb for the OS before swapping starts?

No, the memory overhead needed by the VMkernel is not part of the memory assigned.

Hi Gabrie,

I know this topic is old, but that is what I was looking for a long time and your answers helped me a lot. There is only one thing I need to know to be satisfied and more slight!  When u said: "With limits you should also be careful since these immediately impact the performance because they will swap to disk all memory above the limit. A common mistake is that admins set the limit equal to the assigned memory. There is no need for that and it will actually impact DRS performance.", I just wanna know "who" will swap to disk?! Host or VM? Since I had set a limit, I understood that ESXi wouldn't swap to disk whom would do this would be VM. Wouldn't it?

In the last example:

2- When the VM starts using more than 756MB, all memory above 756MB will be used from the Swap disk, which is very slow compared to physical RAM.

In this case, what I understood was: ESXi has physical memory enough, but because of set limit, ESXi won't give more memory to VM, however for VM 1024 GB was set, so VM starts to swap to disk.

I don't know if I made myself understood. Really thanks!

rafaelrwrc wrote:

Hi Gabrie,

I know this topic is old, but that is what I was looking for a long time and your answers helped me a lot. There is only one thing I need to know to be satisfied and more slight!  When u said: "With limits you should also be careful since these immediately impact the performance because they will swap to disk all memory above the limit. A common mistake is that admins set the limit equal to the assigned memory. There is no need for that and it will actually impact DRS performance.", I just wanna know "who" will swap to disk?! Host or VM? Since I had set a limit, I understood that ESXi wouldn't swap to disk whom would do this would be VM. Wouldn't it?

In the last example:

2- When the VM starts using more than 756MB, all memory above 756MB will be used from the Swap disk, which is very slow compared to physical RAM.

In this case, what I understood was: ESXi has physical memory enough, but because of set limit, ESXi won't give more memory to VM, however for VM 1024 GB was set, so VM starts to swap to disk.

I don't know if I made myself understood. Really thanks!

Swap will be done by the host and it will be the memory of the VM that will be swapped. It is always host memory since we're talking about physical RAM of the ESXi host and by setting a limit you will limit the VMs usage of that physical RAM.

Dont get confused by swapping that is done inside of the guest, this is purely a guest OS thing.

And on number 2: yes you're correct. The guest OS sees 1024MB, a limit is set to 756MB, all memory that is in use by the VM that is above the 756MB will not be given from physical RAM, so the host will swap this VM memory to disk.

In most of these cases I don't think the VMware administrators are setting these memory limits to be equal to the assigned memory.  I also see this at many places and when I show it to the VMware administrators often they did not even know that they could set a memory limit.

I think there was an issue with an older version of vCenter, probably back in the v3.5 days, that set these limits and they are still set because the administrators are unaware of them.   PowerGUI with the vmware admin pak has a script to check for memory limits on VMs and removing them.  I normally use this at new customers I visit.  If the limits are still set at the same as the assigned memory it is not an issue, it is when the assigned memory has been increased on these VMs and the limit left in place because the administrator did not know about it, this starts to cause performance issues with the administrators wondering why a VM that they have added more memory to is now performing worse than it was before the memory was increased.

You can check out this blog as well:  http://vsential.com/archives/vsphere-memory-counters-de-mystified/