Memory Ballooning

[tku703]

Hi all,

I'm new to this forum and kinda have a noob question on memory ballooning.  Now i know that memory ballooning is one of ESX RAM reclaiming techniques and it balloons a guest OS RAM and swaps it to the guest OS swap file the host created, is that right?

Say I have a VM with 2GB of RAM allocated, ballooning occurs and balloons to 1GB.  I guess my question is, this 1GB thats ballooned, is this reserved for the VM (1GB the host can't touch) or this 1GB moved off the VM?  Im just trying to understand how ballooning works and comparing that with the information under the "Resource Allocation" tab of a VM in Vsphere client.

Hope this makes sense.

[Busbar]

Please keep in mind that memory configuration depends on what is reseved and what is maximum memory configuration for the VM.

for the sack of simplicty, if the machine has reserved 1 GB memory and 2 GB memory max, then it will always gurantee the 1 GB but it can (on demand) go for 2 GB memory, ESXi will monitor the memory usage of the machine and if it is not hot pages will baloon them to claim the memory back to the host, to be used with other VMs.

hope that it is clear now.

[sparrowangelste]

"By default the balloon driver is used to reclaim idle memory."

"When the balloon inflates the GOS will first assign all “unused / free”  pages to the balloon driver. If this is enough it will stop. If this  isn’t enough the OS will decide which pages it will page out  until it  reaches its threshold."

http://www.yellow-bricks.com/2010/03/11/mem-idletax/

so that 1gb your vm has, if it has more idle memory or swappable mem, it can still be swapped out.

[tku703]

Wow, everyone, thanks alot all the replies.  So, correct me if i'm wrong here:

1.  When VM1 needs RAM, the ESXi host finds another VM (VM2) not using RAM

2.  Ballooning occurs on this idle VM2

3.  The host sends this reclaimed RAM from VM2 to VM1.

is that right?  So basically, the ballooned RAM (say 1GB using the example above) is unusable to the VM2 and it just there for show?  Also, where is the host swapfile for each VM located?

Thanks again!!

[sparrowangelste]

balloning should only occur if there isnt enough ram free.

but if there isnt enough free ram in the host, then yes 1-3 would happen...

for vm2 the ballooned ram would be pshycial  host mem it no longer has, but the vm still thinks it has 2gb.

the swap file is normally located with the vm, vswp unless you specify it is somewhere else

[Busbar]

AFAIK balooning is done dynamicly even if there is free memory to reclaim cold pages, correct me if I am mistaken

[rickardnobel]

Busbar wrote:

AFAIK balooning is done dynamicly even if there is free memory to reclaim cold pages, correct me if I am mistaken

The ballooning is actually only done when the host is under some memory pressure, as in something like only 4-6 % free physical memory. If not under that condition it will not do any ballooning (except a configured Memory Limit at a certain VM).

The ballooning process could not easy reclaim cold pages either, since the balloon driver must "hold" that pages by force inside the Guest Operating System, to prevent the OS from putting other data there.

[Busbar]

Thanks mate for elaborating that.

[proden20]

Ballooning and swapping are two different things.  Ballooning forces Windows to release unnecessary RAM address space on the host and force that data to pagefile.sys, thereby freeing up physical RAM for other VM's to use.

VMWare Swapping occurs when either a MB limit set by the administrator is reached or ballooning isn't solving the host's constraint problem.

[tku703]

So if a VM is ballooning its RAM, what data is being swapped?  I thought ballooning occurs on a VM thats running idle and only ballooning un-used guest RAM?  Is this data being swapped to the guest OS swap or host swap file?

[Busbar]

the swapping will occur with the VM is trying to use more memory and there is limits configured or there are no memory to supply by the host, then swapping will occur.

[rickardnobel]

tku703 wrote:

So if a VM is ballooning its RAM, what data is being swapped?

Hopefully no internal swapping needs to be done, but that the Guest Operating System could release pages used for cache or similar.

tku703 wrote:

I thought ballooning occurs on a VM thats running idle and only ballooning un-used guest RAM?

It does not happen when a VM is idle, but when the ESXi host is under memory pressure. The VMkernel has also no idea of what will be "ballooned" since that it is the Guest Operating System who will decide.

tku703 wrote:

Is this data being swapped to the guest OS swap or host swap file?

If the Guest is forced to swap it will swap only to its internal swapping file, e.g. pagefile.sys. The guest has no way of accessing the external Vmkernel swap file.

[asvidra]

Try to Explain with a simple scenario(if i am wrong please correct my understaning)

Lets take an ESX Server has three Virtual Machines(VM1,VM2,VM3), each using a RAM of 8 GB.

VM1 uses its maximum of 8GB RAM and now it needs more speed if any resource is try to add to it.

VM2 uses just 3GB out of 8GB, so now VM1 use the remaining RAM speed from VM2 to store its new resource. this is called as Memory Ballooning.

If VM2 needs more RAM, then the VM1 allows the RAM Speed it taken from VM2 and it came to hung state.

"Utilizing the unused memory is ballooning".

Message was edited by: asvidra

[rickardnobel]

asvidra wrote:

Lets take an ESX Server has three Virtual Machines(VM1,VM2,VM3), each using a space of 80 GB.

VM1 uses its maximum space of 80GB and now it needs space if any resource is try to add to it.

VM2 uses just 30 GB out of 80GB, so now VM1 use the remaining space from VM2 to store its new resource. this is called as Memory Ballooning.

That is not really the way it works. If VM1 has 80 GB of RAM configured it has no way of ever using more than that. Ballooning can not take any RAM from any other VM to give VM1 in this specific case.

[rickardnobel]

tku703 wrote:

it balloons a guest OS RAM and swaps it to the guest OS swap file the host created, is that right?

No, not really. If the VM is forced to swap it will do so only to its own internal swapping space, like the pagefile.sys in a Windows Server. The VM is not aware of the swap file created by the ESXi host.

[adrianych]

Sorry for bringing up this old thread as I found some in-consistancies in some of the theories above...

I have a Physical server with 72GB RAM and 4x 4core CPU.

- running 5 VMs with total 12vCPUs (2 vCPUs x4 VMs + 4vCPU x1 VM) and 38GB RAM (4GB x2 VMs + 8GB x2 VMs + 6GB x 1 VM)

- one of the 5 VMs is running Win7 64bit (4v CPU and 6GB RAM)

The VM that I noticed with ballooning is the Win7 64bit VM (running Acronis).

When the backup jobs started I noticed the performance chart of the VM (approx 45 min) :

- RAM usage was a U shaped, starting from 6GB , dropped to 3GB after 15 min then back up to 5.5GB

- RAM Balloon was a n shaped, started from 4GB, dropped to 3.5GB but remained there till almost when RAM usage went 4GB, then dropped to 0.

- CPU chart shows violent swings between 10% and 30%

So what was the ballooning for ?

[proden20]

I have seen this happen when the guest is paging heavily and can't keep up with the application's request for RAM (particularly under SQL.)

The basics:

- Make sure you do not have any limits set for CPU & Memory

- Make sure you have the latest VMWare tools driver installed

- Make sure you have 1.5x the RAM allocation (9GB) assigned to your pagefile (or select "System Managed")

- Ensure that your pagefile is not sharing the same disk spindles as the backup target

- Ensure any memory leak type bugs are resolved in Acronis by updating your software

- Ensure that you are using the Recommended memory size for your backup application and workload

- Determine if you can throttle Acronis memory consumption

See if any of those help.

[rickardnobel]

Adrianych wrote:

So what was the ballooning for ?

Based on the amount of RAM in your host and the vRAM attached to your VMs you should really never see any ballooning, no matter if the VM is paging internally or not.

Could you verify if the VM has any memory limit configured?

[proden20]

In our situation, we found that someone had forced sqlserver.exe priority over the Windows kernel, which forced an epic battle between OS and Application. The end result was the vmkernel trying to alleviate an erratic guest's memory requests with vmkernel swap and ballooning.

Once the DBA's reversed what they had done, all was calm and the ballooning and swapping stopped.

Strange, but true.

[rickardnobel]

Dennis Procopio wrote:

In our situation, we found that someone had forced sqlserver.exe priority over the Windows kernel, which forced an epic battle between OS and Application. The end result was the vmkernel trying to alleviate an erratic guest's memory requests with vmkernel swap and ballooning.

It seems incredible strange that the balloon or vmkernel swap should get involved in any way because of how the internal memory handling is done. The principle in general is that the vmkernel has no, and needs no, knowledge of what the guest place inside its own memory and should not have any opinions if the RAM is being heavily used by either 10000 instances of notepad.exe or a misconfigured SQL server. What the os and the running processes does in RAM "should" be invisible for the vmkernel, until either a vmkernel RAM Limit is reached or if being close to the amount of physical ram on the host.

[proden20]

Its incredible, but it happened.

I checked with the DBA's. "Boost SQL Server Priority" was turned on.  Turning it back off stopped the ballooning.  Probably not this user's problem, but I'll post it in case anyone else sees this crop up.

Boost SQL Server Priority

http://www.sqlbadpractices.com/boost-sql-server-priority/

[rickardnobel]

Dennis Procopio wrote:

Its incredible, but it happened.

I can of course not claim that it did not happen, but it is of course very strange and "should" not happen. :smileyhappy:

In some cases where multiple strange effects is visible it might be possible to belive that different actions were correlated which might have been also a coincidence.

[proden20]

It should not happen under the assumption that the operating system is properly managing its memory resources.  In this case, VMWare Technical Support logged in, checked the VM's config, reinforced that something inside the guest OS was causing the ballooning, and all was alleviated immediately when we disabled Boost SQL Priority.  Not much left to coincidence.

The OS could not serve up RAM pages as fast as the application was requesting them, nor could it page them away.  vmemctl kicked in to try to alleviate the OS' request for RAM, and subsequently vmkernel swap kicked in as well.

[rickardnobel]

Dennis Procopio wrote:

It should not happen under the assumption that the operating system is properly managing its memory resources.

I can not doubt that it happen when you say it did, however the balloon driver should have absolutely nothing to do how the VM is handling its memory resources. If the VM has good or bad memory management is something that either the vmkernel or other VMs should never know and never has to worry for.

The OS could not serve up RAM pages as fast as the application was requesting them, nor could it page them away.  vmemctl kicked in to try to alleviate the OS' request for RAM, and subsequently vmkernel swap kicked in as well.

In speaking of what "should" happen is that first when the outside physical host RAM is close to be exhausted the vmkernel could order the balloon driver inside the VMs to inflate and prevent them from using some part of what they belive is their own physical RAM.

One possible explanation is that if the host was quite close to using all physical RAM and a certain VM for some reasons, e.g. incorrect SQL handling, suddenly demanded much more RAM than before and this RAM was not available on the physical host, then this could have been a normal reason for balloon inflate. However, just a guess since I of course do not know the situation where you saw this.

[proden20]

I see what you are saying. That is a possibility.  There was 8GB free on the host, and this was the only VM on the host (no DRS or HA.)  Perhaps the vmkernel was being sensitive, and the SQL change calmed the noise down to the point it stopped swapping and ballooning.

[rickardnobel]

Difficult to tell long after the event. :smileyhappy: Interesting discussion anyway.

[adrianych]

Could you verify if the VM has any memory limit configured?

How do I do that ?

Its just a Win7 64bit PC running Acronis 11.5 VM edition with "SQL 2005 express" which came with the installer.

So again....kindly advise what is ballooning and what it does ?

Ballooning seems to be a "bad" thing to a VMware event to avoid....

[proden20]

Set Memory Limit to "Unlimited" as shown.

Ballooning is a way for vSphere to free up RAM on the ESX host when there isn't enough available to meet the guest's requests.  If the VM runs out of addressable, physical RAM, ESX will invoke the balloon driver provided by VMWare Tools to force less important pages in RAM to pagefile.sys.  This frees up RAM.  Ballooning is bad because when you see it this means that either the host is constrained or your VM has a limit set on it.  (Or, in rare cases, the Guest OS isn't managing its own memory allocation properly.)

If you are set to "Unlimited," your VM is either in a limited resource pool (do you use resource pools?) or something is going crazy with Acronis as I described.

I'd err on the side of a memory limit being set on the VM or parent resource pool before assuming its the application.  Keep checking off the list I gave you.