Why am I ballooning here? [Solved]

[jwnchoate]

I have a large number of high memory vm's. As such, I am always watching memory allocation. We have a number of excessively allocated VM's, and its a constant battle to keep my app people from overreaching for resources.

I have had some strange cases where a busy host will show a lot of ballooning, but they aren't paging internally, nor is my host granting memory above physical memory. I ran a test recently. I took a linux vm and set it to use 12GB of ram and 8vcpu's. I placed it on a relatively idle box and ran some tests.

I used a ram disk and copied some large files into it. I expected to see my vm's granted memory rise and squeeze up to the hosts physical memory. I was mainly looking for the effects of when one hosts begins to use a lot of memory.

Here is the Hosts view of memory. The 40G mark is at the top of the graph. As you can see, when I start to copy files into the ramdisk, the granted line rises breifly, Active memory rises, but then the ballon kicks in and Granted memory falls. All the while, the page file on my linux host is ZERO. Is this ballooning really going to the .vswp file?, if so, why does Memory Swap used also flat lining just above 0 the whole time.

Here is the linux host, its graph is all over the board. First, Granted and Consumed seem to be almost identical all the way. When Active memory spikes, Granted and Consumed Plummet? My Balloon is higher than my Granted and Consumed memory and my VM's swap file is still ZERO.

1. I was under the impression, that a memory load would cause the other VM's which were idle to balloon and page do disk, not the busy host itself.

2. If I am ballooning more memory than I have granted and consumed, and my swap file is ZERO, where is this going? The linux Free command shows as if it is all sitting in RAM.

3. Why is Granted memory not going up, there is many gigabytes left on the host.

4. VM doc says ballooning really shouldnt occur in until high overallocation occurs, and physical memory runs out, but it appears that in this and in other production cases, that VMware can start ballooning before its technically necessary to do so.

============================

the above is the result of a test, I have recently had similar results in my production, where we have several pre-production vms sitting idle with a lot of ram. Then one of them gets busy and starts to balloon like mad. for example, I may have 6 8G servers and they all use 500MB consumed with 150M active. Then one host kicks in and its slow as heck, but my esx host shows 20% usage.

[khughes]

I'm running into a similar situation trying to upgrade from 4.0U1 to 4.1 where we have Citrix XenApp VM's (2 vCPU + 2GB RAM w/ no reservations), they run fine on 4.0 but moving to 4.1 the performance hits rock bottom. Even running as the only VM on a 4.1 host ( dual dual processors + 30GB RAM) vmware support is saying the VM is ballooning and swapping in memory while the VM is only showing it using maybe 40% of the ram allocated to it.

-- Kyle

"RParker wrote: I guess I was wrong, everything CAN be virtualized "

[jwnchoate]

Here is my pre production setup, I have stacked most of these guys on one host to give an example. There are a large number of idle boxes, but as we go to prod I will spread them out.

However, there are 22 vm's. Many 8G and 4G vm's. if you look most are idle and not using memory. Total VM MB used is 32.6G, but I have 100G configured for all. VMware is supposed to allow overallocation, and I could understand if all theboxes were hammering away, but the 32G is still below the host's physical RAM at 40G. It appears as if its ballooned a bunch of unused memory.

VMware has granted aound 62G, and Consumed is around 35G. Why so much of it is ballooned, the boxes are mostly idle. I suppose 35G consumed, 30G balloon can add up to the granted, but im clearly not using that much memory within the guests. From what I can tell, even if there was some ballooning, there should not be 30G of it. Its as if vmware saw the excess and just decided to go ahead and balloon a lot of it anyway.

[esloof]

During my weekly VMware training sessions, I’m always getting a bit excited when we reach the module where the ESX and virtual machine memory is covered. I’m trying to explain how important it is; not to use the virtual machine memory limit setting and very often students believe me after checking out their own production environment for ballooning activity and 9 out of 10 times they find a few virtual machine which have a memory limit configured. The memory limit is the upper limit on memory the ESX host makes available to virtual machine. Host memory reclamation happens regardless of the current host free memory state. For example, even if host free memory is in the high state, memory reclamation is still mandatory when a virtual machine’s memory usage exceeds its specified memory limit. If this happens, the ESX server will employ ballooning and, if necessary, swapping to reclaim memory from the virtual machine until the virtual machines host memory usage falls back to its specified limit.

I’ve recorded a video which shows you the Proofpudding and collected some useful links about this topic.

Scott Herold: Memory Behavior when VM Limits are Set

VMware KB: Virtual machine boots very slowly when Memory Limit

Arnim van Lieshout: PowerCLI: Reset CPU and Memory Limits

[jwnchoate]

Excellent idea, but I just checked, no limits set.

On a side note, I recently migrated my bigged memory user to another box and the effect was no change to balloon, but a little drop in granted memory.

[jwnchoate]

Correction. As for the test at the top, the linux vm was originally a cloned physical box. It did have a memory limit at 4G, will explain the memory issue with the test. Ill have to keep that in mind when I migrate a vm and increase its memory.

Now, look at the second post. none of those guys got a limit. Its what started me testing. I have been migrating out all the boxes with the highest mem usage and balloon isnt dropping...im slowly migrating out so as to let things setting a bit before moving on.

[jwnchoate]

Using ESX top, it does appear boxes with very large mem settings and sitting idle or low mem usage have been ballooned away.

GID MEMSZ GRANT SZTGT TCHD MCTL? MCTLSZ MCTLTGT MCTLMAX OVHDUW OVHD OVHDMAX

61 8192 2051.16 514.28 28.67 Y 5324.51 5324.51 5324.51 169.95 261.15 355.12

64 8192 2216.22 521.62 57.35 Y 5324.51 5324.51 5324.51 169.95 263.55 364.69

65 8192 2133.3 515.73 28.67 Y 5324.51 5324.51 5324.51 169.95 267.95 362.77

69 8192 2161.51 532.87 0 Y 5324.51 5324.51 5324.51 169.95 279.15 372.03

68 8192 1550.08 1445.71 116.33 Y 5283.72 5283.72 5324.51 169.95 314.8 416.84

63 4096 968.5 399.45 28.68 Y 2662.11 2662.11 2662.11 105.77 189.99 283.76

38 4096 1227.16 1407.73 76.43 Y 2567.4 2567.4 2567.4 93.58 203.7 307.34

[esloof]

I’ve one last question, you might want to check the amount of memory kept in reservation. If there’s a lot of memory reserved for a VM, it won’t be used by other VMs and they start ballooning even though at host level there’s enough free memory.