Understanding ESXi memory

[7007VM7007]

Hi All

I am in the process of learning ESXi 5.5 on a host at home and have some questions about how memory is managed by ESXi. I have read quite a bit online but I am still a bit confused!

My test host has 32GB. I have created a number of VMs and have allocated them 50GB of memory in total. When I click my vCenter server in the web client and then click the summary tab it says:

Free memory 1GB

When I then click the monitor tab it says:

Granted: 50GB

Consumed: 29GB

Active: 24GB

Shared common: 1.5GB

Swap and balloon: 1GB

I know I have (intentionally) overcommitted the memory but why is vCenter alerting me all the time that the host memory usage is high and why is the server swapping if only 24GB of memory is active? From what I understand, active memory is what is ACTUALLY used so in this case I am using 24GB out of the total of 32GB so I should be able to use 8GB more of memory without swapping?

Also, how much swapping of memory is acceptable? All my storage is on SSDs (including the VM swap files).

Am I misunderstanding something here? :smileyconfused: I thought I would only get memory alerts/alarms when the active memory approached 32GB...

[Alistar]

Hi there,

the ESXi has quite complex and smart memory allocation technique. Let me break it down a little bit for you:

Once the ESXi is booted up, it consumes a certain amount of system resources for itself. When you start booting up Virtual Machines, each of them has a certain memory overhead according to its vCPU count and RAM amount which is used for VMkernel's resource scheduling purposes. The VMkernel actually needs a certain amount of memory to keep performing well (as any other OS) and therefore some thresholds are set.

Once the first ("soft") threshold is reached, the Transparent Page Sharing (see TPS | VMware vSphere Blog - VMware Blogs) mechanism is activated. If the memory commitment grows even more, VMkernel begins to reclaim some unused memory inside the OS (but never more than 35% of its allocated memory) using the Baloon driver (inflate = retrieve unused memory from the running VM and provide it to VMkernel).

When the RAM use grows further, memory compression algorithms are engaged (VMkernel analyzes the pages and these which promise at least 50% compression ratio are compressed, up to 10% of total VM's memory). When the worst scenario happens, the VMkernel will start swapping Virtual Machines' memory to their .vswp files.

Please take note that swapping should never happen, even to an SSD - if you take a look, RAM access is measured in units of nanoseconds, whereas SSD access times are measured in tenths of milliseconds, this means that swapping on SSD will be still ~10.000x slower than to RAM.

Therefore in short:

• Free Memory: Quite Straigntforward - this is the amount of free pages for in-memory manipulation done by the VMkernel - this does not only concern user-invoked processes (a SSH console session for example), but also the operations inside the VMkernel itself.
• Granted: This is how much memory you have chosen to allocate to all virtual machines that are running on the ESXi host.
• Consumed: This is how much RAM is "eaten up" by all worlds.
• Active: This memory is being actively accessed.
• Shared Common: Using Transparent Page Sharing between Virtual Machines, this is how much memory has been saved by that technique
• Swap and Baloon: This is the number saved by balooning and swapping techniques.

For extensive information, look here: Memory Counters

[7007VM7007]

Thanks for the helpful answer!

Reading through your answer made sense but I'm still confused as to why my host is swapping if the active memory amount is below the physical amount of memory I have?

ie: 32GB of physical RAM

28GB is active

1GB swap ---> why is there swapping occurring if the active memory is less than the physical memory?

Also, I thought I could way overcommit my memory and, provided the active memory was less than the physical memory, there would be no swapping?

:smileyconfused:

[Alistar]

As I have described above, the VMkernel needs some part of memory to be solely for its own use in order to perform. Below is a table taken from our live environment (if you put it in a graph, it is almost a logarithmic curve):

GB RAM	Reserved MB
24	2378
48	2682
96	3467
192	5129
512	9985

This memory value is always provided to the host and never swapped out and would make around 2,5 GB for your 32GB setup. This is why VM performance suffers because their ram gets swapped out to disk, never VMkernel which stays in memory at all times. The active memory is being "touched" and needs to physically fit in with the rest of what is happening under the hood. You should be looking at "consumed" value as this represents all memory pages in RAM.There is some space for overcommitment in ESXi, but it all depends on use cases (it is best used for many identical machines).

Therefore: 29G consumed + 2.5G overhead +1G free ~ 32G physical memory.

Try to read up some more on resource management inside ESXi if you wish to understand this issue deeper: http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf

[7007VM7007]

Thanks again! :smileyhappy:

So I have adjusted my VMs now to use 29GB of memory in total between them. In the Monitor - Performance tab of the web client I now have (while ALL VMs are on):

Active: 5GB (currently used and active memory)

Balloon: 0 (zero is good)

Consumed: 29GB (this is due to me allocating 29GB memory to ALL VMs)

Granted: 31.5GB (This made up of 29GB consumed + 1.6GB system memory usage and??)

Shared common: 1.1GB (memory saved from sharing)

Swap used: 0 (zero is also good here)

(FYI: System memory usage on the summary tab is: 1.6GB and free memory is 4.3GB)

So looking at the above I would say the system is now configured to run well? No swap or ballooning is happening which is good from what I understand.

Just want to make sure I am on the right track in understanding this...thanks!

[Alistar]

Congratulations! You pretty much hit the sweet spot here. Just remember to stay on at least 982MB free memory at all times - this is above the limit that intiates the first immediate memory saving technique - Transparent Page Sharing. The formula is 900 + 1% of all RAM above 24GB. For your example it's 900 + (32-24)*10,24 = 981,92 MB.

Transparent page sharing is done even when there is no memory contention, but only scans the memory pages once per 60 minutes.

Anyways, I was happy to explain all of this to you and help you gain insight :smileyhappy:

[7007VM7007]

Thanks! :smileyhappy: Everything (so far) is performing great. The SSDs make a huge difference.

Would you mind confirming (or correcting) my previous post on the memory amounts? I don't quite understand how the "granted" memory amount of 31.5GB was arrived at?

Can you also comment on where the following fits in:

System memory usage on the summary tab is: 1.6GB and free memory is 4.3GB

I guess what I am trying to understand is, exactly how is the 32GB (or 32725MB) of memory broken down.

I think I am close to understanding this :smileylaugh:

Appreciate your help!

[Alistar]

When you take a look at the link where the metrics are explained, you will see that Granted is a metric of:

• Amount of machine memory or “physical” memory that is mapped for a virtual machine or a host. On a host: Sum of all granted metrics for all powered-on virtual machines, plus machine memory for vSphere services on the host.

So this is the memory VMkernel has granted (in its own physical constraints aka the memory sticks) to have everything running - various Daemons ("services" in Windows terms) inside the VMkernel (CIM Providers, HA agent, management agents, remote GUI and SSH console sessions...), Virtual Machine Worlds (world is a group of processes bound together in VMware terms), VMkernel resource scheduler itself - there are many cogs and spinning (and dynamic, at that) and literature would be your best stop to see each and every component the VMkernel is actually composed of. You can take a look via esxtop in an ssh session.while performing tasks in vCenter or in the VM itself to see how everything moves around.

And well, free memory is memory that can be allocated on-demand to any component mentioned above. Its resource management. Just go around, experiment and read on the matter - it is truly a work of art :smileyhappy: