Recommended action to reclaim Memory in ESX

[LeftTurn22]

Hello there,

I have a 5-host VMware ESX 4 production environment running a number of Microsoft Windows Server 2008 virtual machines. Each virtual machine has been originally created with 4G physical memory visible to the guest OS according to the amount of memory VMware recommends for this guest OS when creating a new VM using typical settings.

While some virtual machines run heavy workloads (e.g. Microsoft SQL Server) other virtual machines run soft workloads (e.g. Microsoft AD Domain services, File and Print services). As a result, virtual machines running heavy workloads actually allocate almost all of the assigned physical memory visible to the guest OS while virtual machines running soft workloads only allocate a small part of their assigned physical memory, thus resulting in very high Consumed Host Memory per VM, Consumed Host Memory usage much larger than the Active Guest Memory and Active Guest Memory much different from what is seen inside the guest OS (e.g. Active Guest Memory is 4G while inside the guest OS only 1G is allocated on a Domain Controller).

Since I need to deploy some new virtual machines and run heavy workloads inside their guest operating systems, I am wondering about the recommended actions to perform to reclaim the unused Memory in ESX and assign it to the new virtual machines.

I understand ESX uses a number of Memory Reclamation techniques to reclaim and save host physical memory, but typically only when it is under memory pressure conditions and sometimes resulting in a negative impact to perofrmance. I also understand these techniques are almost fully automated even if the ESX Administrator actually can fine-tune some Advanced Settings through the VMware Sphere client.

Would you experts recommend shutting down the existing virtual machines and assign an amout of physical memory visible to each guest OS to better suit the respective workloads (e.g. allocate 1,5G to the Domain Controller above instead of 4G) or better leverage the Memory Reclamation techniques by manually modifying the default settings ?

Thank you in advance for your support.

Massimiliano

[mlubinski]

If you have hosts with Nehalem CPUs (E55xx) then High Host Mem usage for "soft VMs" is probably caused by bug in VMware with allocation of large pages. If this is true, and you have such servers, then first change following parameter on one of ESX host (reboot of a host is needed): Advanced Settings->Mem.AllocguestLarge Page set to 0.

Otherwise if it does not help, for low mem VM (DC, fileserver, printserver etc) assign lower mem like 1.5GB or 2.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

[LeftTurn22]

Hello,

thank you for your reply.

My hosts actually have Nehalem E5520 CPUs. After reading your reply and performing some search I finally ended with VMware KB Article 1014019 which addresses the exact issue as described below and provides a patch to fix the issue:

"Guest operating system's memory usage might be overestimated on Intel systems that support EPT technology or AMD systems that support RVI technology. This issue might cause the memory alarms in vCenter to go off spuriously even if the guest is not actively accessing a lot of memory"

The KB article confirms large Host Memory usage for "soft VMs" is acually caused by a bug in VMware with allocation of large pages. While changing Mem.AllocGuestLargePage to 0 is a valid workaround to adress the high memory usage on such hosts, the KB article recommends undoing it (if it has been modified) and then applying the patch to fix the issue.

After going through the steps described below and looking at the data reported by the vSphere Client my final thought would be to always assign a vitual machine guest OS an amout of physical memory to exactly suit its workload, but I would like to share the thought with you experts.

Here are the steps taken using an existing Windows XP SP3 32-bit VM with 3G of physical host memory assigned along with the corresponding memory consumption from the vSphere Client before applying the patch:

==================================================

Consumed Host Memory: 1677,00 MB

Active Guest Memory: 880 MB

Current Memory usage from Task Manager inside the Guest: 180 MB

==================================================

I have then migrated the powered on VM to a different host using VMotion, put the host in Maintenance Mode, applied the patch, rebooted the host, exited Maintenance Mode and migrated the powered on VM back to patched host. I have also shutdown the VM and powered it back on to wipe the Consumed Host Memory out from the host and finally looked at the data reported by the vSphere Client:

==================================================

Consumed Host Memory: 2107,00 MB

Active Guest Memory: 81 MB

Current Memory usage from Task Manager inside the Guest: 180 MB

==================================================

According to the displayed data the vSphere Client issue has been addressed since the Active Guest Memory reported almost reflects the actual current Memory usage from Task Manager, while my current host (running only this "test" VM) Memory usage is >3G !

The described phenomenon seems to affect only Windows-based guests. The following data refers to an existing FreeBSD 5 64-bit VM with 4G of physical host memory assigned:

==================================================

Consumed Host Memory: 537,00 MB

Active Guest Memory: 163 MB

==================================================

What's going on out there experts ?

[mlubinski]

Hi,

What does this xp have installed inside? Is this 2gb host mem usge

just after boot or maybe after some time after booting?

Sent from my iPhone

On May 15, 2010, at 12:34 PM, "massimiliano.rizzi" <[e-mail]

[LeftTurn22]

Hello,

this is a Windows XP SP3 test installation + MS Office 2007. The 2GB Consumed Host Memory usage is after about 4 hours after booting.

I have also attached to the post a screenstamp from the vSphere Client and from within the guest OS.

Thanks.

Massimiliano

[VMmatty]

I don't want to get too far off topic, but I want to make sure you understand that the KB you linked and the use of large pages is not a bug. The bug that was fixed in the KB article was that vCenter was causing alarms based on the high memory usage. The patch fixes it so vCenter does not fire off high memory usage alarms when guests are using large pages.

The use of large pages, and the lack of memory sharing with transparent page sharing, is by design and working as expected.

I wouldn't jump to use the advanced setting to disable the use of large pages. Windows 2008 (among other operating systems) uses large pages by default and may see performance benefits from doing so. ESX will automatically break up the large pages into small pages as soon as you overcommit memory (assign more memory to VMs than is actually available in the host).

[mlubinski]

Well in my case I had new cluster with all updates enabled and after

moving vms from 3.5 to new setup immediately host mem usage jumped to

max value. I contacted vmware about that and engineer said that with

u1 it was fixed but seems that next patches (after u1) broke it again.

I was explicity told to set large pages to zero and that solved my

issue permanently. That is why I mentioned this :smileyhappy:

Good luck

Sent from my iPhone

On May 15, 2010, at 3:18 PM, VMmatty <[e-mail]

[Samcer]

Hi Massimiliano after reading your post I tried to set up (as quick as possible) a test environment just to prove the problem.

My lab is based on ESX 4U1 208167 on Intel 53xx (no Nehalem). I understand it is not your equipment...

In my environment i tested: Win 2008, Win2003 and Linux with 1 and 2 GB Ram (no applications, just AD in Windows VMs)

From the "memory usage" point of view, Win 2008 seems the worst VM while Win 2003 is working better. (To be honest Win2008 was 64 bit while Win 2003 is 32 bit SP2)

Anyway even for win 2003, I see something "wrong" in the "Resource Allocation" Tab (point to the Win VM and see "guest memory" bars --&gt; shared memory).

The picture I see for the Linux VM (which is 64 bit) is much different and it seems to show me right metrics.

I would say (after this quick and dirty test) the problem could sit in the VMware tools for windows VMs

Coming back to your first question "deploying VM with large OS memory" I would also take care about memory overhead which increase by increasing the memory assigned to the VM (see Resource Management Guide page. 28)

my 2 cents

Sam

[benq201110141]

Hi,

was a resolution for this found?

I'm having the same 'issue' of vm Consumed Memory not accurately reflecting Active Guest Memory of the vm.  more over, the consumed memory is for a large number of vm's up to 75%-100% of Configured Memory, where actual o/s mem usage and Active Guest Memory usage is around <25% of Configured Memory.

Our environment is:

Cluster: 3x ESX, 4.1.0 Hosts (HP DL380 G7)

I note that the O.P. said applying the patch fixed his issues.  but VMMatty said that patch relates to the vCenter incorrectly triggering alarms, and not a fix for the vm's mem issue.

any feedback is appreciated.

cheers

ben