Using 30GB memory for VMs on a 32GB machine

[geertayan]

I'm trying to start 3 VMs with a statically reserved amount of memory (10000 MB) on a 32GB machine installed with VMware ESX 3.5, latest patches applied. When starting the last VM I get the error: "Insufficient memory resources".

When I look at the memory settings in VIC I see that there should be 30137.0MB free for using within VMs:

Total: 32766.3MB

System: 2357.3MB

Virtual Machines: 30137.0MB

Service Console: 272.0MB

However, when looking into the performance statistics for the machine I see that 24GB memory is granted even though only 2 machines with 10000MB are running. I would expect this to be less (22GB). I would expect to be able to start another VM with 10000MB.

When I lower the amount of memory reserved to 8000MB I can start the VM. After starting I can raise the reservation back to 10000MB again without any error. However, when the VM is brought down and up it does not start.

Is there any trick to lower the amount of reserved memory for ESX so it can be used for VMs? I already tried to disable VMware HA and DRS to no avail.

[Ken_Cline]

OK, couple things...

1. Why are you reserving the memory for the VMs? When you reserve the vRAM, you are forcing it to be available in physical memory. If you do not reserve the vRAM, the allocation can be satisfied via the .vswp paging file that gets created on VM startup.

2. Each VM has overhead memory that is used for managing the VM. The amount of overhead memory will increase with the amount of vRAM allocated to the VM.

If you REALLY need that much memory allocated to each VM, I would suggest that you reduce or eliminate the RAM reservation and allow ESX to manage the memory as it sees fit (it can usually do a better job than any human administrator). Be aware that, when you lower the reservation, you will be creating a .vswp file when you power on the VM. This .vswp file will be roughly equal in size to (RAM Allocation - RAM Reservation), so if you allocate 10GB to your VMs and you Reserve 5GB, then your .vswp file will be approximately 5GB. This file will (by default) be created in the same directory as your .vmx file.

Ken Cline

VMware vExpert 2009

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

[geertayan]

We reserve the memory because:

- These VMs run linux and we want to emulate the real world as best as possible

- We want to use excess memory (memory not used for the java processes) for disk cache

And most importantly:

- we experience severe performance problems in java when not setting the reservation to the amount of memory allocated to the VM. System traces tell us that it tries to allocate memory and fails to do so resulting in extreme high cpu load and extremely bad application performance.

Unfortunately the 10GB memory amount is a business constraint. If we wouldn't be able to load 3x10GB VMs onto the 32GB machine we would run into sizing issues (we are talking about ~120 VMs).

[geertayan]

After thinking a bit I would also like to see how much memory overhead the VMs have. According to esxtop they use exactly 10000MB.

[Ken_Cline]

After thinking a bit I would also like to see how much memory overhead the VMs have. According to esxtop they use exactly 10000MB.

Look here: http://communities.vmware.com/community/vmtn/general/performance?view=documents&numResults=50 - there are several good documents that discuss memory utilization and monitoring.

Ken Cline

VMware vExpert 2009

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

[Ken_Cline]

We reserve the memory because:

- These VMs run linux and we want to emulate the real world as best as possible

- We want to use excess memory (memory not used for the java processes) for disk cache

And most importantly:

- we experience severe performance problems in java when not setting the reservation to the amount of memory allocated to the VM. System traces tell us that it tries to allocate memory and fails to do so resulting in extreme high cpu load and extremely bad application performance.

Fair enough - although not Reserving the memory should not cause allocation failures. As long as you Allocate the memory to the VM, the guest OS should be able to malloc() it.

Unfortunately the 10GB memory amount is a business constraint. If we wouldn't be able to load 3x10GB VMs onto the 32GB machine we would run into sizing issues (we are talking about ~120 VMs).

With only those three VMs on the system, the Reservation setting should have very little impact on overall performance. I suggest that you try leaving your allocation at 10GB and setting your Reservation to 8 or 9GB. This will allow your VMs to start by leaving enough room for overhead. You should also notice that the amount of Active memory will decline and the amount of Shared memory will increase over time due to transparent page sharing.

Let us know how it works out!

Ken Cline

VMware vExpert 2009

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

[geertayan]

We do see VMs with 10GB allocated but not reserved the same issues. Very bad application performance because of java trying to allocate memory pages and not being able to. Maybe we should open a support case for this (time is a very limiting factor though). But for now I guess we might go with your suggestion if we cannot find a way to get this running.