ESXi & W2K8 Memory

[gregchait]

I have been supporting many virtual environments for years; almost exclusively with Windows 2003 as the guest OS.  This autumn I began supporting two environments now using Windows 2008 as the standard, one which is growing at a fairly rapid rate.

The issue I am having is the high rate of memory consumption.  It seems like 2/3rds of all our guest VMs are consuming 100% of their allocated memory.  This does not seem right to me.  Two servers which are of particular interest to me is our SharePoint server and its SQL back-end.  They are both allocated with 8 GB of memory (a gross overallocation IMO - but lost the battle on this).

vCenter is stating that each of the servers average around 700 MB of active memory and peak at 900 MB (SQL) and 1.9 GB (SP) over the course of a full work week.  These are some of the direct questions I am trying to answer:

* I am puzzled at this discrepancy and how I can resolve consumed memory?

* Is this directly related to Windows 2008 being a 64-bit OS?

* Are there configurations I can explore to improve this?

* Is it simply a matter of carefully tuning allocated memory?

* Why are some OS' consuming all their memory while others are not?

To make matters more complicated, I ran a full PerfMon on all memory attributes and the amount of "available memory" does not come close to aligning with active memory through vCenter.

Any thoughts or advice would be greatly appreciated...

~grc

[AndreTheGiant]

Windows Server 2008 (and also R2) has a different memory management approach, comprare to 2003.

It's more like Linux that take all the available free memory and use it for the filesystem cache.

This make the free mem very slow, but you can see in the task manager if you have also a lot of swapping (not good, in this case you have to increase the mem) or a lot of cache.

Andre

[bulletprooffool]

Andre is correct - 2k8 is a bit of a hog in the way that it pre-allocates memory.

Are you overutilising your memory for your ESX estate, or just concerned at the memory in use?

If your environment is still using less than 70% of available memory, you have nothing to be concerned about.

Irrespective, it is worth making sure that you have the up to date VMtools installed.

[gregchait]

I guess to reiterrate my biggest concern in one statment, it would be "how should I efficiently address consumed memory?".  I suppose with W2K3 the OS released back memory from host consumption as needed.  It would seem that based on what I am seeing/reading this is more of a fine tuning approach.

I just cannot have two servers using 655 MB of average active memory but consuming 8GB from the host.

~grc

[AndreTheGiant]

You can decrease the VM memory if you see that few is required.

But remember that the host can reclaim memory (when it need) with baloon driver (in VMware Tools).

Andre

[gregchait]

Andre (loved you in Wrestlemania III)

When you say "decrease the VM memory" are you just suggesting what is allocated to it?

Greg C.

[AndreTheGiant]

Know the real amount of memory that is need to work fine is not simple...

The active memory is not a good indicator... so you can check into the guest OS and watch for the allocated memory.

Andre

[Scissor]

Do you have memory reservations set on these "problem" Guests?

[gregchait]

Zero reservations, limits or resource pools.  It's all open.

Basically my concern is that I have 56GB on each box in a 3 Host Cluster.  I grant 1 proc & 2GB RAM to about 25 VMs and then have about 8-12 that have 4-8.  They are all consuming their allocated limits.  I never saw this with w2k3, so its substantially throwing off my capacity planning.

Meanwhile, my procs aren't being touched.  Extremely low-utilized...

~grc

[Scissor]

Can you show a chart or screenshot of this behavior?

[gregchait]

I won't be able to access the environment until later this evening, but I'll throw up a chart.

What I can tell you is we have a SQL server with 2 processors and 8 GB RAM.  (As previously mentioned I oppose this allocation, but was overridden)

The server boots and starts with about 2 GB memory consumed.  That grows over the course of 3-4 days until it maxes out the 8 GB and then it stays pegged at the 8GB memory used.  Meanwhile, while that's occuring the active memory averages 655 MB and peaks at 1 GB.  When setting up PerfMon it states that the lowest amount of memory available in MB is "4748".  This definitely would not correalate with active memory, which I would assume would be an inverse of of available memory.

I'll get that chart up later.  We run a fairly vanilla environment.  I can't imagine anything with our configuration that would cause this.  I guess I am just wondering if this is completely expected behavior.

Thanks

~grc

[ats0401]

In my opinion active memory is almost a completely useless metric in 2008 servers; I ignore it completely. If you want to know if you have enough memory, monitor it from the GUEST side. Monitoring from the ESX side usually only tells you if ESX has enough to give to the VM, not so much if the VM is sufficiently sized.

Many applications (like exchange 2007) grab all available memory and load the database into memory to reduce IOPS. It then acts as a "memory manager" and only free up pages to applications that request them; otherwise it holds onto all consumed pages.

What I would is is decrease the allocated memory to a very low value (2GB or 4GB) and then monitor the metrics from the OS side to see if it is memory starved (look for a high number of hard page faults).

Check out these two links for server 2008 memory monitoring

http://blogs.technet.com/b/askperf/archive/2008/06/10/the-basics-of-page-faults.aspx

http://support.microsoft.com/kb/2267427

[gregchait]

Andy

Thanks for the response.  The servers that are most in question are the ones that we have allocated more than 2GB of memory to; primarily SQL and SharePoint.  Right now they are extremely underutilized.  I am guessing that the example of Exchange you use would apply to SQL (2008 R2) and SharePoint (2010) as well.

I'll check out these links and see what I can come up with.  In the meantime, I do need to push forward and order more memory for my hosts.  I definitely did not expect this level of memory consumption.

Thanks

~grc

[J1mbo]

Something else to mention is that 2k8 uses large pages, which ESX(i) does not consider for TPS opporunities unless it is otherwise short of RAM, in which case they are broken into standard pages and then considered.  So in other words, the utilitisation from the vSphere client will generally be near 100% of that allocated, if there is sufficient RAM to accomodate the same.

[gregchait]

Large pages was the culprit.  After making the configuration change in my three host cluster last night, my consumed memory for ~50 VMs went from 115 GB to 63 GB immediately.

I guess the only question that now poses for me, is what is the residual effect of making this change?

Anyhow, thanks for the help here...

Greg C.

[ats0401]

I assume you disabled large pages in your VM's.

VMware recommends to leave large pages enabled, there is a noticable performance gain using large pages (people have reported 15-20% performance gain using large pages)

Large memory pages are not shared by TPS. This is why you will see more memory consumed with large pages enabled.

However, VMware will break up large pages into small pages and share them with TPS when the host is under memory pressure.

My recommendation, leave large pages enabled and let vsphere works it magic if the host comes under memory pressure.

check out these links-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021896

http://www.yellow-bricks.com/2010/11/07/how-many-pages-can-be-shared-if-large-pages-are-broken-up/

http://www.yellow-bricks.com/2009/03/06/virtualized-mmu-and-tp/

[gregchait]

Andy

So in essence what you're saying is do not change the host setting "Mem.AllocGuestLargePage to 0", but rather input "

monitor_control.disable_mmu_largepages = TRU" within each VMX?

Greg C.

[ats0401]

I'm not sure, it's enabled by default within VMware so I've never messed with it.

I would just change Mem.AllocGuestLargePage back to the default of 1 if it was me.

You shouldn't have to do anything per VM that I'm aware of.

You'll have to make the decision to what you want to do

(performance gain with large pages enabled or increased page sharing with TPS with large pages disabled)

Large pages will give you the perception that you are running out of memory on the host, but TPS will kick in once

you hit the threshold if 94% memory consumption on the host.

If you can, run some tests and see if you have a noticable performance drop by disabling large pages.

Also, with large pages enabled, put a host or two in maintenance mode and force the host to become memory constrained

and see how well the TPS works when it breaks up the large pages into small pages (I personally think it will work very well). That might help you decide.

[Cynomus]

Here is an example of what he is talking about.

These 6 servers are Dell R910s with 24 cores hyperthreaded and 128 GB RAM each.

Each server has only about 30 VMs running W2K8 R2 or lower.

If this were all W2K3, each server could run 120 or more and still have room to spare.

VMware needs to update the VMtools to address the 64-bit memory management more efficiently.

Anyone else find any solutions towards improving this memory management?

We want to make sure we have enough memory for when the server demand is high, and have always been able to do this prior to W2K8.

[Generious]

Have you had a look at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020524

and

http://blogs.kraftkennedy.com/index.php/2010/05/27/vmware-kb-clarifies-page-sharing-on-nehalem-processors/

and take a read of

http://www.yellow-bricks.com/2010/05/27/kb-article-1020524-tps-and-nehalem/

[davecook79]

We had a similar server that was running 2K8, SQL and SCCM. I had a similar battle to you when the app owner was asking for silly amounts of RAM, I lost as well.

However I tried to put limits on the server as this was in VMware I figured the host should be oblivious and would just max out if it hit the limit and then I could adjust. However the server would hit the limit almost imediatly and then begin its paging. Obviously the app owner got quite upset over his server performance so I gave in and let it have all the 8GB. In actual fact I ended up giving it 16GB as the performance when running largish reports was poor.

I was wondering if its more a problem with how SQL and SPS work. As I know that SQL will grab all memory it can and hold onto it until its told to let it go, have you tried to limit SQL server in the OS and see what effect that has on Proceedings? I assume you are not seeing a performance hit so this could be a good thing to test?